IMPORTANT: Big Issues are cropping up re Agents transitioning from Claude Sonnet 4.6 to Claude Sonnet 5, with Fable 5 used as another experimental control
We’ve been working recently with Sonnet 5 and Fable 5.
Fable 5 is magnificent but costly. Sonnet 5 has some substantial issues.
Specifically, if you have agents running on Claude, or you’ve built a product offering on its Sonnet 4.6 model, you need to immediately test and re-evaluate their operational integrity if they are running now on Sonnet 5.
We have picked up extensive issues between Sonnet 4.6 and 5, ranging from adherence to instructions, refusal of instructions by Sonnet 5 that are accepted by Sonnet 4.6 and Fable 5, and the generation of different outcomes to varying degrees of severity.
The baseline of activity on Sonnet 4.6 is very strong and consistent. But as soon as the operating model is switched to Sonnet 5, you can get — and almost assuredly will get — compliance and operational compliance issues.
Fortunately, Fable 5, which represents the strategic direction for Anthropic, does not exhibit these transitional issues. But Fable 5 also is expensive enough to preclude it as an option.
We will share more information as we can
.
Thanks for sharing, Mark. But jeez, do these companies test their AI models on real-world applications before they release them (we know the answer to that)?
I'm sure they'll say it's too hard to build a test rack for real-world use cases and then try to determine differences in behavior and responses across the different models. But that's an excuse, not a reason.
Heck, can't they leverage their very own AI to verify the differences? Just buy some more GPUs!!