Detailed Analysis
A Reddit user's head-to-head comparison of Claude Sonnet 5 against a model called "Fable 5" has surfaced an important nuance in how AI coding assistants should be evaluated: per-token pricing is a poor proxy for per-task cost. The tester ran both models through two coding challenges — adding a RAG debugger to the 400K-line Open WebUI codebase, and building a browser-based Clash Royale-style game from scratch. On the simpler debugging task, Sonnet 5 did come in cheaper ($12.05 vs. $16.11), though it took over twice as long as Fable 5 and produced a narrower solution that only handled one document at a time rather than the full knowledge base. On the more demanding game-building task, however, the economics inverted: Sonnet 5 took nearly twice as long (93 minutes vs. 48), generated more code (7,056 lines vs. 5,817), burned more than double the output tokens (404.9K vs. 183.5K), required multiple rounds of bug fixes, and ultimately cost more overall ($33.85 vs. $27.97) despite nominally being the "cheaper" model on a per-token basis.
This finding matters because it challenges a common assumption in the AI coding tools market — that lower list prices automatically translate to lower real-world costs. As models are increasingly marketed and compared on token pricing, developers and engineering teams making procurement decisions often default to whichever model's rate card looks better. But the Reddit test suggests that verbosity, error-correction overhead, and iteration cycles can dwarf raw token pricing differences, especially on complex, multi-step coding tasks where a model's first-pass reliability determines how much rework is needed. Fable 5 completing the game in a single run with 31 passing tests and no major bugs — versus Sonnet 5 needing several fixes to get a "probably better" UI working — illustrates that task completion quality has a direct, compounding effect on total spend.
The results also feed into an active debate about Claude Sonnet's evolving strengths and weaknesses. The poster notes that Sonnet 5 is "clearly better" than its predecessor, particularly for frontend work and tool use, which aligns with Anthropic's broader positioning of Sonnet as a strong all-around coding model. But the tendency toward higher output-token counts and multi-pass corrections on harder, longer-horizon tasks points to a known tension in large language model coding performance: models that are highly capable at generating polished-looking UI can still struggle with underlying logic correctness on complex, stateful systems like real-time game engines with matchmaking, energy systems, and win conditions. This is a recurring pattern across frontier coding models, where benchmarks that reward surface-level competency don't always capture the downstream cost of debugging cycles.
More broadly, this comparison is a microcosm of a maturing conversation in AI-assisted software development: raw benchmarks and token-price comparisons are increasingly seen as insufficient for evaluating coding agents, and practitioners are pushing for task-completion-based metrics — total time, total cost, bug rate, and lines of functional code — as more meaningful yardsticks. As competition intensifies among Anthropic's Claude models, OpenAI's GPT line, and other entrants like the "Fable" model referenced here, this kind of grounded, task-level benchmarking is likely to become more common, pressuring vendors to optimize not just for cheaper tokens but for fewer tokens and fewer corrective iterations needed to reach a working, well-tested product.
Read original article →