Detailed Analysis
A Reddit user conducted an informal but practically grounded benchmark of seven AI models — including two Claude variants, GPT-5.5, Gemini 3.5 Flash, GLM-5.1, Qwen 3.7 Max, and the newly released Claude Fable 5 — on a single, real-world coding task: building an offline YouTube player in TypeScript that indexes yt-dlp downloaded videos, stores metadata in SQLite, serves a YouTube-like web interface, and tracks playback position. The author evaluated only the final output without reviewing intermediate code, making functional end-product quality the sole criterion. The results revealed significant variation in feature depth, cost-efficiency, and reliability across models, with no single entrant achieving an unambiguous victory.
Claude's performance in this test was notably mixed and highlighted an emerging tension in the model lineup. Claude Opus 4.8 performed the worst among the paid API models in terms of cost-to-output ratio — consuming 15 minutes and costing $1.82 while delivering only a basic solution with real-time search and suggestions, prompting the author to describe it as having "some kind of stroke." Claude Opus 4.6, tested as a follow-up, delivered a comparable or slightly superior result (adding channel and in-progress filters) at a fraction of the token usage and cost ($0.758), and was the only model to substitute Bun for Node.js, suggesting stronger practical judgment. Claude Fable 5, the headline entry, landed near the bottom of the feature richness rankings despite a mid-range cost of $1.563, producing output comparable to the much cheaper GLM and Qwen models. The implication across all three Claude outputs is that Anthropic's models underperformed relative to their pricing tier on this task.
The standout performer by feature count was Gemini 3.5 Flash, which added watched/in-progress/completed filters, a channel sidebar, real-time search, a reindex button, and a range of UI refinements — all in approximately five minutes at $0.906. However, it also introduced the most significant bugs: some videos failed to render entirely, and it loaded Tailwind CSS via CDN, introducing an internet dependency that directly contradicted the offline use case. The author's eventual decision to fix and use Gemini's output is instructive — it suggests the model generated a high-value starting point that required manual correction rather than a deployable product, reinforcing the idea that feature ambition and reliability are not always correlated. GPT-5.5 similarly overreached, adding non-functional UI elements despite producing a richer feature set than the Claude models.
The comparison surfaces a broader debate about what constitutes good model behavior in response to constrained prompts. The author explicitly requested simplicity ("no fancy tech"), yet rewarded models that added unrequested features — then acknowledged the contradiction. This reflects a genuine tension in LLM-assisted development: literal instruction-following versus anticipatory elaboration. Models like GLM-5.1 and Qwen 3.7 Max scored well on reliability by adhering closer to the prompt, while more "creative" models like Gemini produced richer but buggier outputs. The local Qwen model (qwen3.6:35b-a3b-coding) offers a particularly interesting data point — running at zero cost over 40 minutes with 8.3 million tokens, it produced a partially functional result with genuinely novel features like chapter lists and sort options, suggesting local models are closing the gap on cloud APIs for code generation tasks given sufficient compute time.
The broader significance of this benchmark lies in what it reveals about value-for-cost dynamics in the current AI model market. As of mid-2026, the landscape has become sufficiently competitive that cheaper models (GLM at $0.443, Qwen at $0.45) are producing functional software comparable to much more expensive flagship models. Claude Fable 5 and Opus 4.8 both failed to justify their price premiums on this task. For developers building practical tools rather than pursuing frontier research capabilities, these results suggest a growing case for cost-tier evaluation alongside raw capability assessment. Anthropic in particular faces pressure to better differentiate its mid-range and premium offerings when competing models deliver similar or richer functional outputs at significantly lower cost.
Read original article →