Detailed Analysis
The Quesma team's decision to rerun their "baba-is-harbor" benchmark reflects the rapid pace of frontier model releases in mid-2026, with four major labs—Anthropic, Moonshot AI (Kimi), xAI (Grok), and Google (Gemini)—all shipping significant updates within a single month. The benchmark itself uses Baba Is You, a puzzle game built around manipulating rule-based logic blocks, as a testbed for evaluating models' spatial reasoning, multi-step planning, and ability to understand and rewrite the "rules" governing a game world in real time. Unlike static QA benchmarks, Baba Is You requires models to hold an evolving state in working memory, anticipate cascading consequences of rule changes, and often backtrack when an approach fails—making it a useful proxy for agentic reasoning tasks that extend well beyond gaming.
The specific question the researchers pose—whether Claude Opus 5 is now cheaper to run than the previous top performer, "Fable 5" (likely an internal or prior benchmark leader)—signals a broader shift in how the AI community evaluates model releases. Raw capability scores are no longer sufficient; cost-per-task and cost-per-solved-puzzle have become first-class metrics alongside accuracy, especially as reasoning models can vary wildly in token consumption depending on how much "thinking" or tool-calling they perform before arriving at an answer. This mirrors a trend already visible in 2025, where Anthropic, OpenAI, and Google began publishing not just benchmark scores but effective cost curves, recognizing that enterprise customers care as much about inference economics as about leaderboard position. A model that solves puzzles brilliantly but burns ten times the tokens of a competitor may be commercially uncompetitive even with a higher raw score.
The framing of the article as a puzzle for readers—"could you guess which model is the most expensive?"—also reflects a maturing benchmark culture where community-run, open-source evaluations increasingly rival or supplement official vendor benchmarks. Projects like baba-is-harbor exist because generic multiple-choice or coding benchmarks (MMLU, HumanEval, SWE-bench) have become partially saturated or gamed through training-set contamination, pushing independent researchers toward creative, harder-to-memorize domains like puzzle games that require genuine on-the-fly reasoning. This is part of a broader 2025–2026 trend of "benchmark diversification," where games (Baba Is You, Sokoban-style puzzles, strategy games like Diplomacy or Werewolf) are used specifically because their rule systems are combinatorially large enough to resist rote memorization.
Taken together, the piece captures two intertwined threads in current AI development: an accelerating cadence of frontier releases from an increasingly multipolar field of labs (Anthropic, Moonshot, xAI, Google all shipping competitive models in the same month), and a growing sophistication in how the community evaluates those releases—moving past simple accuracy toward cost-efficiency, robustness, and resistance to benchmark contamination. For Anthropic specifically, Claude Opus 5's performance and pricing relative to peers on this kind of puzzle benchmark serves as an independent, third-party signal of how well its reasoning improvements translate into practical, cost-effective agentic behavior, which matters increasingly to developers building automated tools and agents on top of Claude's API rather than using it purely for chat.
Read original article →