Detailed Analysis
A recent independent benchmark study, conducted by a team including former IMO medalists, evaluated how different large language models perform on the 2026 International Mathematical Olympiad problems—a particularly rigorous test given that the problems are guaranteed to be absent from any model's training data. The study, published alongside a detailed technical report and audit trails on GitHub, compared frontier models (referred to as "sol" and "fable") against Anthropic's Claude Sonnet and Claude Opus, as well as the open-weight model GLM, across multiple configurations: bare webapp usage, Anthropic's own Claude Code harness, and a custom multi-agent orchestration system called AutoFyn developed by the researchers.
The results reveal a notable capability gap. The unnamed frontier models achieved perfect or near-perfect scores regardless of the harness used, suggesting their underlying reasoning capabilities are strong enough to not require heavy scaffolding. Claude Sonnet and Opus, by contrast, performed poorly in raw webapp form, showed meaningful improvement when run through Claude Code, and improved further still under AutoFyn's custom orchestration—yet never fully closed the gap with the frontier models even with the most sophisticated harness applied. Interestingly, GLM, an open-weight model, tracked closely with Sonnet's performance curve, improving similarly when given equivalent harness support, indicating that harness engineering can meaningfully lift weaker base models but cannot fully substitute for superior underlying reasoning capacity.
The findings carry important implications for how the AI field thinks about "capability" versus "orchestration." A recurring theme in 2025-2026 AI discourse has been whether agentic scaffolding, tool use, and multi-agent pipelines can compensate for gaps in a base model's raw reasoning ability. This study offers concrete evidence that harnessing helps substantially but has limits: on the hardest problem in the set (P3), every sub-frontier model in every harness configuration—including one run allowed to churn for 20 hours—converged on the same correct partial solution and then stalled at an identical missing reduction step. This suggests the harness was effective at supplying retrieval, verification, and iterative refinement, but could not inject the specific creative insight the problem required. In other words, orchestration can extend and validate a model's existing reasoning trajectory, but it cannot manufacture the "aha" moment when a model's training-derived intuition simply doesn't reach far enough.
The study also surfaces a persistent and consequential issue: hallucinated correctness in ostensibly verifiable domains. Even in mathematics, where proofs can in principle be checked step by step, the researchers documented at least one case where Sonnet confidently claimed a false solution to Problem 3, requiring manual expert verification to catch. This underscores that hallucination is not merely a problem of factual recall or open-ended generation—it persists even in domains with objective ground truth, where confident-sounding but flawed reasoning chains can slip past automated self-checks. For Anthropic and competitors alike, this reinforces the importance of independent verification layers, whether human expert review or adversarial grading by separate models, especially as agentic systems are increasingly trusted to operate autonomously on high-stakes reasoning tasks. As harness engineering matures across the industry, this benchmark suggests the next frontier of differentiation will hinge less on scaffolding sophistication and more on closing the raw reasoning gap between models—a challenge that scaffolding alone cannot solve.
Read original article →