Detailed Analysis
A developer-published benchmark testing a novel wire format called GCF against JSON and a third format called TOON has produced notable results for Anthropic's Claude model family, with Claude Sonnet 4.6 achieving 100% comprehension accuracy on GCF across every run, and both Claude Opus 4.6 and Claude Haiku 4.5 scoring 96.2%. The evaluation involved 500 symbols and 200 edges per payload, 13 extraction questions per run, and no system prompt or format instructions — meaning models received only raw data and a question. Across all 10 models and three providers (Anthropic, OpenAI, and Google), GCF averaged 90.7% comprehension accuracy, compared to 68.5% for TOON and 53.6% for JSON, representing a substantial gap. The benchmark comprises over 1,300 total evaluations across 23 comprehension runs and 28 generation runs, with full methodology and raw logs published openly.
The contrast between GCF and JSON performance illuminates something significant about how token efficiency and format structure affect model reasoning. JSON's representation of 500 symbols consumed 53,341 input tokens, compared to GCF's 11,090 — a roughly 4.8x difference. This token bloat appears to degrade not only efficiency but accuracy, with JSON failures characterized by verbose, exhausting enumeration attempts that still yield wrong answers. The published failure artifact shows Claude Opus spending 143 lines manually enumerating symbols before arriving at an incorrect result, a pattern consistent with models losing track of context across extremely long structured inputs. GCF errors, by contrast, were minor off-by-one or off-by-two misreads of section headers, suggesting the format degrades gracefully rather than catastrophically.
The generation results reinforce the comprehension findings. All three Claude models — Opus, Sonnet, and Haiku — scored 5/5 on GCF generation using only a three-line primer, with zero prior training on the format. This is particularly striking for a format that did not exist when these models were trained, indicating that Claude's instruction-following and structural generalization capabilities are robust enough to produce valid, decoder-parseable output from minimal specification. TOON, by contrast, saw near-complete failure on generation tasks across most models, with Opus scoring 0/5, suggesting that format legibility and generatability are not uniformly correlated properties.
The broader significance of this benchmark lies in what it reveals about the relationship between data serialization design and LLM performance. As AI systems are increasingly embedded in pipelines that consume and produce structured data, format choice becomes an architectural decision with measurable accuracy implications. JSON's dominance as an interchange format was built around human readability and ecosystem ubiquity, neither of which is optimized for token-constrained language model inference. The GCF results suggest that purpose-designed formats with compact, consistent structural grammar can meaningfully outperform incumbent formats on comprehension tasks — not because models are trained on them, but because their token economy reduces the working memory burden imposed on attention mechanisms.
The fact that this benchmark was conducted and published by the GCF format's own creator introduces an important caveat: independent replication is necessary before the results can be treated as definitive. The author has published the full evaluation suite as open-source Go test code, the raw logs, and a DOI-registered whitepaper, which lowers the barrier to verification. If the results hold under independent testing, they contribute meaningfully to a growing body of evidence that prompt and format engineering — including the structure of data passed to models, not just the instructions — has first-order effects on reasoning accuracy at scale.
Read original article →