Detailed Analysis
A collaborative study from Microsoft Research, Stanford, Berkeley, and Carnegie Mellon University has exposed a significant gap between the listed per-token prices of frontier AI models and the actual cost of completing real-world tasks. Researchers evaluated eight frontier reasoning models across nine task domains, finding that in more than one in five head-to-head comparisons, the model with the lower advertised price was ultimately more expensive to operate. The most dramatic discrepancy reached a factor of 28x. The headline finding centers on Gemini 3 Flash, which carries a listed price 78% below GPT-5.2, yet across the full task benchmark it cost 22% more to run in practice — a direct inversion of what the pricing page would suggest.
The mechanism driving this divergence is token consumption behavior, particularly the use of "thinking tokens" generated by reasoning models as they work through problems. Thinking tokens accounted for over 80% of total output cost in the study, and consumption varied wildly between models — with one model burning 900% more thinking tokens than another on an identical query. Because per-token pricing is multiplied by the number of tokens consumed rather than the number of tasks completed, a model that reasons verbosely will inflate costs regardless of its unit rate. This distinction — between price-per-token and cost-per-job — is rarely surfaced on pricing pages, which present static rates that implicitly assume consumption is fixed or comparable across models.
A further layer of complexity emerges from within-model variability. The study found that the same query, run against the same model, produced cost swings of up to 9.7x between individual runs. This introduces non-determinism directly into cost of goods sold (COGS), meaning operators cannot rely on stable margin calculations even after selecting a model. The implication is that total LLM expenditure is a function of sticker price multiplied by a variable, model-specific, and partially stochastic consumption figure — none of which is visible at the point of model selection.
For businesses building products or services on top of large language model APIs, the study's findings carry concrete financial risk. Companies that charge flat or subscription-based fees for AI-powered features are particularly exposed: if per-task token consumption is unpredictable, the heaviest or most complex users can silently push per-unit margins into negative territory. The standard practice of anchoring pricing strategy to listed API rates, without measuring actual token burn across representative workloads, effectively means that COGS is unknown until the bill arrives. This is a structural oversight in how LLM costs are currently communicated and internalized by product teams.
The research arrives at a moment when the AI model market is intensely competitive on advertised price, with providers regularly announcing rate cuts as a signal of efficiency and accessibility. The study suggests that this competition may obscure rather than illuminate true cost dynamics for practitioners. As reasoning and chain-of-thought capabilities become standard features of frontier models, thinking-token consumption is likely to grow as a share of total cost, making the gap between list price and operational cost wider and more consequential. The practical takeaway is that empirical cost benchmarking — measuring token consumption on production-representative queries before committing to a model — is now a necessary engineering discipline rather than an optional optimization.
Read original article →