Price is not cost: we are using the wrong variable to measure the cost of LLMs

Detailed Analysis

The article "Price is not cost: we are using the wrong variable to measure the cost of LLMs" presents a structural critique of how the AI industry currently evaluates the economic viability of large language model deployments. The author's central claim is that per-token pricing — the dominant metric used by providers like Anthropic, OpenAI, and Google to charge for LLM usage — is fundamentally misleading because it charges users for compute consumed regardless of whether the output is correct or useful. This decouples the unit of payment from the unit of value, creating an accounting illusion in which "cheap" inference can mask extremely high effective costs when outputs require verification, correction, or re-generation.

The article's second assertion introduces a compounding reliability problem that makes the first concern structurally worse. Drawing on straightforward probability math, the author demonstrates that even a high per-step accuracy of 95% degrades to roughly 77% reliability across a five-step agentic chain, 60% across ten steps, and below 36% across twenty steps — assuming errors are statistically independent. The author is careful to note that real-world agentic processes violate that independence assumption, meaning cascading or correlated errors will push actual reliability even lower than these already sobering figures. The practical implication is that meaningful automation — the kind that involves multi-step reasoning, tool use, and sequential decision-making — carries a hidden "verification tax" that must be paid either through human oversight, additional inference-time compute (which the author regards as less reliable than human review), or an accepted degradation in output quality.

Taken together, the two assertions lead the author to a strong architectural conclusion: the barrier to cost-effective automation via LLMs is not primarily economic but structural. This connects to a companion essay the author references, which argues that LLMs cannot reason faithfully. The distinction is important: if the problem were purely economic, falling token prices driven by competition among frontier model providers would eventually resolve it. But if the reliability ceiling is architecturally constrained by how autoregressive language models generate outputs — probabilistically, without grounded logical guarantees — then price reductions alone cannot close the gap between current capabilities and the reliability thresholds required for autonomous, unsupervised workflows.

This critique lands at a particularly charged moment in the AI industry. Anthropic, OpenAI, Google DeepMind, and others are heavily marketing "agentic" capabilities — multi-step autonomous task completion — as the next frontier of commercial value. Anthropic's Claude models, for instance, are being promoted through its "computer use" and agentic API features as capable of executing complex workflows. The reliability decay argument poses a direct challenge to these commercial narratives, suggesting that benchmarks measuring single-step task performance are systematically misleading about real-world deployment viability. The industry's standard evaluation frameworks — pass rates on coding benchmarks, MMLU scores, and similar — do not account for compounding error rates across chained operations, which is precisely where the economic and practical value of agents is supposed to emerge.

The broader trend this article participates in is a growing wave of technically grounded skepticism about the gap between AI capability demonstrations and production-grade deployment. Researchers and practitioners have increasingly noted that "vibe-driven" adoption of LLM-based automation — driven by impressive demos and falling API prices — may be obscuring a reliability debt that manifests only at scale or in high-stakes sequential tasks. The author's framing of cost as a function of outcome reliability rather than token consumption is a meaningful conceptual contribution to this discourse, pushing the conversation away from pricing wars and toward harder questions about what statistical reliability thresholds would actually need to look like before agentic AI becomes a sound operational bet for enterprises.