Detailed Analysis
A developer debugging token usage during an extended coding session uncovered a striking inefficiency in how AI coding agents consume API budget: the vast majority of cost in a single $2.95 API call came not from generating output, but from re-reading and re-caching context already present in the window. The trace data shared makes the problem concrete — Call 1 incurred $2.53 primarily because it wrote 401,702 tokens to cache, while Call 2, benefiting from a warm cache, cost only $0.28 despite producing over 2,500 output tokens. The output itself was essentially free by comparison. This asymmetry reveals that in long agentic coding sessions, the dominant cost driver is not inference but redundant context management — the agent repeatedly ingesting the same files, histories, and outputs that it has already processed.
The insight the author draws is intuitive once stated plainly: a skilled human engineer does not reopen the same file on every turn of a debugging session. They build a mental model, take notes, and retrieve only what is immediately relevant. Current AI coding agents largely lack this discipline. Instead, they tend to grow their context windows monotonically across a session, re-exposing the model to previously seen material with each successive call. When models like Claude use prompt caching mechanisms (which charge for cache writes at a higher rate than cache reads), a cold or partially cold cache on a large accumulated context produces exactly the kind of cost spike the author observed — a single call that is expensive not because of what the model said, but because of what it had to re-read.
The author's response is a locally hosted MCP (Model Context Protocol) server called CostAffective, which exposes three core primitives to coding agents: `remember()` for persisting important facts, `stash_context()` for externalizing large outputs from the active context window, and `recall()` for targeted retrieval of specific stored information. The tool also incorporates repository intelligence via Tree-sitter, a widely used incremental parsing library, to enable more structured understanding of codebases without requiring full file re-ingestion. Critically, the entire system runs locally without external API dependencies, keeping sensitive codebase content on-machine and eliminating a potential cost and privacy surface.
This development reflects a broader and rapidly maturing area of applied AI engineering: context economy. As frontier models like Claude expand their usable context windows into the hundreds of thousands of tokens, the naive approach — simply feeding everything in — becomes economically untenable at production scale or even in extended single-user sessions. Prompt caching, offered by Anthropic and others, helps amortize re-reading costs when context is stable between calls, but it does not solve the fundamental problem of agents that keep appending rather than curating. The MCP ecosystem, which Anthropic introduced as a standard for tool integration with Claude, is increasingly being used by developers not just to give agents capabilities but to give them memory architecture — a layer that sits between the agent loop and the raw context window and manages what the model actually needs to see at any given moment.
The pattern the author identifies points toward a design principle that is likely to become standard in agentic AI tooling: agents should maintain an explicit separation between long-term knowledge (persisted facts about a codebase or session), working memory (what is needed for the current task), and ephemeral outputs (large intermediate results that may never need to be re-seen). CostAffective is a practical, open-source instantiation of that principle. The broader question the post poses to the community — how others are managing long-session context growth — signals that this is an active, largely unsolved engineering problem, and one where developer-built tooling is currently outpacing built-in agent frameworks. As agentic AI usage scales, the economics of context management will increasingly determine which approaches are viable in practice.
Read original article →