Detailed Analysis
Claude's web search and retrieval architecture, as reverse-engineered through Anthropic's tool documentation and independent analysis by developer Mikhail Shilkov, operates through a layered pipeline that differs significantly from the intuitive assumption that a large language model directly reads and processes web content. The system separates search and retrieval into two distinct tool calls: WebSearch returns only a post's URL and title (stripping page age and encrypted content entirely in Claude Code's implementation), while WebFetch handles body retrieval on a per-URL basis chosen by the model. This architectural split is deliberate — loading full page bodies for every search result would rapidly exhaust context windows — but it means the main model makes citation and relevance decisions based on highly compressed metadata before ever touching page content. The citation layer is further constrained by hard character caps: 150 characters in the general web_search spec, tightened to 125 characters in Claude Code's internal prompting rules, meaning any quoted text from a source must survive radical compression into a single short string.
The most consequential discovery in this analysis concerns the WebFetch pipeline itself. Rather than feeding raw page content to the primary Claude model, the system converts HTML to Markdown via Turndown, takes the first 100 KB of the resulting plain text, and routes that through Haiku 3.5 — a smaller, faster model — which produces a summary calibrated against the caller's original prompt. Only that summary travels upstream to the main model. The author's own instrumentation, logging WebFetch input and output against a personal homepage, confirmed that what the primary model ultimately received was approximately 1,000 characters distilled from a substantially larger page. The practical implication is that Haiku 3.5 functions as an editorial gatekeeper: it decides which parts of a page are salient to the query and discards the rest before the flagship model ever engages with the content.
This architecture raises substantive questions about how web-accessible content should be structured for AI retrieval systems. The 100 KB ceiling on Turndown-converted Markdown rarely truncates a typical English-language blog post, so raw length is not the primary variable — the Haiku summarization layer is. Content that signals relevance early, maintains clear semantic structure through standard Markdown heading hierarchies, and encodes key claims in self-contained sentences is likely to survive Haiku's compression pass more intact than content that relies on narrative continuity, cross-paragraph anaphora, or visually styled layouts that Turndown cannot represent. The author's hypothesis — that writers should optimize for Haiku as the effective audience rather than the main model — is a meaningful reframe, though it remains empirically unvalidated in terms of measurable citation rate differences.
Two additional architectural constraints compound the summarization problem. First, WebFetch does not execute JavaScript, which means single-page application blogs built on React, Vue, or similar frameworks likely return near-empty content to the fetcher, effectively rendering such sites invisible to Claude's search path regardless of their textual richness. This is a significant blind spot given the prevalence of SPA-based developer blogs and documentation sites — precisely the kind of technical content Claude is frequently queried to synthesize. Second, the Turndown HTML-to-Markdown conversion strips CSS-positioned layout and div-based structure, meaning information encoded visually rather than semantically (sidebars, callout boxes styled through class attributes, multi-column layouts) is likely lost before Haiku even begins its summarization pass, while standard Markdown primitives — H2/H3 headings, ordered and unordered lists, fenced code blocks, and pipe tables — are far more likely to survive.
The broader significance of this architecture sits at the intersection of AI content discovery and the ongoing evolution of how language models interact with the live web. The pattern of interposing a lightweight model as a summarization intermediary before routing content to a more capable primary model reflects a practical engineering tradeoff between context efficiency and fidelity, but it introduces a systematic lossy compression step that content creators, publishers, and SEO practitioners have not yet widely accounted for. Traditional search optimization targeted crawler bots and ranking algorithms; the emerging challenge is optimizing for a summarization model operating under prompt-conditioned relevance constraints with no guaranteed consistency across queries. The author's framing — whether "write for Haiku, not for the main model" is the correct mental shift — captures a genuinely novel problem in the AI content ecosystem, one that remains largely in the empirical hypothesis stage but is grounded in documented architectural reality.
Read original article →