Detailed Analysis
A user on r/ClaudeAI has surfaced a subtle but consequential infrastructure problem affecting Claude's ability to crawl and fetch web content. The issue traces back to a specific IP address, 34.162.230.222, which is part of Anthropic's officially published crawler range documented at claude.com/crawling/bots.json. Despite being legitimate Anthropic infrastructure, this same address carries a 100% abuse confidence score on AbuseIPDB, a widely used reputation database that many content delivery networks (CDNs) and web application firewalls (WAFs) consult by default. When the site owner tried to have Claude fetch content from their docs subdomain, the request failed with a ROBOTS_DISALLOWED error—even though robots.txt was fully permissive, curl requests succeeded, and other AI crawlers had no trouble. The root cause wasn't robots.txt logic at all; it was a reputation blocklist silently rejecting the request before robots.txt was ever evaluated, on a different backend stack than the one serving their main domain.
This matters because the failure mode is nearly invisible to the people who most need to see it. The error message Claude receives (ROBOTS_DISALLOWED) points users toward debugging robots.txt configuration, when the actual blocker is a reputation-based filter operating at a completely different layer of the stack. Site owners see their pages load fine in a browser, their robots.txt validates correctly, and their monitoring shows no failures—because from the server's perspective, nothing is broken; a blocklist rule is functioning exactly as designed. Meanwhile, anyone relying on Claude (or similar AI agents) to fetch their content, including sites that have gone through the effort of publishing an llms.txt file to explicitly welcome AI crawlers, may find that access silently fails with no clear signal pointing back to the actual cause. This is a classic case of a well-intentioned security default (block IPs with poor reputations) colliding with a use case that its designers likely never fully anticipated (legitimate AI agents fetching documentation on behalf of users in real time).
The structural issue underlying this bug is that Anthropic's crawlers, like those of many cloud-based services, operate on shared cloud IP ranges rather than dedicated, permanently owned addresses. Shared IPs accumulate reputational baggage from whatever other tenants previously used them, or from users who report AI-crawler traffic itself as "abuse" regardless of whether it's harmful. Because reputation blocklists key off IP addresses rather than verified crawler identity, a single tainted IP within Anthropic's published range can quietly cut off access to an unknown number of sites simultaneously, particularly those using CDN/WAF products where such blocklists are enabled by default rather than opt-in.
This incident sits at the intersection of two accelerating trends: the rise of agentic AI systems that autonomously browse and fetch live web content on users' behalf, and the web's existing anti-bot infrastructure, which was largely built to stop scrapers and malicious traffic rather than to distinguish between good-faith AI agents and bad actors. As more products like Claude gain real-time browsing capabilities, and as more sites adopt standards like llms.txt to signal AI-friendliness, mismatches between crawler infrastructure and legacy reputation systems are likely to keep surfacing. Anthropic could mitigate this by working with reputation database providers to whitelist verified crawler ranges, rotating away from tainted shared IPs, or improving error transparency so that ROBOTS_DISALLOWED doesn't obscure blocklist-level rejections. Until then, site owners who want to guarantee accessibility to AI agents may need to explicitly audit their CDN/WAF reputation-blocklist settings rather than trusting robots.txt alone as the final word on crawler access.
Read original article →