Detailed Analysis
A Reddit user's deep dive into building practice exams for Anthropic's four Claude certifications—CCAO-F, CCDV-F, CCAR-F, and CCAR-P—surfaces a subtle but consequential failure mode in LLM-generated content: near-duplicate items that evade standard similarity detection. Using Claude Code to generate question banks from publicly published exam guides on Skilljar, the author found that roughly 78% of items in an early batch were functionally redundant. These weren't verbatim copies but questions testing identical decision logic dressed in different scenarios and distractors—a pattern of superficial variation masking underlying repetition. Notably, both lexical similarity checks and embedding-based cosine similarity failed to catch these duplicates, with scores as low as 0.13-0.18 for pairs later confirmed to be functionally identical. This is a meaningful technical finding: it suggests that semantic embedding methods, often treated as a robust proxy for "meaning," can be fooled when surface features (wording, framing, scenario details) diverge even as the underlying reasoning structure stays fixed.
The methodology the author eventually landed on is worth highlighting because it reflects a broader lesson about evaluating AI-generated content: automated metrics are necessary but insufficient, and human (or fresh-context AI) judgment remains essential for catching certain categories of error. The breakthrough came from a deceptively simple test—handing items to readers with no memory of having written them and asking "which of these are the same question wearing a different hat?" This fresh-context evaluation approach, plus a second technique of stripping answer keys and having the model re-derive answers independently to check for consistency, represents a practical quality-assurance pattern that's increasingly relevant as more people use Claude and similar models to generate structured educational or assessment content at scale.
The underlying diagnosis—that a source document (in this case, an exam guide) only supports a finite number of genuinely distinct testable decisions—is a useful frame for thinking about the limits of LLM content generation more broadly. The author estimated the Architect Foundations guide could support around 90 raw decisions, collapsing to roughly 60 as an actual test-taker would encounter them, and scaled the practice bank accordingly rather than padding it with six redundant forms. This "decision surface" concept pushes back against a common assumption in AI-assisted content creation: that more generated volume is better. Instead, it argues that once generation exceeds the genuine informational capacity of a source, additional output becomes recombination rather than new signal, training pattern recognition rather than actual competence—a critique that echoes broader concerns about synthetic data quality and model training on self-generated content.
This grassroots project also reflects the growing ecosystem forming around Anthropic's certification program itself, which the company has been building out as enterprise and developer interest in Claude has expanded through 2025 and 2026. As certifications become a credentialing pathway for Claude-focused roles—architecture, development, and administration—third-party study tools, much like Udemy banks for other tech certifications, are emerging organically from the community rather than from Anthropic directly. The NDA constraints the author operates under (generating content only from published guides, not recalled exam material) also underscore a tension familiar from other certification ecosystems: balancing community-driven study resources against the integrity of proprietary exam content. More broadly, the piece is a small but concrete case study in the current moment of AI development, where the interesting failures are no longer about models refusing tasks or hallucinating facts outright, but about subtler issues of redundancy, evaluation validity, and the limits of automated quality metrics—problems that will only grow more important as LLMs are used to generate larger volumes of educational, assessment, and reference material.
Read original article →