Detailed Analysis
A hobbyist rom-hacker known as "kugel" has been running an informal but rigorously documented benchmark on successive Claude models: port Super Metroid Map Rando seeds to the SA-1 coprocessor, a task requiring rewriting bank references across a 4MB ROM in 65816 assembly, a domain kugel deliberately has zero expertise in. The test is designed to isolate model capability from human knowledge, since kugel can only describe symptoms observed in gameplay and cannot verify or write any of the underlying fixes himself. Where Opus 4.6 produced a foundation that booted but was riddled with corruption — garbled tiles, walk-through walls, crashing doors, no working saves, and functionality limited to a single emulator — the newer Fable model, working from that codebase across three sessions, brought the same project to a fully playable state across three emulators and real SNES hardware via an FXPak Pro, with saves surviving power cycles and graphics verified byte-exact against the unmodified game.
The technical substance of the fixes is what makes this account notable beyond a simple pass/fail result. Fable's write-up (produced by the model itself, at kugel's request) details a series of forensic debugging feats: discovering that 44% of an inherited disassembly's "rewrites" were artifacts of a tool that had walked through padding bytes as if they were code; tracing a single corrupted byte that desynced graphics decompression across the entire game; catching an off-by-one labeling error in a community-maintained disassembly that had silently broken Samus's right-facing sprite in every build ever produced; and identifying that the emulator's own memory-inspection API had been misreading ROM as VRAM, meaning years of "measurements" the project relied on were invalid. Rather than continuing to patch a flawed memory-mapping scheme, Fable redesigned it outright, cutting total rewrites from over 8,000 to roughly 1,150 while incidentally fixing lag and restoring title-screen music nobody realized was missing. This progression — from incremental patching to recognizing that the underlying model of the problem was wrong and needed replacing — is a qualitatively different mode of engineering than pattern-matching fixes onto symptoms.
Equally significant is the account of how debugging proceeded when the model had no visual access to the running game at all. Fable's testing was entirely headless, done through memory-diffing against a clean ROM reference, meaning all bug reports arrived as kugel's plain-language descriptions of what he saw on screen ("firing a missile melts the game into a slow checkerboard," "this cactus walks faster left than right"). The model had to reconstruct reproducible test cases from those descriptions alone, at one point building tooling that could ingest a player's savestate to recreate an exact crash condition it could never have generated synthetically. Several unrelated-seeming bugs were traced back to a shared root cause — a batch of 254 phantom rewrites scattered across enemy AI and projectile-dispatch tables — and fixed in a single purge once the pattern was recognized, illustrating an ability to generalize from isolated symptoms to systemic causes rather than treating each bug report as independent.
This case sits within a broader pattern of AI labs and enthusiast communities using long-horizon, real-world engineering tasks — rather than static benchmarks — as more meaningful tests of model capability, particularly for domains like retro console assembly where training data is scarce and correctness is unforgiving (a single wrong byte can corrupt an entire ROM). It also reflects a growing trend of models being tasked with auditing not just code but the tools and instrumentation used to evaluate that code, since Fable's biggest breakthroughs came from realizing that established measurement methods (a disassembler, an emulator's memory API) were themselves unreliable. As models are increasingly asked to operate in from-scratch, low-resource technical domains with human collaborators serving purely as sensory proxies, this kind of anecdotal but detailed case study offers a more textured picture of capability growth than benchmark leaderboards typically convey — while kugel's own framing, that "one project is not a controlled benchmark," is an appropriately modest caveat on drawing sweeping conclusions from a single (if unusually well-documented) example.
Read original article →