Detailed Analysis
A Reddit user building an AI-assisted motorcycle video editing pipeline with Claude and Remotion is encountering a core limitation that sits at the intersection of prompt engineering, multimodal AI capability, and the architectural constraints of current large language models. The workflow in question is designed to ingest raw moto footage alongside creative references and music, then generate contextually appropriate edits — with clip selection timed to musical moments like drops and transitions. The user reports that despite extensive iteration, Claude consistently selects shots that are technically adequate but contextually hollow, such as choosing a well-lit helmet over a dynamic lean-in moment during a high-energy musical climax. The stack includes Remotion, ffmpeg, PySceneDetect, OpenCV, and WhisperX, and the user has migrated from a more complex Claude CLI / DaVinci Resolve setup — abandoned partly because DaVinci Resolve's free tier does not support scripting — to a leaner Claude Desktop / Code configuration running on a Pro plan with Claude Sonnet.
The fundamental problem the user is experiencing is not primarily a prompting failure, though that is a contributing factor. It reflects a genuine architectural gap: Claude, like most large language models, does not natively "watch" video. The tools in the user's stack — PySceneDetect and OpenCV — are responsible for segmenting footage into discrete scenes, but they operate on low-level visual heuristics such as pixel difference thresholds and frame entropy, not semantic or cinematic understanding. What gets passed to Claude is almost certainly a set of metadata descriptions, timestamps, or extracted frame data rather than the video itself. This means Claude is reasoning about the footage from a representation that may already have lost the most important signal: the kinetic, visceral quality of a lean or a throttle snap that makes a moment a "highlight." The model cannot feel the energy of a shot it has never actually seen; it can only reason from what the pipeline describes to it.
The instruction-following inconsistency the user describes — rules being ignored, priorities mixed, output becoming unpredictable — is a well-documented behavior in complex, multi-constraint prompting scenarios. When a system prompt contains many competing directives (musical timing, visual impact, clip ranking, style guidelines), Claude's attention mechanism does not uniformly honor all of them, particularly toward the end of long context windows or when instructions conflict implicitly. This is sometimes called "instruction drift" and is exacerbated when the model must also handle large amounts of metadata about dozens or hundreds of clips simultaneously. The solution typically involves structuring instructions hierarchically, reducing the number of simultaneous constraints, and using explicit chain-of-thought prompting to force the model to reason step-by-step about each clip before making a selection decision, rather than pattern-matching to the nearest plausible answer.
The broader trend this case illustrates is the gap between what AI-assisted creative workflows promise and what current LLM-based architectures can reliably deliver without deeper multimodal integration. Tools like Google's Gemini 1.5 Pro and certain vision-language models are beginning to accept video frames or even short video clips directly, which would allow for genuine semantic analysis of motion, composition, and energy within a shot — the kind of understanding the user is looking for. Claude, as of mid-2026, processes images but not raw video streams, making the quality of the intermediate representation layer — how PySceneDetect and OpenCV encode "what happens" in a clip into something Claude can act on — the single most important variable in the entire pipeline. Users attempting to build at this frontier frequently discover that the AI's role is best framed not as an autonomous creative agent that understands footage, but as a structured reasoning engine that can apply rules and rank options, provided the upstream data pipeline translates visual content into rich, semantically meaningful descriptors first.
The user's question — whether this is a prompting issue, a rules issue, a lack of video understanding, or a fundamentally unsuitable approach — is best answered as "all four, in descending order of fixability." Prompt structure and rule clarity are immediately addressable. The absence of native video understanding is a known constraint that requires an architectural workaround, such as richer scene descriptors or integrating a dedicated video captioning model as a preprocessing step before Claude ever receives clip data. And while the overall approach is not unsuitable, it requires accepting that Claude is functioning as a reasoning and ranking layer, not a cinematography-aware creative intelligence — a distinction that, once internalized, tends to unlock more realistic and productive workflow design.
Read original article →