Detailed Analysis
A user constrained to Claude Opus 4.7 through Microsoft 365 CoPilot describes a persistent failure pattern in AI-assisted SQL generation, specifically around mathematical calculations within query logic. The user, who self-describes as having only basic SQL knowledge, constructed a 65-line prompt that includes full table schema documentation — field names, data types, and semantic descriptions — as well as explicit formulas using correct field names. Despite this level of detail, Opus consistently produced queries with incorrect calculations. The user also possesses a working version of a closely related query, confirming that the underlying data and logic support the desired output, meaning the failure is attributable to the model's code generation rather than an impossible specification.
The user's iterative debugging strategy reveals a thoughtful but ultimately insufficient approach. Moving from one-shot prompting to a structured multi-turn planning protocol — where the model was instructed to analyze the problem, ask clarifying questions, and only then generate code — produced no meaningful improvement in output correctness. Most telling is the final attempt: providing the working SQL query as a reference and explicitly instructing the model to analyze its formula and incorporate it into the new query. The model returned output with no relevant changes, suggesting a failure not just in mathematical reasoning but in instruction-following at a basic level. This pattern points to a compound problem: the model may be pattern-matching to plausible-looking SQL syntax without accurately propagating arithmetic logic, and the CoPilot interface may be introducing context limitations or system prompt constraints that further degrade performance.
The broader issue here is one of environment-conditioned capability degradation. Claude Opus's performance through third-party integrations like Microsoft 365 CoPilot can differ substantially from its behavior in native environments like Claude.ai or Claude Code, where users have access to features like extended context windows, artifacts, iterative code execution, and more direct prompt control. The absence of Claude Code is particularly relevant: that tool is specifically designed for agentic coding tasks, allowing the model to test, run, and debug code in a feedback loop rather than generating it blindly. Without that execution layer, the model cannot self-correct through empirical validation.
This case also illustrates a well-documented limitation in large language models when tasked with precise arithmetic embedded in procedural code. SQL aggregation queries — particularly those involving multi-step calculations across grouped records — require exact logical sequencing that LLMs can approximate structurally but frequently misexecute mathematically. The user's instinct to provide a reference working query was sound in principle, but without a mechanism for the model to execute and compare outputs, the instruction to "analyze the formula" becomes an abstract directive the model can satisfy superficially without actually altering its computational approach. Prompting strategies that work around this limitation typically involve decomposing the formula into atomic steps, having the model verify each step independently, and building the final query incrementally rather than as a single generation.
Read original article →