Large language models can produce a correct answer to a database question while relying on a computation that does not hold up under inspection, according to a proof-of-concept study. The finding points to a gap in how data agents are tested: a right answer may conceal a flawed route to that answer.
The paper calls this mismatch the “Structure Gap” — the distance between natural-language reasoning and the actual operators used to compute a result. Its proposed remedy, Trace Integrity, asks whether the recorded computation is supported and auditable, rather than treating the final answer as the whole test.
Making the route inspectable
The approach uses execution contracts: structured records that connect a user’s request to the relevant database schema, the planned operations, assumptions, execution queries, verification status and final answer. The goal is to make the computation behind a result inspectable as well as the result itself.
A deterministic validator then checks the computation at the operator level. The reported failure categories included missing joins, wrong aggregations, missing filters, invalid schema references, incorrect grouping keys, wrong sort or limit operations, mismatches between an answer and its trace, and inconsistencies between a contract and the SQL query.
The paper also advances an “Isolation Principle”: systems should, by default, specify the intended computation before accessing value-level data, and document why such access was needed when an exception occurs.
Three ways to ask for SQL
The researchers tested 100 stratified examples from the BIRD Mini-Dev benchmark. Gold and generated SQL were run against the same database, and generated traces were compared with normalized reference traces. The test used claude-haiku-4-5 with the model temperature set to 0.0.
Each example was evaluated under three prompting conditions: Direct SQL, Operation Summary + SQL, and Contract-First SQL. The conditions received the same question and schema context and used the same executor and trace validator, allowing the methods to be compared on common examples. The three conditions together produced 300 method-example predictions.
The evaluation recorded five signals: Answer Accuracy, Execution Success, Trace Integrity Pass Rate, Answer-Trace Consistency and CAIT Rate. CAIT, short for correct answer, invalid trace, was the share of correct answers backed by invalid traces, while answer accuracy was based on whether the returned row set matched the gold SQL result.
The scores did not move together
On Direct SQL, answer accuracy was 20.0% and execution success was 84.0%. Trace Integrity Pass Rate was 39.0%, Answer-Trace Consistency was 84.0%, and CAIT Rate was 55.0%.
Operation Summary + SQL recorded 22.0% answer accuracy, 83.0% execution success, a 43.0% Trace Integrity Pass Rate, 67.0% Answer-Trace Consistency and a 59.1% CAIT Rate.
Contract-First SQL had the highest answer accuracy, at 24.0%. Its execution success rate was 82.0%, Trace Integrity Pass Rate 40.0%, Answer-Trace Consistency 82.0% and CAIT Rate 45.8%.
That meant the leading method changed depending on what was measured. Operation Summary + SQL had the highest Trace Integrity Pass Rate, at 43.0%, but also the highest CAIT Rate, at 59.1%. Contract-First SQL had the lowest CAIT Rate, at 45.8%, while its Trace Integrity Pass Rate was 40.0%.
CAIT cases numbered 11 under Direct SQL, 13 under Operation Summary + SQL and 11 under Contract-First SQL. Across the 300 predictions, 51 queries did not execute, representing 17.0%.
These were descriptive metric profiles, not causal effects. No inferential tests or confidence intervals were reported.
A warning about one-number scorecards
The authors interpret the demonstration as showing that answer accuracy, Trace Integrity Pass Rate and CAIT Rate measure different properties. They recommend reporting those signals together and using auditable computation traces for deployed data agents.
The study does not establish that Contract-First SQL solves text-to-SQL or provide stable rankings of prompting formats.
The authors characterize the work as a scoped proof-of-concept using 100 examples, one model and fixed prompting and execution, so these percentages are not stable rankings of models or prompt formats. The deterministic validator is not a full semantic-equivalence checker and may penalize valid SQL rewrites; BIRD Mini-Dev can contain multiple valid SQL programs for the same question.
Questions remain about whether the pattern generalizes to other models, prompting strategies, datasets and production environments, and about how validators can recognize semantic equivalence without over-flagging valid SQL rewrites.
The supplied document is an arXiv version 1 preprint dated 26 Aug 2026.
Paper data and sources
Original title: Trace Integrity for LLM Data Agents: A Vision for Auditable Structured Reasoning in Real-World Systems
Authors: Srimonti Dutta, Akshata Kishore Moharir
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text