A preprint benchmark of eight large language models found a recurring failure in structured-data questions: when the evidence did not support an answer, no-refusal was the default under every reliability operator, and silent unsupported answers accounted for 74.8% of observed failure forms.
The results also separated two kinds of performance. GPT-5.5 had the best reliability result, with average MRS of 24.21%, while Claude-Sonnet-5 had the best robustness result, with average ASR of 9.10%. MRS summarized evidence-grounded abstention, whereas ASR summarized conditional robustness failure: whether a previously correct answer held after the table representation changed.
Testing the edge of the evidence
The benchmark used structured-data tasks from AIDABench-QA and DABench. Seven reliability operators broke necessary evidence, while 12 robustness operators preserved answerability and the correct answer while changing the representation or adding redundancy.
An Agentic-LLM-based generation framework instantiated 19 perturbation operators. The final benchmark contained 2,340 accepted perturbations. Candidates went through automated validation and review by 10 experts; cases were kept only when at least 90% of votes judged them valid, and approximately 86% of candidates survived.
Eight representative LLMs were evaluated using several complementary measures. Reliability was summarized by MRS, along with full-, partial- and no-refusal rates; robustness was summarized by ASR and RAD. Robustness was assessed only for questions each model had answered correctly before perturbation, so the comparison concerned preservation of an existing correct answer.
Conflicting evidence was the hardest case
One of the sharpest gaps involved conflicting evidence. Conflict-based operators averaged an MRS of 0.46%, compared with 16.49% for information-deletion operators. In the benchmark, evidence-grounded abstention was much less common when available information pointed in incompatible directions than when a required piece was simply absent.
No-refusal was the default under every reliability operator, and silent unsupported answers made up 74.8% of observed failure forms. Broken down by failure stage, schema/evidence binding accounted for 27.1%, schema ambiguity detection for 19.6%, value-sufficiency checking for 18.4% and evidence-consistency checking for 16.7%. Multi-step dependency propagation accounted for 10.3%, input-completeness checking for 4.5% and structural-context recovery for 3.3%.
The model order was broadly consistent across the two source datasets, with a Spearman correlation of 0.81 and p = 0.014. But the MRS results were dataset-sensitive: the reported MRS drop was 8.31 points for Qwen3-30B-A3B and 3.98 points for Claude-Sonnet-5.
A different ranking emerged under table changes
Robustness told a different story. Claude-Sonnet-5 had the best average robustness, while Qwen3-30B-A3B was consistently the least robust model. The operator labeled NRI ranked first for every evaluated model. CSR ranked second for seven of eight models but was more model-selective; ROS, HSS and COS were comparatively weak.
Difficulty showed a positive association with ASR, although the relationship was not monotonic. The reported L3-to-L0 effectiveness ratio was 2.82, and the Spearman correlation between difficulty and ASR was 0.59, with p = 0.044. All eight models had higher ASR at L3 than at L0.
What the benchmark does not settle
The authors recommend making evidence-boundary checking an intrinsic stopping condition: an analysis system should check whether the available table supports the requested answer before continuing. They also recommend training models to remain consistent across equivalent structured representations, treating evidence-boundary checking and representation-invariant reasoning as distinct capabilities.
The scope is narrower than a real-world reliability verdict. This was a constructed benchmark based on two source datasets, controlled perturbations and eight evaluated LLMs, with 2,340 accepted perturbations over 19 operators. It does not establish that any tested model is reliable in real-world deployment, that the measures predict user trust or operational risk, or that the results generalize to every LLM, dataset, table format or analysis workflow. The document is a preprint identified as arXiv:2608.24145v1 [cs.CL], dated 25 Aug 2026.
Paper data and sources
Original title: TrustDABench: Benchmarking Reliability and Robustness of LLMs for Structured Data Analysis
Authors: Boshen Shi, Yize Liu, Chen Zhao et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-25
DOI: Not available
Original paper · Full text