Ten frontier language models struggled to spot legally important information missing from incomplete legal questions, according to a new arXiv preprint. On the benchmark’s deficient queries, no model scored above 0.46 on F2, a measure that weights recall—how many missing elements are found—more heavily than precision. Median recall was 0.44, while GPT-5.2 led the reported results with an F2 score of 0.455 and recall of 0.666.
The benchmark focuses on a practical problem: a legal assistant may need details that an underspecified question does not provide. The authors say a safe system should recognize when information is insufficient, identify missing facts that could affect the outcome, ask targeted questions and give interim guidance only under explicit assumptions.
A test built around missing information
The InsufficiencyBench dataset contains 202 items: 58 fully specified base queries and 144 deficient variants. It spans six legal domains and 24 US jurisdictions, and the queries were authored and annotated by two attorneys with more than 20 years of combined experience.
Researchers began with complete queries that had sentence-level legal annotations, then created incomplete versions by removing or minimally revising sentences that supplied tagged material elements. The taxonomy includes eight categories of missing information, including jurisdiction, controlling text, procedural posture, the parties’ status, facts of harm, timing, consideration and the user’s goal.
Too cautious in some cases, too confident in others
The models showed two broad problems. Some hedged indiscriminately, while others answered deficient queries silently under fabricated presumptions. No model both reliably identified and qualified incomplete queries while also responding directly to complete ones.
The trade-off was visible in the comparison between two systems. GPT-5.2 over-flagged 72.4% of complete base queries, while DeepSeek-V4-Pro over-flagged 22.4%. But DeepSeek-V4-Pro answered 63.9% of deficient queries without acknowledging a gap. The pattern is observational evidence from this benchmark and does not establish why the systems behaved that way.
The hardest gaps were procedural
Models were better at noticing missing controlling text than missing procedural posture. Across the 10 models, mean recall was 0.64 for controlling text and 0.09 for procedural posture. The authors caution that these category comparisons are directional because the amount of supporting data varied between categories.
The study also separated the decision to flag a gap from the quality of the explanation that followed. DeepSeek-V4-Pro produced substantive conclusions judged to be fabricated on 30.2% of instances, with a reported safety rate of 0.698. Explanation accuracy clustered between 0.63 and 0.77 across the models; GPT-5.2’s score was 0.763.
A benchmark, not a test of legal practice
The paper is a version-one arXiv preprint dated 20 August 2026. Its evaluation used single-turn responses from 10 models on 202 legal-query items covering six US legal domains and 24 jurisdictions. The authors report that the dataset is limited to common-law contentious matters.
The scoring relied on a fixed language-model judge, whose agreement was moderate and was not validated against human scores. Alternative judges retained the difficulty finding: maximum F2 was 0.371 with Claude-Haiku-4.5 and 0.443 with GLM-5, while their maximum recall scores were 0.538 and 0.573 respectively.
Those limits mean the study does not establish how the systems would perform in multi-turn clarification dialogues, how expert human reviewers would score the answers or whether the results generalize beyond the reported US domains and jurisdictions. It also does not show that any evaluated model is generally safe for legal practice.
Paper data and sources
Original title: InsufficiencyBench: Evaluating LLM legal advice on underspecified user queries
Authors: Samuel J. Vincent, Daniel Calloway, Fangyi Yu et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text