Language models were much better at spotting defined-term problems than at finding contract errors involving party references or contextual capitalization in a new benchmark.
The arXiv preprint, version 1 dated 20 Aug 2026, introduces ContractScrub, a benchmark that tests whether large language models can find and locate drafting defects during final contract scrubbing. It covers 44 English-language contracts, 3,014 annotated tasks and nine error categories, with 10 models evaluated.
The gap between easy and hard checks
Recall — the share of annotated issues a model found — was the primary measure, with precision and F1 also reported. GPT-5.5 had the highest listed recall, at 0.750. Gemini 3.1 Pro had the highest listed F1 score, at 0.655, compared with 0.632 for GPT-5.5.
Mean recall was 0.835 for Defined Terms, 0.689 for Terms Defined Multiple Times and 0.781 for Unused Defined Terms. It fell to 0.362 for Incorrect Party References, 0.427 for Incorrect Capitalization in Context and 0.351 for Undefined Capitalized Terms. These are pooled benchmark summaries; no uncertainty intervals are reported for the category means.
Built from lawyer-reviewed contracts
The benchmark was constructed by selecting and reviewing source contracts, annotating existing issues and inserting additional issues to create the final reference answers. Its schema was designed by licensed attorneys, and nine lawyers carried out further annotation and dataset creation; each had at least eight years of practice, while eight had more than 10 years and six had more than 15 years.
Models were prompted in separate, category-specific instances and had to return a single JSON object with one key for each category.
Longer references proved harder to follow
In an analysis of Incorrect Section References, recall generally declined as the distance between a mistaken reference and the section it referred to increased. The trend was most pronounced beyond 10,000 characters, roughly five to six pages, although distances were manually annotated for only half of the relevant gold items.
The reasoning ablation reported average-recall differences of 0.090 for Claude Opus 4.7 and 0.107 for GPT-5.5, with the largest differences reported for Unused Defined Terms and Uncapitalized Defined Terms. The comparison involved only two models and was not randomized, so it does not establish that the reasoning setting caused the differences.
GPT-5.5 cost $1.38 per contract and took nearly nine minutes, while Gemini 3.1 Pro returned within 90 seconds. These are benchmark inference costs and runtimes, not a real-world cost-effectiveness analysis.
A benchmark, not a safety guarantee
The authors interpret the results as showing that strength on isolated general capabilities does not guarantee reliable performance when several abilities must be combined in a professional workflow. They argue for narrowly targeted, realistic benchmarks and view expert-assisted integration as more plausible than full automation at current performance levels.
The results do not show that current models can safely automate contract scrubbing, replace lawyers or improve legal outcomes and productivity in practice. The benchmark measures performance against known expert annotations, so it cannot assess defects that were absent from those annotations.
The benchmark covers only 44 contracts in English, with documents approximately 10 to 15 pages long. The authors say this limits how confidently the results can be extended to other languages, legal traditions, deal types and document lengths. They also caution that requiring structured JSON may depress scores or mix contract-review ability with instruction-following ability.
Appendix E says the dataset will be made publicly available upon acceptance.
Paper data and sources
Original title: ContractScrub: A benchmark for final review of legal contracts
Authors: Yejin Bang, Kirsty Fielding, Brandan Oliver et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text