An arXiv preprint reports that AutoVerifier reached 93.05 macro accuracy, the average score across four benchmarks, and had the best reported accuracy on each one. In the study's controlled prompt comparison, the combined Prompt + Code version scored 1.76 percentage points above Prompt Only.
The study addresses reference-based binary answer verification and asks whether recurring verifier errors can be converted into rules that remain auditable, editable and reusable.
From recurring errors to code
AutoVerifier records recurring errors as rule cards, validates proposed deterministic logic by replay, and promotes accepted logic into code modules. It sends model-judgment cases to prompt guidance and uses a model-based fallback for abstained cases.
Before benchmark scoring, the construction stage used a separate 5,000-example pool designed to expose recurring verifier errors. Construction agents proposed updates, and the module library, prompt, parser, priority order and scripts were frozen before evaluation.
By Round 10, accepted rules covered 2,109 of the 5,000 construction-monitor examples. Rounds 1 through 5 contributed 1,840 covered examples, while rounds 6 through 10 added 269.
Prompt changes were not the whole story
Macro accuracy was 91.03 before prompt rewriting, 91.29 after rewriting and 93.05 with code modules. The reported local micro-accuracy scores for those three conditions were 90.84, 91.15 and 92.95.
AutoVerifier's margins over the best compared baseline were 0.15 percentage points on VerifyBench, 0.53 on VerifyBench-Hard, 0.22 on SCI-VerifyBench and 0.10 on VerifierBench.
The paired 95% confidence intervals for the code-module gains all stayed above zero. The reported differences were 0.55 points on VerifyBench, with an interval of 0.23 to 0.87; 1.94 on VerifyBench-Hard, with an interval of 1.08 to 2.81; 3.12 on SCI-VerifyBench, with an interval of 2.44 to 3.80; and 1.45 on VerifierBench, with an interval of 1.01 to 1.90.
What the modules handled directly
The code modules made 2,665 direct decisions across the evaluation, corresponding to 32.13% fewer fallback calls. They were credited with correcting 149 prompt-only errors. Direct coverage ranged from 26.65% to 38.04% across the benchmarks, including 78 corrections on SCI-VerifyBench.
The evaluation contained 8,295 examples: 2,000 from VerifyBench, 978 from VerifyBench-Hard, 2,500 from SCI-VerifyBench and 2,817 from VerifierBench. The 5,000-example construction pool was separate from that evaluation total.
The checks behind the result
A generic heuristic control covered 502 examples but introduced 15 regressions. In the archived promotion-check ablation, removing full replay accepted 51 candidates and admitted 17 replay errors; a support-only policy accepted 95 candidates and admitted replay, protected-slice and counterexample failures.
The leakage audit found zero exact question matches, zero exact question-reference matches, zero exact verifier-triple matches and zero fuzzy verifier-triple matches. It retained 84 fuzzy question-reference matches and 0.80% verifier-triple five-gram overlap as diagnostics.
A preprint with a defined scope
A Qwen3-4B fallback diagnostic reported macro accuracy of 89.31 with the fallback alone and 91.62 with code modules, while avoiding 32.13% of fallback calls. The authors describe this as a fallback-replacement diagnostic, not evidence that the rule library is invariant across fallback models.
The work is an arXiv version-one preprint dated 26 August 2026. The paper describes a supplementary package with prompts, anonymized synthetic examples, schemas, hashes, code modules, routing ledgers and aggregate audit records, while raw third-party data remain governed by their original licences and access terms.
Paper data and sources
Original title: AutoVerifier: Residual-Guided Non-Parametric Optimization for Reference-Based Answer Verification
Authors: Zebei Zhao, Zhihao Shi, Minqi Shi
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text