A two-stage approach to judging several candidate answers kept its reported Guarantee Success Rate above the study’s 90% target across the tested scoring functions and datasets. In a matched comparison using GPT-OSS-120B with a KL margin score, the two-stage versions recorded Guarantee Success Rates of 95% to 97% and coverage of 64% to 72%. Matched single-stage results recorded 48% to 79% for Guarantee Success Rate and 50% to 58% for coverage.
Guarantee Success Rate, or GSR, is the share of repeated calibration-and-test splits in which agreement on the cases the system accepts reaches the chosen target. Coverage is the share of test cases the system accepts instead of declining to choose. The single-stage comparator selected directly from the full set of candidates, while the two-stage systems were evaluated in matched comparisons.
The shortlist comes first
The framework breaks the decision into two steps. Stage I identifies a shortlist from the candidate answers. Stage II chooses one answer within that shortlist, but abstains when it cannot certify a choice. Stage I uses EMP as its localization score; Stage II can use EMP, KL or Vote per-candidate scores, paired with either top-1 or margin confidence.
A synthetic simulation examined whether confidence and agreement retained an expected ordering as candidate counts increased. At 20 candidates, direct top-1 agreement lost that ordering, while top-3 agreement remained monotone. Within the top-three shortlist, margin confidence further restored the reported monotonic pattern.
The pattern also appeared in the reported real-data analysis. Ranking loss, a measure of violations in the expected confidence-and-agreement ordering, increased with candidate count for direct single-stage selection. For Qwen2.5-72B on TL;DR, direct ranking loss rose from 0.14 to 0.34, while the loss for Stage I and Stage II stayed at or below 0.08.
What was tested
The empirical review covered four preference benchmarks with human-annotated preferences: TL;DR, Chatbot Arena, HH-RLHF and AlpacaEval. It created approximately 3,000 multi-candidate instances for each benchmark and candidate-count pairing, using sets of five, 10 and 20 candidates.
Each dataset-and-model combination went through 1,000 repeated random calibration-and-test splits, with half of each split used for calibration and half for testing. The study also made matched single-stage comparisons within that evaluation protocol.
In the reported Qwen2.5-72B analysis on TL;DR, single-stage GSR fell from 82% with five candidates to 50% with 20 candidates. Two-stage GSR remained above the 90% threshold across that range. This example is specific to that model and benchmark.
A route for escalating hard cases
The paper also examined three-tier cascade configurations. Across the reported configurations, two-stage cascades achieved GSR of 91% to 95% and covered 77% to 81% of instances. The strongest single model covered 64% to 70%, and it was invoked for fewer than 30% of instances in the cascade results.
Additional checks examined whether the two stages behaved consistently together. In every displayed configuration, the conditional probability of missing the correct shortlist fell as the Stage II threshold increased and was below 1% at λ = 0.8. A component-by-component ablation found lower GSR and coverage whenever a component was removed; the largest displayed drops followed removal of shortlist renormalization and variance normalization.
A conditional result, not a blanket promise
The formal result is a conditional high-probability guarantee. Under the paper’s stated assumptions and its cross-stage monotonicity condition, selective agreement is at least (1 − ε)(1 − α) with probability at least 1 − δ over the Stage II calibration sample. That is a bound for the selective system, not a promise that every accepted decision is correct.
The guarantee requires calibration and test instances to be exchangeable, meaning the sampling setup must allow them to be treated as interchangeable. If the deployment domain diverges from the calibration domain, the finite-sample bounds may become less tight. The composed guarantee also depends on the cross-stage monotonicity condition.
The experiments used publicly available datasets with existing human-preference annotations and collected no new human data.
Where the evidence stands
The document is identified as arXiv:2608.25824v1, dated 26 August 2026. It is a preprint, and the supplied record does not report peer review or acceptance. The authors report support from the NVIDIA Academic Grant Program, a Royal Society Grant and the AIRR Gateway project.
Taken together, the reported results support a limited conclusion: in the tested multi-candidate settings, the shortlist-first variants met the study’s 90% GSR target across the reported scoring functions and datasets. In the GPT-OSS-120B and KL-margin comparison, they also showed 64% to 72% coverage, compared with 50% to 58% for matched single-stage results. The formal guarantee remains tied to its assumptions, including exchangeability and cross-stage monotonicity.
Paper data and sources
Original title: Localize-Then-Decide Guarantees for LLM Judgments
Authors: Xinyu Li, Yi Zhou, Guanqun Cao et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text