An offline benchmark replay found that hybrid selection, combining an LLM judge's ranking with answer frequency, reached 70.82% to 70.95% accuracy on identical stored pools of eight candidates. Majority voting reached 63.82% on the same pools, a difference of 7.00 to 7.13 percentage points.
That comparison varied with candidate availability. First-ranked selection was 6.02 percentage points higher than majority voting in a rare-availability regime and 7.87 points higher in an intermediate regime, but 3.22 points lower when correct candidates were common.
Judge reliability varied with candidate supply
The primary ranking analysis covered 15,336 questions, 67,163 pools and 823,988 correct-wrong candidate pairs. Rank AUC, the chance that a correct candidate is placed above a wrong one, uses 0.5 for chance ranking.
Here, pgen means the empirical fraction of correct candidates under the study's adaptive collection protocol, rather than an unconstrained single-call success probability. The availability-reliability curve was below chance at the lowest availability and reached a fitted halfway point at 14.7% pgen (95% CI, 14.3% to 15.1%).
That relationship persisted in a control where every displayed pool contained exactly two correct and six wrong candidates. Across 12,184 questions and 146,208 correct-wrong pairs, the fitted midpoint was 12.2% pgen (95% CI, 11.8% to 12.6%).
The study's oracle ceiling, the best result available within a candidate pool, exceeded actual multi-answer accuracy by 13.13 to 14.38 percentage points. When the common initial answer was wrong but an outnumbered correct candidate was present, the open committee selected the most frequent wrong answer in 73.25% of cases, while the evidence board did so in 59.61%.
The setup changed the comparison
In a medical-protocol comparison, each protocol used 7,350 case runs across 2,450 MedXpertQA questions. Majority voting accuracy was 41.06%, compared with 42.20% for the open committee and 41.89% for the evidence board. The differences versus majority voting were 1.14 percentage points (95% CI, 0.42 to 1.90) and 0.83 percentage points (95% CI, -0.05 to 1.74), respectively.
Relative token use was 4.96 times the single-response baseline for majority voting, 10.73 times for the open committee and 6.07 times for the evidence board.
Across 12,477 matched questions, Mimo no-think and medium-thinking banks had the same mean rank AUC, 0.8493, but different availability-reliability fits. The reported curve-fit scores, R2, were 0.022 for no-think and 0.928 for medium-thinking, whose fitted midpoint was 19.0% pgen.
Larger pools narrowed the gap
The first-ranked comparison was 13.52 percentage points higher than majority voting at k = 2, 0.75 points higher at k = 14 and 0.34 points lower at k = 16. The 95% confidence intervals at k = 14 and k = 16 included zero.
Estimated incremental judging cost was US$0.10 to US$0.39 per 1,000 additional correct answers for pools of six or fewer; the estimate was US$6.28 at k = 14 and economically undefined at k = 16. These figures depend on reconstructed token counts, stated model prices and a 90% cache-hit and 10% cache-miss assumption.
At k = 16, results also differed by Mimo setting. For no-think pools, majority voting accuracy was 20.28% and DeepSeek selection accuracy was 57.73%, a 37.44-point difference (95% CI, 36.37 to 38.49). For medium-thinking pools, the corresponding figures were 61.83% and 64.79%, a 2.96-point difference (95% CI, 2.22 to 3.73).
A replay, not a live system
The selection result came from an offline replay of 81,390 pre-computed pools, each containing eight candidates, drawn from 16,278 benchmark questions. Candidate text, rationales, answer frequencies and stored judge orderings were fixed, apart from a shuffled-order control.
That design measures selection rules on stored candidates; it does not directly test live feedback in which agents repeat, reject, adopt, revise or transform information. The rules were replayed on stored pools rather than prospectively assigned to newly generated live systems, so the comparison does not establish that communication, judging or selection caused the observed differences.
A narrower lesson for system designers
The authors' interpretation is to measure the empirical supply of correct candidates first, evaluate a judge in the task, generator setting and candidate regime where it will operate, and compare pre-specified selection rules with their costs. The evidence supports that evaluation sequence within these benchmark tests, but does not show that the same pattern holds in live systems.
Paper data and sources
Original title: Candidate supply and answer selection shape the value of LLM judging in multi-agent systems
Authors: Jia-Hao Ji, Sijie Li, Jiabei Cheng et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text