A preprint examining 11 open-source automatic speech-recognition (ASR) models found a pattern in VoxPopuli scores: the six models with the best word error rates (WER), 5.4% to 5.8%, had accept-ref values from 0.18 to 0.30. Models with WER of 6.5% or higher were at or below 0.10.
Accept-ref is the rate at which a model chooses the benchmark's written reference in tests where the audio does not uniquely determine the words. The study pairs it with masked-number recovery and orthographic switching, tests that ask whether a model will fill in a hidden number or adopt a benchmark-linked spelling or spacing convention.
The public score did not tell the whole story
On the masked-number test, several models had higher accept-ref rates on public benchmarks than on held-out fresh sets. The top public-benchmark models reached approximately 0.40.
Models also switched written conventions in ways that tracked the corpus or subpopulation being tested. Six of 11 exceeded the 0.5 honorific switch-rate baseline, and eight of 11 exceeded 0.5 for archaic spacing.
The probes put audio and text at odds
The paper defines three reusable behavioral probes: reference disagreement, masked-number recovery and orthographic switching. Together, they test whether a model follows the audio when the reference is ambiguous, restores a hidden number, or changes spelling and spacing to match a benchmark-linked convention.
The evaluation drew on VoxPopuli English and, where applicable, LibriSpeech, along with a private held-out DaiKon set of 450 conversational clips. In the Hugging Face VoxPopuli version analyzed, 40% of test speakers also appeared in training.
A model-consensus procedure flagged 1,113 edits on 745 VoxPopuli test clips: 586 substitutions, 441 deletions and 86 insertions. Ninety-three percent also appeared in human annotations, though the panel itself was model-based.
The behavior depended on the voice
The benchmark-linked behavior depended on the voice around the target words. Original recordings and clones of evaluation-set speakers produced stronger benchmark-like output than generic voices or many fresh speakers from the same domain.
On fresh-clone audio, a controlled splicing test compared benchmark-donor and control-donor segments. The accept-ref contrast was 0.096 for Phi-4, with a reported interval of 0.074 to 0.120, and 0.093 for Canary, with an interval of 0.073 to 0.115. The contrast was also positive for Higgs and Cohere; Granite's 0.010 contrast was not significant.
Clues appeared inside the models
Activation patching, which replaces one run's internal signals with another's, pointed to both the sound-processing encoder and the later transcript-generating decoder. Replacing target-frame signals often restored the audio-supported output for insertions, but was less effective for deletions and substitutions, which were more consistent with a later decoder override.
The learned internal direction had a low-rank structure: a one-component version recovered 65% to 80% of the full-direction effect for the first three models. It was reported to steer four of the six models with elevated behavior; when researchers induced it, accept-ref changed from 0.02 to 0.07 for Cohere, 0.01 to 0.05 for Parakeet, 0.01 to 0.11 for Canary and 0.02 to 0.04 for Granite. Ablation, or removal of the direction, corresponded to an 82% to 92% fall for Cohere, Canary and Parakeet.
Noise also separated the models. At 10 dB of additive noise, accept-ref fell from 0.22 to 0.03 for Canary-Qwen, from 0.17 to 0.04 for Parakeet and from 0.20 to 0.07 for Higgs; Cohere remained at 0.29 under noise and 0.26 under reverberation.
A warning about benchmark scores
The authors interpret the results as evidence that some high-scoring open ASR models use narrow acoustic cues associated with benchmark recordings to activate benchmark-optimized transcription policies. In that reading, a public score could look better without showing better general-purpose transcription, but the study does not establish that every high-scoring model is affected or that the behavior harms all real-world audio.
The evidence covers only 11 open-source models and constructed underdetermined-audio probes across the tested corpora. The analyzed VoxPopuli version had 40% test-speaker overlap with training, probe-specific sample sizes and conditions were uneven, and the study does not identify the training-time cause or directly quantify real-world WER improvement or degradation. It is an arXiv preprint, version one, dated 20 August 2026.
Paper data and sources
Original title: Towards Quantifying Benchmark Optimization in ASR Models
Authors: Theo Lebryk, David Ayllon, Alice Baird et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text