Selected generative language models closely matched human preferences when asked to choose between two French speech-recognition transcripts in a new computational evaluation. GPT-4.1 recorded agreement scores of 94 for cases with complete annotator agreement, 85 for cases with at least 70% agreement and 79 across the full dataset. Qwen3.5-35B scored 92, 83 and 78 across the same groups.
The comparison was narrower when the models were asked to assign qualitative categories to transcript quality. The study reports that, for the pairwise task, generative language models outperformed word error rate, character error rate and the strongest embedding-based semantic metrics. The authors present this as a task-specific result from the HATS benchmark, using the fixed prompts tested in the evaluation.
A benchmark built around disagreement
The study compares encoder and decoder large language model families as tools for evaluating automatic speech-recognition output. It uses the HATS dataset, which contains French ASR hypotheses and human judgments, and measures how closely automated scores align with those judgments.
The researchers divided the judgments into three groups: cases with 100% agreement among annotators, cases with at least 70% agreement and the full dataset. The supplied analysis does not report the number of utterances, candidate hypotheses or annotators, so the reported scores cannot be placed in a sample-size or uncertainty framework.
The two basic lexical measures produced different results on the 100% agreement subset. Word error rate agreed with human judgments 63% of the time, compared with 77% for character error rate. Character error rate was therefore the stronger of the two in that subset, although its size and uncertainty interval were not reported.
Model setup changed the semantic scores
The researchers also tested BERTScore, a similarity measure calculated from language-model embeddings. They computed it from representations taken at every transformer layer of each evaluated encoder and decoder model, then compared the scores with HATS judgments. The layer that worked best was not the same for every model.
Sentence-CamemBERT-large and Qwen3-Embedding-8B were identified as the best-performing embedding models in the analysis. Encoder models were described as comparable to the strongest decoder models despite having fewer parameters. The supplied analysis does not give exact best BERTScore estimates or uncertainty intervals.
A related measure called SemDist was tested across every combination of model, transformer layer and pooling strategy against the HATS annotations. In the last-layer results, the maximum listed score was 90 for Sentence-CamemBERT-Large and 89 for Qwen3-Embedding-8B.
The SemDist results also make model size an unreliable shortcut for selecting an evaluator in this benchmark. Performance did not track size alone, and the reported correlations could change with the chosen layer and pooling strategy.
High agreement did not settle qualitative labels
For the pairwise experiment, each generative model received a one-shot prompt containing one annotated example followed by the two hypotheses it had to compare. The experiments used fixed instruction-style prompts, and the exact templates were provided in the appendix.
In this relative judgment task, the model chose which of two transcripts was better. GPT-4.1's agreement score declined from 94 in the fully agreed subset to 85 in the at-least-70% subset and 79 in the full dataset. Qwen3.5-35B showed a similar pattern, moving from 92 to 83 and then 78.
The picture was less settled when the models assigned qualitative categories instead of choosing a preferred transcript. For GPT-4.1, the reported correlation with the continuous SemDist signal was -0.66 by Spearman's measure and -0.63 by Pearson's measure. The text describes GPT-4.1 as the best overall model in that comparison.
The predicted categories followed the expected semantic ordering, but neighboring classes overlapped substantially, especially the categories labelled useful and bad. HATS did not contain human qualitative labels, so these classifications were compared with Sentence-CamemBERT-large SemDist rather than checked against a separate human labelling task.
What the evaluation does not establish
Taken together, the evaluation reports strong results for selected encoder and decoder representations when their configurations are chosen appropriately, alongside high agreement from generative models making pairwise choices. It also reports that qualitative labels preserved the expected ordering but did not cleanly separate neighboring levels of transcript quality.
The evidence is limited to computational tests of selected models, layers, pooling strategies and fixed prompts on French ASR hypotheses from HATS. The supplied analysis does not report sample sizes, confidence intervals or significance tests for the scores discussed here.
The direct classification result is not a human-validated label system because HATS lacks human qualitative labels. The analysis does not establish a universal best model, layer, pooling strategy or model-size rule beyond this evaluation.
The document is an arXiv version 1 preprint dated 26 August 2026. Whether the pattern extends beyond this French benchmark remains an open question for other languages, ASR domains and human-evaluation datasets.
Paper data and sources
Original title: Generative vs. Encoder Large Language Models for ASR Evaluation: A Comparative Study
Authors: Thibault Bañeras-Roux, Shashi Kumar, Driss Khalil et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text