A sentence-level negative log-likelihood score, called Sentence NLL, may offer a fairer way to compare language models across languages. It scores semantically equivalent sentences as whole units rather than normalizing by how many characters, bytes or tokens a translation uses. In a preprint analysis, five other commonly used scores tracked those surface counts strongly, while Sentence NLL did not show a statistically significant link to them.
The result does not make Sentence NLL an unbiased or definitive measure of model performance. The authors describe it as the least biased of the commonly used intrinsic metrics they examined, while noting that it can still reflect differences in information density, or how much content a language expresses in a given sentence.
Testing the comparison problem
The study examined whether scores based on semantically equivalent parallel content could support more consistent cross-language rankings than measures normalized by tokens, bytes or characters. The researchers first used controlled monolingual language models trained on parallel data, then checked whether the pattern held in multilingual large language models.
The main controlled sample contained 50 GPT2-small monolingual models. They were trained across 10 languages, with five vocabulary-size settings, allowing the analysis to compare model scores while varying a basic feature of the model vocabulary.
A separate scaling sample contained six GPT2-medium models trained on Chinese, English and Arabic. Those models used either a 32k or 65k vocabulary, giving the researchers two vocabulary-size settings at a larger model scale.
For evaluation, the analysis used FLORES-200, including a 40,000-sentence in-domain test split, and a sample of 1,000 sentences from PUD. It also examined translated material and multilingual-model results, with the evidence drawn from several parallel-data and translation settings rather than from human participants.
Why the choice of score changed the picture
The researchers compared several intrinsic metrics, including bits per character, bits per byte, perplexity, mean reciprocal rank, Token NLL and Sentence NLL. They used Spearman’s rank correlation, a measure of whether two rankings move together, to test whether a model’s score was associated with character count, byte count or corpus token count.
The associations were strong for the five normalized metrics. Bits per character had a correlation of -0.81 with character count, with a p-value of 0.0082. Bits per byte correlated -0.94 with byte count, perplexity correlated -0.93 with corpus token count, mean reciprocal rank correlated 0.89 with that count, and Token NLL correlated -0.93; each of the last four results had a reported p-value below 0.001 except mean reciprocal rank, which was reported as 0.001.
Sentence NLL showed weaker and nonsignificant correlations: -0.38 with characters, with a p-value of 0.28; 0.14 with bytes, with a p-value of 0.71; and -0.19 with corpus token count, with a p-value of 0.19. In plain terms, the score was less closely aligned with the surface units that can differ because languages use different writing systems, spacing or tokenization choices.
The authors therefore argue that Sentence NLL over semantically equivalent parallel sentences is the most promising option among the commonly used intrinsic measures in the analysis. Its appeal is that the comparison is made across matched content, while the score is not divided into units whose counts may vary for reasons unrelated to the underlying meaning.
A narrower claim than a universal benchmark
The evidence supports a more careful comparison tool, not a universal answer to the question of which language model is best. The study does not show that Sentence NLL is free of confounds, that it measures language understanding or user-perceived quality, or that downstream benchmarks are automatically comparable across languages.
The main evaluation covered 10 languages because the available parallel data were limited. The translation-alternative experiment focused on English-German pairs, and the approach depends on high-quality parallel sentences that many languages do not have.
That dependence matters because the method compares semantically equivalent material. If the available translations are uneven, or if languages differ in intrinsic information density, the resulting score may capture those properties as well as what the model has learned. The authors leave open how well the framework will work for languages without reliable parallel data and for linguistic phenomena whose complexity differs across languages.
The work is an arXiv preprint dated 25 August 2026, and journal publication was not reported in the supplied record. Its central contribution is a warning about apparently precise cross-language rankings: before treating a numerical gap as a model difference, evaluators may need to ask how much of that gap comes from the units used to score the text.
Paper data and sources
Original title: Apples to Apples? Towards Comparable Crosslingual Language Model Evaluation
Authors: Xiulin Yang, Ethan Gotlieb Wilcox, Catherine Arnett
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-25
DOI: Not available
Original paper · Full text