An arXiv version 1 preprint reports a 3.09-point accuracy gap in one language-model reasoning condition between African American Vernacular English (AAVE) and Standard American English (SAE), even though paired texts were highly similar in meaning. The analysis traces the comparison across tokenization, pre-training, post-training and inference.
Closely matched meaning, different model results
Researchers assessed semantic equivalence with cosine similarity, a measure of how closely two numerical representations align, and compared the results with character-perturbation and translation controls. All five MULTI-VALUE dialect pairs had similarities above 0.98 and exceeded those control baselines, while similarity for the SAE-AAVE pairs in PARALLEL AAVE was 0.92.
The parallel-text data included 2,019 PARALLEL AAVE pairs and 429 MULTI-VALUE pairs covering SAE, AAVE, Appalachian, Chicano, Indian and Singapore conditions.
The study also examined a reconstructed REDIAL reasoning benchmark containing 1,200 items. In its GPT CoT condition, accuracy was 91.1 for SAE and 88.0 for AAVE, a difference of -3.09 points.
The gap was not confined to tokenization
Across BPE, Unigram and WordPiece tokenizers, AAVE had higher token fertility than SAE. Fertility is the number of tokens used per word; the average gap was 0.07 tokens per word.
Forced character tokenization at inference did not systematically reduce accuracy gaps across nine models: five gaps widened and four narrowed. AAVE input entropy was higher in 93.0% of paired samples under both tokenizations, with both p-values below 0.001.
Dialect information also remained highly decodable from answer-step hidden states, the model's internal representations. Logistic-regression accuracy was above 99% with canonical tokenization and above 86% with character tokenization, compared with a 50% chance baseline.
Training signals and rewards varied by comparison
In the pre-training analysis, matched SAE-AAVE documents were associated with lower gradient similarity than unrelated SAE documents, with a mean z-score of -2.64. Across all nine models, AAVE also had approximately 0.56 higher cross-entropy loss, a measure of prediction error.
Across 10,800 pooled observations, the correlation between gradient similarity and correctness was negligible (r = -0.013; p = 0.19).
Sample-level reward preferences varied by task in an analysis of ten reward models from three providers. The pooled gaps leaned toward SAE for Algorithm (+0.43) and Math (+0.04), and toward AAVE for Logic (-0.08) and Planning (-0.14).
For one AI2 Llama 8B reward model, the reported gap was +0.41 in the base model and -0.19 after instruction tuning. At the token level, dialect-exclusive tokens received higher scores than SAE-exclusive tokens; the pooled raw gap was -0.55, with a within-reward-model standardized effect of d = -0.27.
Similar internal profiles, unequal task results
At inference, the tested dialect, noise and translation transformations had comparable hidden-state profiles. AAVE was associated with a 3.09-point accuracy loss, compared with 0.04 to 1.23 points for the comparison transformations, while generation cross-entropy changed by less than 0.03 nats.
The findings remain tied to tested systems
The authors interpret the combined evidence as showing that the dialect tax is encoded and accumulated at every stage of the language-modeling pipeline, and that individual interventions do not resolve the broader gap.
The study is limited by heterogeneous and partly synthetic corpora, restricted model coverage, a lack of model-training ablations, compute limits on multiple random seeds and the inability to verify behavior in closed systems. The findings are therefore limited to the datasets, models and conditions tested.
Paper data and sources
Original title: The Dialect Tax: Dialectal Biases Persist throughout the Language Modeling Pipeline
Authors: Elle
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-24
DOI: Not available
Original paper · Full text