A preprint audit of language-model watermarking found that apparent fairness across languages can depend heavily on the model regime and the cutoff used to classify text as watermarked. In the study’s instruction-tuned AYA panel, the mean true-positive rate (TPR)—the share of watermarked outputs detected at a chosen threshold—was below 0.7 in 16 of 18 cells. Yet AUC, a threshold-independent measure of how well scores separate watermarked and unwatermarked text, remained above chance in 197 of 198 cell-language entries. The pattern suggests that calibration problems, rather than a missing signal alone, may explain much of the collapse.
The paper’s central warning is about measurement: a watermark can look ineffective under one shared threshold while its scores still retain some separation between the two groups. The audit is designed to show how language, watermarking scheme, generator, regime and threshold choice shape cross-lingual disparities, not to identify one scheme that is best in every setting.
An audit built around matched outputs
The researchers evaluated every cell in a grid covering 11 languages, six watermarking schemes, three open-weight model families and two generation regimes. Each cell contained 500 matched pairs: one watermarked and one unwatermarked output produced from the same prompt.
For each cell, the comparison used 500 watermarked and 500 unwatermarked outputs, with shared prompts and seeds. Generation used a fixed temperature of 0.7, a maximum of 200 new tokens and a minimum of 100.
The framework compared a global empirical false-positive-rate threshold with thresholds recalibrated separately for each language. It also used the threshold-independent AUC check, three automated quality measures—MAUVE, rescaled BERTScore F1 and reference-perplexity preservation—and a disparity calculation that separated variation within and between typological families, or structural groupings of languages.
The threshold changed the verdict
The base-model results were much stronger. In the base-FLORES panel, four of the six schemes—KGW, SynthID, EXPEdit and DIP—had mean TPR above 0.93 in every generator cell, and no per-language minimum fell below 0.798.
Two apparent weak spots showed why the threshold matters. Unigram-Gemma had a mean AUC of 0.986, with a lowest language value of 0.910, while XSIR-Qwen had a mean AUC of 0.978. When thresholds were recalibrated by language, mean TPR was 0.799 and 0.794, respectively; the reported absolute gaps reached 0.799 and 0.579.
Across the detection results, the measured disparity was concentrated mainly between structural language groups. Among cells with measurable disparity, between-family shares ranged from 61.3% to 99.9%, with a median above 90%. That locates where the gaps clustered, but it does not show that typological family itself caused them.
Detection came with a quality trade-off
Detection was only half the picture. In the base regime, EXPEdit and SynthID ranked lowest under all three quality measures. For EXPEdit-Gemma, MAUVE ranged from 0.01 to 0.06 and perplexity preservation from 0.08 to 0.21; SynthID-Gemma ranged from 0.03 to 0.42 and 0.19 to 0.37, respectively.
The instruction-tuned panel produced a different overall picture: MAUVE was above 0.95 for KGW, Unigram, DIP and XSIR, while perplexity preservation was above 0.90 for DIP and XSIR and above 0.88 for KGW and Unigram. But extreme cases remained: Arabic MAUVE was 0.07 for SynthID-Mistral and 0.02 for EXPEdit-Mistral.
The metrics also disagreed about which schemes looked best. Across the full grid, no pair of quality paradigms cleared a Spearman rank correlation of 0.6, a measure of how similarly they ordered the results. Mean correlations were 0.58 for MAUVE and BERTScore, 0.38 for MAUVE and perplexity, and 0.37 for BERTScore and perplexity.
Quality disparity showed the same broad structure as detection: the typological partition accounted for at least 70% of disparity in 32 of 36 scheme-paradigm-regime cells, with a median between-family share of about 85%.
One summary score could hide opposite outcomes
The researchers therefore caution against reducing the results to one combined score. The joint detection-quality spread was 0.066 for DIP in the base regime and 0.070 for EXPEdit, even though DIP paired detection with preservation while EXPEdit paired broad detection with poor preservation. XSIR in the instruction regime had a larger spread of 0.159 alongside uniform detection collapse.
A map of the problem, not a final answer
Because this is a preprint based on synthetic generated text, its findings are evidence from the tested evaluation grid, not a universal verdict about multilingual deployment. The audit covered 11 languages and six schemes, while the study’s open questions include more languages, tokenizers, generators, human judgments and operational settings.
The base and instruction-tuned panels also differ in model variant and prompt source, so their contrast cannot by itself isolate the effect of instruction tuning or prompt regime. Nor does the family-level decomposition identify the language grouping itself as the cause of the disparity.
The narrower conclusion is that multilingual watermarking audits need to report thresholds, model regime, quality paradigm and structural grouping together. Changing those choices can change the apparent fairness and quality verdict.
Paper data and sources
Original title: Auditing Cross-Lingual Fairness in Language Model Watermarking
Authors: Alexander Nemecek, Osama Zafar, Debargha Ganguly et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text