Multilingual language models trained on romanized text were reported as the strongest configuration across every evaluated transfer regime, ahead overall of both orthographic text and IPA transcription. The study compared the three input forms in autoregressive multilingual pretraining to see which supported cross-lingual transfer—the ability to carry useful learned behavior between languages. It found that the reported advantage over text widened as model scale increased.
A three-way test across four language pairs
The work covered eight languages organized into four pairs: English-Spanish, Russian-Polish, Hindi-Urdu and Tamil-Malayalam. It compared orthographic text, the usual written form, with IPA, a sound-based transcription, and romanization as input representations.
The controlled pretraining compared the representations at 467 million, 709 million and 1.03 billion parameters while holding architecture, data, vocabulary size and training procedure constant. Each representation used a jointly trained Byte-Level BPE tokenizer, the tool that breaks text into reusable subword pieces, with a 100,000-entry vocabulary. The fixed vocabulary was meant to keep surface form as the controlled difference.
The four bilingual corpora contained approximately 21.7 billion words. After tokenization, the word-matched corpus yielded 50 billion text tokens versus 33 billion IPA or romanized tokens. The paper notes a compute disadvantage for IPA and romanized training under FLOP-matched training, so that difference is an important qualification when reading the comparison. Evaluation used direct prompting and supervised fine-tuning, with baseline improvements assessed by approximate randomization and p < 0.05 treated as significant.
A later switch was less reliable
A separate Text-to-Rom condition tested a different strategy: text-pretrained models were fine-tuned on romanized downstream data. That route showed large regressions on pretraining languages across benchmarks. It helped unseen languages lacking script coverage, but by smaller margins than romanized pretraining, and its unseen-MASSIVE benefit disappeared at the medium and large scales. The result was therefore conditional on the task, scale and whether the language’s script had been covered during pretraining.
Token overlap and sequence length told different stories
Token overlap offered one clue, although it was only a tokenizer-level proxy rather than a direct transfer outcome. Overlap was meaningful mainly for English-Spanish. Romanization raised it for Russian-Polish and Tamil-Malayalam but did not close the Hindi-Urdu gap, while IPA had its highest overlap for Hindi-Urdu.
Sequence length showed another difference. Compared with Latin- and Cyrillic-script languages, Hindi, Urdu, Tamil and Malayalam produced roughly 2.5 to 3.5 times more tokens under text than under romanization. IPA and romanization compressed those languages toward the range of the other languages. The result matters for interpreting training exposure because the same word-matched corpus became 50 billion text tokens but 33 billion IPA or romanized tokens.
The result has clear boundaries
The evidence remains bounded by the study’s design: eight selected languages, three model scales and the specific tokenizers used for each representation. The largest tested scale was 1.03 billion parameters, and the report gives no confidence intervals or pooled effect estimates for its qualitative ranking and scale trend. That leaves open whether the same ordering would hold outside these language pairs or at larger scales.
The paper also discloses a post-submission integer-overflow bug in the BPE tokenizer. It reports patching the accumulator, verifying output below the threshold and retraining affected tokenizers and models. The disclosure is relevant because tokenizer behavior was part of the controlled comparison.
The document is identified as arXiv:2608.25904v1, dated 26 Aug 2026, and is a preprint. The authors state that code and datasets will be released upon acceptance.
Paper data and sources
Original title: One Form to Transfer Them All: Pretraining Multilingual Language Models Beyond Native Orthography
Authors: Muge Zhang, Aaron Jencks, Krishna Badikela et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text