A Turkish language-model preprint reports a sharp jump in passage retrieval for an encoder trained with a CLM-to-MLM curriculum rather than masked-language training alone. On Turkish MS MARCO, the curriculum scored 10.86% on R@1, compared with 2.96% for MLM-only. Its R@10 score was 26.34%, versus 9.92%, and its mean reciprocal rank, another retrieval score, was 16.23%, versus 5.59%.
The comparison held the data pipeline, architecture and total number of steps constant. MLM-only ran for 10,000 pure-MLM steps; the curriculum used about 2,500 CLM steps followed by 75% MLM on an approximately 9.4-billion-token subset. The ablation was single-seeded, so it tests one run of this recipe rather than showing that the pattern is stable across repeated starts.
The accompanying geometry check pointed in the same direction. The curriculum run had a first-component variance share of 0.1187, against 0.2810 for MLM-only, and a lower raw mean cosine of 0.9595 versus 0.9731. Its STS Spearman score was 0.5026 versus 0.4833. The paper describes that combination as less anisotropic, or less tightly concentrated, embedding geometry; the comparison came from the same single-seed ablation.
A stronger signal on TrGLUE, but not everywhere
The full training plan was far larger than the ablation: 237.3 billion tokens, with CLM making up 16.6% of the plan alongside MLM. That scale matters when reading the headline result, because the paper does not establish that the retrieval advantage seen in the short ablation persists through full pretraining.
A separate experiment compared two branched annealing settings over the same listed lower-half interval: 10.1 billion tokens from steps 5,001 to 9,810. The anneal branch used an 8,192-token context, while anneal1k used 1,024 tokens, the amount of text available to the model in each training example.
Across five paired seeds, anneal1k's TrGLUE score was 0.49 points higher on average than anneal's, with a standard deviation of 0.26. The paired test returned t = 4.28 with 4 degrees of freedom and a p value of about 0.013; all five seed comparisons favored anneal1k. It was also 0.75 points above the model-soup alternative at about 4.3% additional cost.
On the overall TrGLUE benchmark, Mogan-anneal1k averaged 78.41 points, with a standard deviation of 0.32, the best result among the compared Turkish ModernBERT models. It remained below BERTurk at 79.87 with a standard deviation of 0.23, while TabiBERT scored 77.83 with a standard deviation of 0.57 and ModernBERT-TR 77.64 with a standard deviation of 0.37. The averages used five official seeds, and task-level variation differed substantially.
Tokenizers and embeddings add another layer
The paper's tokenizer and data work adds another strand to the story. Its quality pipeline combined Turkish FineWeb2, recent Common Crawl months, and printed and institutional text. Turkish BERT's quality decisions were distilled into a fastText classifier with 94.4% agreement and an approximately 90-fold speed-up. The reported tokenizer had a 50,048-token vocabulary and outperformed all compared Turkish tokenizers on compression and fertility across two independent test sets.
Code handling was another reported strength at the tokenizer level. On a corpus spanning 18 languages and 540 files, the final Mogan tokenizer recorded code compression of 2.790 and 100% lossless roundtrip, compared with 2.196 and 49% for TabiBERT. That result shows what the tokenizer test measured; it does not, by itself, establish that the tokenizer caused downstream model differences.
On TabiBench, MoganBert-TR scored 77.73 overall, ranked second among monolingual encoders and led the code-retrieval category with 60.57, 3.62 points above TabiBERT. Its overall score sat 0.19 points below ModernBERT-TR's 77.92 and 0.15 points above TabiBERT's 77.58. Those margins should not be treated as firm rankings: the work used one run for its model, reference systems were not remeasured in the same environment, and the scores were not backed by significance testing.
A separate embedding model, MoganBert-Embed, was produced through teacher distillation and multi-signal contrastive fine-tuning. It ranked first among student models on MTEB(Turkish), with an overall score of 68.30, and reached 99.5% of its 7.57-billion-parameter teacher's score with a backbone 51 times smaller. In plain terms, the reported student came close to the much larger teacher on that benchmark, but the result remains tied to a single teacher and a final model soup whose weights were selected using MTEB.
An intermediate check after Phase 1 of the embedding work also reported a large change in geometry: raw mean cosine fell from 0.9841 at the start to 0.0851, effective rank rose from 117.9 to 157.2, and zero-shot information-retrieval NDCG@10 rose from 0.2361 to 0.5927. These figures describe the reported Phase 1 checkpoint rather than the final benchmark ranking.
Why the result needs restraint
The strongest result therefore has a narrow evidential base. The objective ablation used one seed, short 10,000-step runs and one retrieval dataset; two probe tasks were saturated. The paper does not show that the CLM-to-MLM advantage persists at full-pretraining scale, so the retrieval numbers are best read as a result from a controlled setup, not a settled property of the model.
The same caution applies to the broader comparisons. TabiBench was single-seed and used externally reported reference scores, while the embedding result relied on one teacher and MTEB-selected soup weights. Those choices make close benchmark rankings provisional.
The paper says its model weights, tokenizer, embedding model and evaluation code will be released openly. It will share data-pipeline code and filtering decisions, but licensing restrictions prevent release of the full corpus. The document is an arXiv preprint, version v1, dated 26 August 2026; no funding source is reported in the paper text or metadata.
Paper data and sources
Original title: MoganBert-TR: A Turkish Encoder Foundation Model Trained from Scratch with a CLM-to-MLM Curriculum
Authors: Furkan Yilmaz, Habibe Aleyna Tasdemir, Muhammed Faruk Gozay
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text