A preprint under review argues that some apparent gains in language-model self-training may be measurement artifacts. In its most striking test, an untrained frozen model appeared to learn and corrupt answers when judged by a single decode.
The same audit found a clear difference between external distillation and the tested self-training methods: the teacher model reached more low-base problems, meaning problems with little baseline evidence of a solution.
When the model did not change
Two greedy evaluations of an untrained model on MATH-500 produced 9 apparent corruptions and 6 apparent learnings, for a corruption-to-learning ratio of 1.5. Using solve-rate estimates from repeated samples instead, the audit reported 1 learning and 0 corruptions on the same samples.
The conventional expansion test also produced a nonzero result when applied to frozen controls. With a threshold of 1 correct sample, the frozen comparison rate was 0.280. Raising the threshold to 2 reduced the pooled frozen rate to 0.058, but did not reduce it to zero.
A replacement version using held-out frozen nulls returned zero detections across all 11 held-out replicates. The contrast is the paper’s central warning: an expansion statistic needs a background error rate measured on the same kind of task.
The teacher reached more low-base problems
On a matched problem ladder, external distillation reached 8 to 11 low-base problems per seed. The self-training arms reached 0 to 2. A statistical comparison estimated an additional low-base coefficient of 1.91, with a 95% confidence interval from 1.25 to 2.56 and p<10−8.
The evidence was weaker on the 10 problems with exactly zero pooled baseline rate. Across seeds, the union was 5 problems versus 2, with p=0.35. A separate 2026 low-base comparison was 6 versus 2, with p=0.17. The study therefore did not establish expansion on that subset.
Because the teacher was an external-distillation control, its result does not show that self-training itself produced those gains. It shows that the matched protocol could reach more low-base problems with an external source of training data.
Self-training also produced corruption
On a constructed difficulty band, STaR produced 106 corruption events and majority-vote self-training produced 88. The matched frozen-model floor had median counts of 10 learned events and 8 corrupted events; 58 STaR events and 60 majority-vote events exceeded the frozen solve-rate envelope of 0.234.
The results varied sharply with the training seed. STaR’s corruption-to-learning ratios were 0.70, 0.74 and 0.45 across three seeds. Majority-vote self-training’s were 0.55, 1.53 and 1.33, crossing 1 — the point at which corruptions outnumber learnings.
A separate policy-gradient arm tested on AIME ended in severe truncation on all three seeds. Truncation reached 100% on one seed, 95.6% on another and 76.4% on the third.
A large computational audit of a short training setup
The study comprised 48 runs, 3.44 million sampled generations and 5.30 billion generated tokens. Its primary evaluation sets contained 200 MATH-500 problems, 60 AIME problems and 1,163 problems in a constructed difficulty band.
The main comparison used Qwen3-8B, three rounds of self-training and external distillation as a positive control. Each evaluation problem was sampled 128 times, and the pipeline tracked transitions through a seven-state ledger rather than relying on a single answer.
A power analysis estimated that about 249 events per arm would be needed for 80% power at α=0.05. A 112-problem pilot produced 24 events, leaving some comparisons less decisive than their raw counts might suggest.
What the audit does not settle
The tested training schedule covered three rounds, so the results do not settle what longer schedules or different self-training designs might achieve.
An out-of-domain refusal probe was too variable to support either a safety-loss or no-effect conclusion. Across identical-checkpoint replicates, the changes were −0.015, −0.115 and −0.005; the authors reported a bound of roughly 11 percentage points rather than a null result.
The paper’s practical message is that each transition statistic should be checked against its own design-matched frozen null, using repeated solve-rate estimates, matched controls, multiple training seeds and enough events for the intended statistical power.
The paper reports releasing run-level records, evaluation subsets, the constructed band, noise-floor runs and reduction code. Raw per-sample records total roughly 15 GB and are available on request.
Paper data and sources
Original title: Phantom Gains: Auditing Self-Improvement Against a Measured Null
Authors: Cheng Xu, Nan Yan, Liming Chen, M-Tahar Kechadi
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text