An arXiv preprint reports uneven gains from a token-reweighting method that gives extra emphasis to tokens a language-model student finds surprising. Called Surprise-aware Reweighting, or SuRe, it had higher reported scores than vanilla on-policy distillation on several math measures at both tested student sizes, but lower scores on both AIME25 measures at the larger size. The document is arXiv version 2, dated 27 Aug 2026.
The paper studies on-policy distillation, or OPD, in which a student supplies the sampled tokens and learns against outputs from a frozen teacher. Its loss uses a sampled-token K2 estimator of reverse KL, comparing the teacher’s and student’s probability assignments on the tokens the student generated. During backpropagation, the sampled token and teacher outputs are held fixed, and the objective has no separate policy-gradient term.
The clue was in the update budget
The central analytic result concerns the student-logit gradient—the signal that changes the scores assigned to possible next-token choices. Its signed direction is set by the teacher–student gap, so it can raise or suppress the sampled token. Its total absolute size, the L1 norm, factorizes into the absolute gap and a student-side term of 1 minus the student’s probability. The identity is for the sampled-token logit gradient at a fixed context, not the full parameter update.
A separate diagnostic used 1.18 million tokens from step 55 of vanilla OPD. Lower-student-probability bins carried a disproportionate share of the summed gradient norms and were enriched for large teacher–student gaps. The authors present that as an empirical association, not as evidence that the student-side factor alone caused the concentration.
Another analysis pooled 1.84 million response tokens to examine endpoint shifts. Only 8.5% of OPD-rollout tokens and 7.1% of Base-rollout tokens had absolute shifts above 1 log-probability unit; Base rollouts showed a negative tail and OPD rollouts a positive tail. That endpoint statistic is different from the teacher–student residual used in K2.
On final OPD rollouts, the absolute teacher–student residual—a measure of how far the two models’ signals were apart—ranked tokens carrying gradient mass more effectively than JSD or entropy. The top 5% by residual covered 54.1% of summed gradient norms, and the top 10% covered 74.6%. JSD covered 28.4% and 47.3%, while entropy covered 23.7% and 42.4% at those cutoffs. The diagnostic used a top-50 approximation and reported no inferential comparison.
A small intervention, mixed results
SuRe uses the token-level finding as a training intervention. It applies a detached, bounded per-token multiplier that up-weights tokens surprising to the student while leaving the baseline teacher–student gap factor unchanged. An alpha value of 0 recovers vanilla OPD. Because the main weights are not normalized, the intervention also changes the overall loss scale.
The computational experiments used a frozen Qwen3-8B teacher and Qwen3-1.7B-Base and Qwen3-4B-Base students. Training used the 57K hard split of DeepMath, seed 42, 32 H20 GPUs and two epochs.
At the 1.7B student scale, SuRe’s AIME24 pass@8 was 23.33%, versus 16.67% for vanilla OPD. On AMC23, its avg@8 was 43.12% versus 39.38%, and pass@8 was 75.00% versus 67.50%. On MATH-500, avg@4 was 67.65% versus 66.55%, while pass@4 was 80.80% for both methods.
At 4B, SuRe was higher on AIME24 pass@8, at 36.67% versus 30.00%; AMC23 avg@8, at 58.44% versus 56.56%; AMC23 pass@8, at 90.00% versus 85.00%; and MATH-500 pass@4, at 88.40% versus 87.20%. On AIME25, however, avg@8 was 14.17% versus 17.08%, and pass@8 was 36.67% versus 40.00%.
Selected out-of-distribution tests were broadly comparable for OPD and SuRe rather than uniformly better than Base; at 1.7B, both were slightly below Base on MMLU-Pro. In the Qwen3-1.7B training trace, SuRe had a higher early gradient norm while actor entropy stayed similar to vanilla OPD. Since the unnormalized weights have a mean above one, the larger norm alone cannot distinguish surprise alignment from a larger effective update scale.
What remains unresolved
An AMC23 strength sweep on the 1.7B student found pass@k rising from alpha = 0 to alpha = 1.0 at small sampling budgets; alpha = 1.0 was best for k = 1, 2 and 4, while alpha = 2.0 weakened the small-k gains. The authors call the pattern empirically non-monotonic and say it does not identify a mechanism.
Controls on MATH-500 complicated the interpretation. Mean-normalized SuRe had avg@4 of 69.20%, versus 66.55% for vanilla OPD. Exact-shuffled, exact rank-reversed and mean-normalized uplift-only weighting scored 67.95%, 67.00% and 68.40%, respectively. Because exact-shuffled also exceeded OPD, the experiment did not isolate exact surprise assignment from generic non-uniform weighting.
A limited second-seed check found a similar gap on MATH-500 avg@4: at seed 42, OPD scored 66.55% and original SuRe 67.65%; at seed 43, the figures were 66.40% and 67.50%. The check was restricted to MATH-500 controls.
The authors’ evidence remains narrow. It focuses on sampled-token reverse-KL OPD, not full-vocabulary distillation or Jensen–Shannon optimization; the experiments were mainly on mathematical reasoning with limited-length traces; and only a limited set of models and OPD configurations was tested, without larger parameter scales.
Those limits matter for the headline result. The benchmark comparisons are nonrandomized and lack reported uncertainty estimates, and the controls do not show whether exact surprise assignment is responsible for the benchmark differences. The preprint therefore offers a diagnostic and an empirical reweighting test, but not evidence that SuRe is uniformly better or that its mechanism is settled.
Paper data and sources
Original title: A Token-Level Analysis of Sampled-Token Reverse-KL On-Policy Distillation
Authors: Bing Shao, Jiazheng Zhang, Long Ma et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text