A preprint reports that OPMD, a language-model distillation method, had the highest average accuracy among the evaluated distillation baselines in all three main teacher-student settings. On the AMC mathematics benchmark, its reported gains over the output-level baseline, OPD, were 3.76, 7.04 and 2.66 percentage points across those settings.
The method works inside the student
The study asks whether teacher-derived intermediate information can be transferred according to its effect on a student's subsequent computation. Put more simply, it looks beyond the final answer to the work happening inside the student as it produces a response.
OPMD uses two stages. First, it constructs a student-compatible teacher. It then trains the native student to reproduce the resulting latent states, meaning its internal numerical representations, without external teacher intervention.
The gains were not confined to one test
The main mathematics experiments used DAPO-Math-17K for training and AMC, AIME 2024 and AIME 2025 for evaluation. The main score, called avg@16, was the mean final-answer accuracy across 16 sampled responses per problem. Checkpoint comparisons used 40 AMC problems and 30 problems from each AIME benchmark, with a generation temperature of 0.7 and a top-p sampling cutoff of 0.95.
With RL-Math as teacher and Qwen3-1.7B-Base as student, OPMD improved AIME 2024 and AIME 2025 by 4.37 and 3.95 points. The teacher swap produced a different result for OPD: changing from Qwen3-4B to RL-Math lowered its AMC accuracy from 35.31% to 33.12%, while OPMD's rose from 39.07% to 40.16%.
The paper also reports a response-length advantage. At every evaluated update count, OPMD had higher accuracy with shorter responses. Even at 256 updates, it exceeded OPD's accuracy at 512 and 768 updates while generating fewer tokens.
The pattern extended to a two-teacher experiment. Relative to M-OPD, OPMD improved AMC, HumanEval+ and MBPP+ by 4.31, 0.61 and 6.61 percentage points, respectively. The reported math gain did not come at the expense of code accuracy in that comparison.
A separate cross-family comparison was more mixed. SFT+MD scored 25.31% on AMC versus 22.50% for SFT, and 1.81% versus 1.46% on AIME 2025. On AIME 2024, however, it scored 3.88% versus 4.33% for SFT, without relying on a logit-level distillation objective.
The evidence stops short of a general verdict
In a selected layer-patching analysis, putting blocks 1-7 or 8-14 into the raw student raised accuracy from 14.84% to 31.88% or 34.06%, respectively. That was still below the full OPMD student's 40.68%, although the patched versions used shorter responses. The paper says these selected cases do not establish general layer specialization or causal mechanisms.
Another ablation tested projected MSE, a direct attempt to match internal representations. OPD plus projected MSE fell from 38.02% to 7.19% with further training, while OPMD rose from 39.06% to 40.68%. The authors describe direct projected MSE as difficult to stabilize. Because the alternatives used different training configurations, the comparison does not isolate prefitting alone.
Including Stage I in the calculation, OPMD reached 38.96% AMC accuracy at 768 student updates, compared with 36.50% for OPD at 1,024 updates.
The paper reports no confidence intervals or inferential significance tests, so these percentage-point differences are descriptive results from the tested configurations.
A solution-route case study used GPT 5.6 luna to independently relabel all 96 archived responses while model identities and correctness labels were withheld. The route annotations were model-generated rather than human gold-standard labels, so they need independent validation.
More broadly, the evaluation covered a limited set of teacher-student pairs and tasks. It did not test larger or broader architectures or tasks beyond mathematics and code, and it does not establish that OPMD will dominate under every training budget or task.
Paper data and sources
Original title: OPMD: On-Policy Mechanism Distillation Beyond Output Supervision
Authors: Yingdong Shi, Zhihong Li, Changming Li et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-09-29
DOI: Not available
Original paper · Full text