An arXiv preprint reports higher benchmark scores from a filter designed to screen out sampled responses whose teacher guidance conflicts with the final verified result. In the authors' experiments, Reward-Aligned On-Policy Distillation, or RA-OPD, had the highest reported average results in the tested math and code comparisons. Its total training time was 4.48 hours, close to 4.38 hours for standard on-policy distillation, or OPD.
The report uses Qwen3 and DeepSeek-R1 student and teacher configurations for math and code tasks, drawing on DAPO-Math-17K for math training and Eurus-2-RL-Data-Code-25K for code training. It reports results across seven math benchmarks and three code benchmarks. The document is an arXiv preprint dated 28 Aug 2026.
A rule for conflicting signals
On-policy distillation uses a response sampled from the student during training while a teacher provides token-level guidance. RA-OPD adds a trajectory-level check, combining those token signals into one distillation return for the full sampled response and comparing that return with a verified outcome reward. A correct final answer receives a binary reward of one, while an incorrect answer receives zero. The method keeps trajectories that satisfy the alignment rule, filters conflicting trajectories, and treats a zero-return trajectory as neutral.
Standard OPD and RA-OPD used one on-policy response per training prompt and two student-training epochs. Uni-OPD used four responses per prompt. Final evaluation used frozen checkpoints and common decoding settings, with 32 samples for each AIME and AMC problem, eight for other math benchmarks, and four for code benchmarks.
Higher scores across math and code tests
On the two Qwen3 math student settings, RA-OPD had the highest reported average. For Qwen3-4B-Base, its avg@k score was 45.88, 5.20 percentage points above OPD and 3.52 points above ExOPD. For Qwen3-8B-Base, it reached 49.43, 5.09 points above OPD and 2.48 points above ExOPD. The corresponding pass@k averages were 61.11 and 61.56, with margins of 4.32 and 3.06 points over the strongest baselines.
In a separate math comparison, the student was DeepSeek-R1-Distill-Qwen-7B and the teacher was Skywork-OR1-Math-7B. RA-OPD reached an avg@k score of 69.34, 4.91 points above OPD and 2.79 above Uni-OPD. Its pass@k average was 83.48, ahead of Uni-OPD by 3.53 points and ExOPD by 4.52 points.
On code tasks, RA-OPD had the highest reported average across the three benchmarks, with 58.97 avg@4 and 65.17 pass@4. The individual avg@4 scores were 78.81 on HumanEval+, 70.97 on MBPP+, and 27.14 on LiveCodeBench v6. The average was 3.93 points above OPD and 2.87 above ExOPD, while pass@4 was 1.51 points above OPD and 1.05 above Uni-OPD.
Filtering was common, but timing stayed close
The paper reports that negative-return trajectories were common in both settings. Mean trajectory-level returns remained below zero. Negative-return proportions averaged 94.25% in the Qwen3 setting and 92.46% in the DeepSeek-R1 setting. Overall dropped fractions averaged 48.63% and 68.23%, respectively.
Total training time stayed close to standard OPD in the reported setup. RA-OPD took 4.48 hours, compared with 4.38 hours for OPD, 4.77 hours for ExOPD, and 14.41 hours for Uni-OPD. Uni-OPD therefore took about 3.22 times as long as RA-OPD. The figures place RA-OPD near standard OPD in total training time, but they do not show literally zero added compute.
Further checks
An ablation compared standard OPD with RA-C, RA-I, RA-Inv and the full RA-OPD mask. OPD scored 44.34 avg@k and 57.66 pass@k. RA-C and RA-I scored 45.31 and 45.83 avg@k, with pass@k scores of 60.63 and 59.75. RA-Inv scored 43.41 avg@k and 56.80 pass@k. The full RA-OPD mask had the highest reported scores in that comparison, at 49.43 avg@k and 61.56 pass@k.
Mean actor entropy, the study's measure of exploration, was closely matched between the methods. On Qwen3-8B-Base, it was 0.183 for RA-OPD versus 0.177 for OPD. On DeepSeek-R1-Distill-Qwen-7B, it was 0.424 versus 0.422. The values were close in both model-family settings.
The evidence stays within the tested setup
The reported evidence covers Qwen3 and DeepSeek-R1 model families, DAPO-Math-17K for math training and Eurus-2-RL-Data-Code-25K for code training, and seven math and three code benchmarks. Standard OPD and RA-OPD used one response per training prompt and two epochs, while Uni-OPD used four responses; final evaluation used frozen checkpoints and the stated decoding settings. The results therefore describe this particular combination of models, data, benchmarks and sampling choices.
The supplied front matter lists author affiliations and contact addresses but no funding information. The text directs readers to Appendix C for detailed training and evaluation settings, but it gives no explicit data, code or supplementary-file availability statement.
Paper data and sources
Original title: When Teacher Guidance Misleads: Reward-Aligned On-Policy Distillation
Authors: Siyuan Gan, Yuhan Li, Xiran Wang et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-28
DOI: Not available
Original paper · Full text