A higher average, with a catch
An arXiv preprint describes an AI training method, OPDVR, that shows higher average scores than sampled-token on-policy distillation in two reported settings, although one benchmark in the same-architecture test goes the other way. OPDVR records 49.1 avg@16 accuracy versus 47.8 for sampled-token OPD in the same-architecture evaluation. In cross-architecture distillation, it records 22.8 versus 20.9 and scores higher on every listed benchmark.
OPDVR applies what the paper calls an extremely simple ReLU gate, a filter that suppresses negative contributions, to sampled-token OPD. In ordinary terms, the gate is meant to keep dense teacher guidance from working against a verifier's judgment about whether a reasoning trajectory is correct. The method combines on-policy distillation with verifiable reward training without introducing additional hyperparameters.
The formal analysis gives that filter a specific rationale. It states that OPDVR's update is never opposite to the verifier-gradient direction, while standard sampled-token OPD can have a negative projection when the two signals conflict. Put plainly, the proposed rule is designed to prevent the teacher signal from pulling against the correctness signal. This is a theoretical alignment claim, separate from the benchmark comparisons.
What the benchmark tables show
The tests used a filtered DeepMath subset of 57k samples at difficulty level 6 or higher for the same-architecture setting. The cross-architecture setting used DAPO-Math-17k and ran distillation for three epochs. Models were evaluated with the paper's avg@16 accuracy measure on six named reasoning benchmarks: AIME24, AIME25, AMC, MATH500, Minerva and OlympiadBench.
In the same-architecture results, OPDVR averaged 49.1, compared with 47.8 for sampled-token OPD and 50.4 for the teacher. OPDVR was higher than sampled-token OPD on AIME24, AIME25, AMC, Minerva and OlympiadBench. MATH500 was the exception: OPDVR scored 84.7, while sampled-token OPD scored 85.5.
In the cross-architecture comparison, OPDVR averaged 22.8 versus 20.9 for sampled-token OPD and scored higher on each listed benchmark. The authors report gains over sampled-token OPD of 2.7 points on AIME24 and 2.1 points on AIME25 in the same-architecture setting, plus 5.5 points on AMC and 1.7 points on MATH500 in the cross-architecture setting.
The paper's broad statement that OPDVR outperforms sampled-token OPD across all six benchmarks therefore needs to be read alongside the same-architecture table, where OPDVR trails on MATH500. The reported evidence supports a higher average in that setting, but it does not show a clean sweep across every test.
A wider variant of the method
The paper also tests GRPD, an extension that replaces a yes-or-no verifier reward with a relative score within groups of eight responses. In the reported comparison, GRPD averaged 49.4, compared with 44.8 for GRPO and 48.4 for OPD. It scored higher than GRPO on every listed benchmark and tied OPD on AIME25 at 31.7.
The authors report GRPD gains of 6.5 points over GRPO on AIME24 and 10.9 points on AIME25, along with a 2.8-point improvement over OPD on AIME24. Those figures come from the six-benchmark evaluation described in the paper.
An inverse-gated variant averaged 44.6, below OPDVR's 49.1 and OPD's 47.8 but above the initial student's 42.0.
The paper also tracks how often the gate was zero. That zero-gated token ratio stayed approximately 0.48 to 0.50 for the 4B student and 0.40 to 0.44 for the 1.7B student throughout training. The paper presents these as approximate plotted ranges.
The evidence has clear limits
The numbers should be treated as reported comparisons rather than precise estimates of repeatability. For the benchmark results, the supplied analysis reports no uncertainty intervals or repeated-run variability. The evidence is also tied to the particular datasets, training settings and six benchmarks described above.
The document identifies itself as arXiv version 2 dated 28 Aug 2026. The supplementary material reports a learning rate of 1e-6 and a maximum response length of 8192, and the paper states that code is available in the OPDVR repository.
Paper data and sources
Original title: On-policy Distillation with Verifiable Reward
Authors: Wenze Lin, Jiale Zhao, Xitai Jiang et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-25
DOI: Not available
Original paper · Full text