A preprint reports that a constrained training method for student language models recorded the highest reported rigorous Reasoning Success Rate in every evaluated setting. In the comparisons highlighted by the authors, final-answer accuracy stayed close to GRPO, one of the study’s reinforcement-learning baselines, in one test and was higher than GRPO in another. The study asks whether a worst-case constraint can reduce reward hacking without sacrificing task performance during reasoning distillation.
The study separates the final answer from the reasoning path used to reach it, asking whether both can be kept strong at once. The proposed method is an unaugmented constrained-reinforcement-learning distillation framework with a weakest-link teacher-adherence boundary, a variance-reduction mechanism, and no teacher access at test time.
A constraint aimed at the weakest step
The weakest-link boundary focuses on the least teacher-aligned part of a solution rather than relying only on an overall average. In the paper’s formal result, under a finite-value assumption for the limiting Markov decision process, an optimal policy satisfies the worst-case averaged constraint almost surely. For general readers, that means the paper states the condition is to hold with probability one within the mathematical setup.
That distinction is reflected in the scorecard. Final Answer Correctness, or FAC, records whether the final answer is correct. Constraint Satisfaction, or CS, records whether the trajectory stays within the stated constraint. Reasoning Success Rate, or RSR, uses an LLM judge to assess the logical validity of intermediate steps.
The final answer was only part of the test
The mathematical tests used Qwen2.5-1.5B distilled from Qwen2.5-Math-7B-Instruct on MATH, Llama-3.2-3B distilled from Llama-3.2-11B-Vision-Instruct on MATH, and Qwen2.5-1.5B distilled from Qwen2.5-Math-7B-Instruct on GSM8K and evaluated on GSM-Symbolic.
On Qwen plus GSM-Symbolic, the proposed method’s FAC was 76.14%, almost identical to GRPO’s 76.24%, while its CS was 83.88%.
On Llama plus MATH, FAC was 20.54% for the proposed method versus 18.78% for GRPO. CS was 56.94%, compared with 14.28% for GRPO.
Across all evaluated settings, the authors report that their method achieved the highest rigorous RSR. In pairwise comparisons using an LLM judge, the proposed method’s reported win rates were 56.2%, 63.5%, and 60.6% across the reported settings.
The code test sharpened the contrast
The code-generation experiment combined MBPP train and validation data from 464 problems with APPS data, then evaluated pass@1 on 500 MBPP test problems.
In that experiment, GRPO reached 92.0% pass@1 but had an RSR of only 1.8%. The proposed method had the highest reported RSR at 34.2% and the smallest reported gap between FAC and RSR, at 11.4 points. All reinforcement-learning baselines, including the proposed method, were implemented on GRPO and shared group-relative advantage estimation and PPO-style clipping mechanics.
The teacher signal has limits
An additional analysis examined whether the teacher signal tracked reasoning quality. The pooled Spearman correlation between worst-case prefix-average teacher log-likelihood and RSR was positive at +0.231.
The authors also acknowledge that teacher likelihood does not perfectly capture reasoning validity. A step can be valid but stylistically unusual and still receive a high cost under the teacher-based measure.
The work is labeled a preprint. Its reported benchmark results point to a closer relationship between answer accuracy and judge-assessed reasoning quality in the tested mathematics and code settings, while the teacher-likelihood proxy remains an imperfect measure of validity.
Paper data and sources
Original title: The Weakest Link: Distilling LLM Reasoning with Worst-Case Constrained Reinforcement Learning
Authors: Matthieu Zimmer, Xiaotong Ji, Tu Nguyen, Haitham Bou-Ammar
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-09-29
DOI: Not available
Original paper · Full text