An arXiv preprint reports benchmark gains for SRPO over GRPO and substantially lower training-compute figures. On AIME’24, SRPO’s reported score was 73.3±1.4%, while normalized training FLOPs were 0.26× GRPO’s.
From outcome to token-level feedback
The study asks whether sparse outcome feedback can be converted into dense, token-level supervision for long-horizon reasoning and agent behavior.
SRPO uses a two-stage pipeline. It starts with an on-policy trajectory and its outcome, creates a compact reflection patch, and resets the base model with that memory. A same-model teacher then scores new on-policy rollouts in that context, and the base model learns from the token-level scores through reverse KL, a token-by-token probability comparison.
The test bed
The evaluation used Qwen3-1.5B, Qwen3-8B and Qwen3-32B, with Llama-3.1-8B-Instruct additionally evaluated for agentic tasks.
The reported benchmark sizes were 30 AIME’24 problems, 500 MATH-500 problems, 1,319 GSM8K problems, 1,200 DeepScaleR problems, 12,087 WebShop tasks, 134 ALFWorld tasks and 300 SWE-Bench-Lite issues. LiveCodeBench was evaluated on a 240-problem slice.
Reported scores
On the mathematical benchmarks, SRPO reported 73.3±1.4% on AIME’24, 81.2±0.5% on MATH-500, 93.8±0.3% on GSM8K and 59.7±0.9% on DeepScaleR.
On agent and coding tasks, it reported 64.7±0.8% WebShop success, 76.8±1.2% ALFWorld success, 31.2±1.0% SWE-Lite success and 35.1±0.8% LiveCodeBench performance. Episodes averaged 10.2 steps.
For the main mathematical comparison, the runs began from the same SFT-400K initialization and matched training settings. The reported means used five independent seeds, with paired-bootstrap uncertainty calculated from 10,000 resamples.
The cost claim
SRPO’s compute comparison uses two reference points. A full-cost breakdown lists 5.4 in the paper’s FLOPs units for SRPO versus 20.8 for GRPO, summarized as about 3.8 times fewer FLOPs. Against a scaled SFT-2M extrapolation, the paper reports 0.08 times the training FLOPs, or 8% of that SFT cost.
At the three tested model sizes, the reported gains over GRPO were +7.8 percentage points for 1.5B models, +5.3 points for 8B models and +3.8 points for 32B models.
When the reflection matched the task
In a test that changed the reflection condition, matched reflections corresponded to 73.3±0.8% on AIME’24 and 64.7±1.5% on WebShop. Mismatched reflections corresponded to 66.5±1.0% and 55.8±1.8%, while the no-reflection condition corresponded to 65.8±1.1% and 54.2±1.9%, respectively.
Among 500 sampled reflections, 67% received scores of at least 4 and 8% received scores of 2 or lower. Helpfulness correlated with improvement at r = 0.72; score-5 reflections were associated with 34% average improvement, compared with 5% for score 1–2 reflections. The analysis reports no confidence interval for that correlation, so it should be read as an association rather than a causal estimate.
Checks and carryover
An overlap check found three flagged MATH-500 instances; removing them changed the score from 81.2% to 81.0%, a 0.2-point shift. On 1,200 unseen DeepScaleR problems, the reported overlap was 0%, while SRPO scored 59.7% against 55.4% for OPD-32B and 55.8% for OPD-72B. The check does not establish absence of every possible pretraining or teacher-data overlap.
After code adaptation, reported mathematical retention was 95.2% for SRPO, 87.2% for GRPO and 80.3% for SFT. SRPO’s post-adaptation scores were 69.8% on math and 31.2% on code.
What the results do—and do not—show
Taken together, these are computational benchmark results from the listed model and task settings, not evidence about human reasoning, user outcomes or safety in deployed agents. The reported gains and compute ratios also do not establish that SRPO will outperform every larger-teacher or reinforcement-learning method, or that the savings will transfer unchanged to other model families, hardware and production settings.
Paper data and sources
Original title: SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning
Authors: Jialong Liu, Yuling Shi, Ning Yang et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-24
DOI: Not available
Original paper · Full text