SAPO, a training method for language-model agents, was reported to score higher than PPO and GRPO in the reported comparisons, which covered the ALFWorld and WebShop tests. The experiments used Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct models.
With Qwen2.5-1.5B, SAPO reported 90.1% aggregate success on ALFWorld, a benchmark with 3,827 tasks across six categories. That was 35.7 percentage points above PPO and 17.3 percentage points above GRPO.
On WebShop with Qwen2.5-1.5B, SAPO’s score was 8.4 points higher than PPO’s, and its success rate was 12.2 percentage points higher. WebShop contains nearly 1.1 million products and 12,000 user instructions.
At the larger model size
With the 7B model, SAPO reported 94.0% ALFWorld success, an 88.6 WebShop score and 82.4% WebShop success. It was reported to outperform PPO, GRPO and recent variants on all three aggregate metrics.
Across the ALFWorld and WebShop experiments at both model sizes, SAPO was reported to outperform PPO by a mean of 15.1 percentage points and GRPO by a mean of 12.1 percentage points. The supplied analysis does not explain how those mean differences were aggregated.
SAPO’s main design choice is to use one causal language model—one that generates text step by step—for three jobs: representing the policy, estimating the value of a state and estimating the value of an action. In plain terms, the same model both chooses what the agent does and judges how useful its situation or decision is during training.
Instead of group-relative sampling, SAPO uses one rollout—one generated attempt—per prompt. It applies a trajectory-level advantage estimator, which assigns turn-level learning signals across a multi-step exchange, combining lambda returns with batch normalization. The authors report stable training under this one-rollout setup.
The speed comparison was narrower
The runtime comparison covered ALFWorld with Qwen2.5-1.5B only. Measured total time per training iteration was 301.4 seconds for SAPO versus 451.2 seconds for PPO, a 33.2% reduction. Generation time was 221.4 seconds for SAPO versus 306.4 seconds for PPO.
In the reported breakdown, PPO’s separate value-inference and critic-optimization stages totaled 61.4 seconds per iteration; SAPO removed those stages from the comparison. The paper reports eliminating the memory cost of a separate critic, but gives no memory-savings figure.
What the results do not establish
The study covers only ALFWorld and WebShop and the two Qwen model sizes in the supplied experiments. For each RL method, results are summarized as means and standard deviations over three random seeds.
The experiment plan includes questions about which components contribute to performance and whether training remains stable over continuous iterations. In the supplied results, however, the stability finding is qualitative: the authors report stable training from one rollout per prompt, without a quantitative stability endpoint.
The manuscript is a preprint under review. The reported gains are limited to the stated benchmark setup, so whether they hold in other environments, with other model backbones or over longer horizons remains open.
Paper data and sources
Original title: SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning
Authors: Dayang Liang, Lang Feng, Bo An, Yunlong Liu
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text