A system called psRL reported up to 5.2× higher training throughput than veRL across four agentic-reinforcement-learning workloads, according to an arXiv preprint. Agentic reinforcement learning refers here to training AI agents through repeated decision steps. psRL is designed to reuse common prefixes—the repeated opening parts of model inputs—rather than process each one as entirely separate work.
The reported speedups ranged from 1.2× to 5.2× over default veRL. Against two veRL variants that used vLLM-style or SGLang-style prefix sharing, psRL’s reported gains on Search and WebShop were 1.5× to 2.5×.
A system built around repeated inputs
psRL combines sharing between batches and within a sequence with scheduling by semantic group, token-wise micro-batching, adaptive block allocation and dynamic block caching.
The evaluation covered Search, WebShop, ALFWorld and a proprietary industrial DTN Agent. The standard workloads used models from the Qwen2.5 family, while DTN used a Qwen3-235B mixture-of-experts model.
Default veRL served as the main comparison, alongside the two prefix-sharing variants. For the standard benchmarks, data, tensor and pipeline parallelism were each set to 2 across 8 GPUs. DTN used data and tensor parallelism at 4, expert parallelism at 8 and pipeline parallelism at 12 across 1,536 GPUs.
The implementation is built on veRL, with primary changes to Megatron-LM, and contains approximately 7,000 lines of code organized into named system components.
Repeated prefixes were common in the tests
One measure, the prefix-match rate, tracks how often an input prefix matched work seen earlier. The reported rates were 94.51% for ALFWorld Step, 90.22% for WebShop Step, 88.92% for DTN Step-wise and 62.24% for DTN Tree-Step. Tree-Step was the lowest of the four.
The timing profile also differed under structured sampling. In WebShop Step, update time rose by more than fivefold compared with trajectory sampling, while rollout time stayed virtually unchanged. The update phase accounted for 40% to 58% of total RL time under structured sampling, compared with 19% to 32% for the linear baseline.
The reported performance advantage
On the update phase of DTN Agent Step, psRL reached 239.1k tokens per second, compared with 62.7k for SGLang-PS—approximately 3.8 times as much. Across the four benchmarks, the paper reports 1.2× to 5.2× speedups over veRL.
Peak-memory comparisons also favored psRL in the reported tests. vLLM-PS and SGLang-PS used 2× to 10× more memory than veRL; on ALFWorld Step-S, vLLM-PS used 73.1 GB versus 6.4 GB for veRL. In the DTN comparison, peak memory was 56.2 GB with veRL and 33 GB with psRL, a reported reduction of 41%.
The paper reported an end-to-end speedup of up to 2.1×. In DTN Step, the listed Log Old Prob time fell from 57.1 seconds to 7.1 seconds, Reference time from 68.3 to 8.3 seconds, and Update time from 462.1 to 88.7 seconds. A WebShop Step iteration was reported at 148.1 seconds.
The gap persisted as scale grew
At 64 devices on Search Step, psRL reached 128.5k tokens per second, 2.7× veRL and 1.9× SGLang-PS. At 12,288 GPUs on DTN, it reached 1.14 million tokens per second, 4.7× veRL and 3.4× SGLang-PS.
The reported sequence-length test showed a sharper contrast. At eight times the sequence length, psRL sustained 12.8k tokens per second on Search Step, a reported 3× to 4× advantage over SGLang-PS and veRL. On DTN Step, it delivered 42.1k tokens per second at the same scale, while all baselines exhausted GPU memory and crashed.
What the component tests showed
A scheduling comparison on DTN Tree-Step showed a narrower worker-time range for psRL: 245.13 to 261.23 seconds. Sharing-agnostic balancing ranged from 235.12 to 299.45 seconds, Prefix-Match-First from 209.45 to 305.12 seconds, and Balance-First reached a maximum of 287.25 seconds.
In a separate memory comparison, dynamic block caching, or DBC, was accompanied by lower reported peak memory. DTN Step and Step-RT were each listed at 33.0k MB with DBC, versus 73.0k and 73.1k MB without it, a reduction of more than 54%. Tree-Step was 30.3k MB with DBC versus 72.5k MB without.
The evidence remains specific to the tested setups
The paper reports descriptive benchmark comparisons, but it does not report run-to-run variability, confidence intervals or significance tests. The reported figures therefore do not show how stable the differences would be across repeated runs.
The scope is specific: four named workloads, selected model families and the reported parallelism configurations, including a proprietary DTN workload. The numbers describe those tests; they do not establish that the same gains would appear across all agentic-RL workloads, models or training systems.
The document is an arXiv version 1 preprint dated 26 August 2026. It says its source code will be publicly available soon, and the supplied text and metadata do not report funding or conflicts of interest.
Paper data and sources
Original title: psRL: Efficient Training for Agentic AI via Training-Time Prefix Sharing
Authors: Mianjie Yu, Zizhao Mo, Huanyu Qu et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text