An arXiv preprint reports three-times-faster inference—the process of generating a model’s response—without additional training while maintaining performance on its benchmark tests. It examines whether a cache that retains the instruction prefix and a recent reasoning window can support efficient long-horizon reasoning with bounded memory.
Prefix Sliding keeps the instruction prefix and a recent sliding window in its cache while evicting older intermediate reasoning tokens. In the paper’s example, the window is 4,096 tokens, at most 4,196 tokens are retained, and the authors describe the arrangement as having a constant cost for each new token.
Long-sequence tests
In a benchmark of 1,024 generated sequences run on one 80GB Nvidia H100 GPU, a 4,096-token Prefix Sliding window was about as fast as regular sliding-window attention. Throughput for both stabilized around 5,000 tokens per second as sequences lengthened, while full attention continued slowing.
At a 32K-token sequence length, reported throughput was 5,479 tokens per second for Prefix Sliding and 1,477 for full attention. At 128K, the figures were 5,224 and 448, respectively.
On AIME25, GPQA and MATH500, the 4,096-window condition scored 33.9, 37.0 and 91.5, compared with 34.2, 37.6 and 91.7 for full attention. The results were averaged across 64 runs, and no formal confidence intervals or hypothesis tests for the headline comparisons are reported.
The main computational tests used Qwen3-1.7B on GPQA, MATH500 and AIME25, alongside a self-created math-problem dataset filtered for guessability, verifiability and difficulty.
The comparison extended to model training
The paper also tested the method during reinforcement learning with GRPO, comparing full-generation backpropagation with a last-window version. Unrelated hyperparameters were held fixed, and the latter setup passed four times as many recent tokens to the trainer.
Under near-equal memory budgets—8,192 maximum tokens for full attention and an 8,192-token Prefix Sliding window—the comparison reported longer reasoning traces and higher rewards for Prefix Sliding.
Generator-trainer KL mismatch was above 0.1 when only the sliding window was passed to the trainer; with 4K tokens passed, the mismatch was lower. For most runs, the authors selected a four-times multiplier, or 8K tokens, and described it as roughly on par with an eight-times, 16K-token setting.
A short validation on DeepSeek-R1-Distill-Qwen-7B passed 32,768 tokens to the trainer while backpropagating through 8,192 under a four-times multiplier; reported performance was comparable to full attention.
Performance varied by task
In an ablation comparing last-k retention, summary methods and pure sliding-window attention, Prefix Sliding had the best reported performance-efficiency trade-off among the alternatives tested.
Without training, LiveCodeBench required a Prefix Sliding window of at least 16,384 tokens to match full attention, with maximum generation set at 262,144 tokens.
HealthBench generations required 2,086 tokens on average. With a 2,048-token window, the setup rarely evicted tokens, leaving little room for a speed-up.
The reported task results differed: LiveCodeBench required a larger retained window, while short HealthBench generations left little room for savings. The work is an arXiv preprint, version 1, dated 26 Aug 2026.
Paper data and sources
Original title: Prefix Sliding for efficient test-time scaling
Authors: Niklas Muennighoff, Zhengyang Wang, Zeyi Chen et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text