An arXiv preprint dated 20 August 2026 reports large speed gains for FlashPrefill V2 when processing 128K-token inputs on NVIDIA H20 GPUs. At the attention-operator level, the method was reported as 27.19× faster than FlashAttention-2 in BF16 and 47.26× faster in FP8; against an FA3/4-aligned dense kernel, the reported speedups were 17.54× and 30.49×.
The reported end-to-end tests also found shorter waits before a response began. Time to first token, or TTFT, was 2.1× to 3.4× faster in BF16 and 3.7× to 4.8× faster in FP8 at 128K. At batch size 16 on Qwen3-30B-A3B-Instruct-2507, the reported baseline TTFT was 123.2 seconds, compared with 36.2 seconds for BF16 V2 and 25.5 seconds for FP8.
Speed did not come with a large benchmark gap
The preprint asks whether FlashPrefill can move from an algorithmic prototype toward practical long-context serving. Its block-sparse prefill attention works on selected blocks of a prompt rather than using a fully dense pattern. The implementation adds mean correction, an FA3/4-aligned sparse kernel, PackGQA, warp specialization, pingpong pipelining, FP8 support, paged KV-cache integration and continuous batching.
The evaluation used the RULER and LongBench benchmarks with three model configurations: Llama-3.1-8B-Instruct, Qwen3-4B-Instruct-2507 and Qwen3-30B-A3B-Instruct-2507. Comparisons were run under identical hyperparameter settings on NVIDIA H20 GPUs.
On RULER, average V2 scores were 87.79 versus 88.82 for full attention on Llama-3.1-8B, 86.23 versus 87.06 on Qwen3-4B, and 91.76 versus 92.05 on Qwen3-30B-A3B. The largest average gap was 1.1 points. At 128K, V2 stayed within 1.8 points of full attention, while the FP8 version added a further 0.4 to 1.2 points of score loss.
LongBench showed a similar pattern. V2 had the highest average among the sparse methods on all three evaluations: 49.31 versus 48.25 for the strongest baseline on Llama-3.1-8B, 46.96 versus 45.90 on Qwen3-4B, and 50.73 versus 49.61 on Qwen3-30B-A3B. It remained within 0.9 points of full attention. On English passage retrieval, V2 scored between 96.0 and 99.5, compared with 91.0 to 97.0 for the baselines.
The serving tests used synthetic traffic
In a separate open-loop serving test, synthetic Poisson arrivals ranged from 1 to 16 requests per second and prompts ranged from 4K to 128K tokens. BF16 V2 measured 0.70 to 0.76 requests per second with P50 TTFT of 17 to 46 seconds. FP8 measured 0.88 to 1.34 requests per second with P50 TTFT of 2.8 to 33 seconds. The supplied analysis reports higher throughput and lower median TTFT than FA3/4 under this setup.
A correction mattered most in the lower-precision tests
An ablation, or test removing one component, found a larger score difference when mean correction was removed from the FP8 version. The uncorrected FP8 variant was 2.3 points lower on average and 6.2 points lower at 128K than corrected V2. The corresponding BF16 differences were 0.5 and 0.9 points; corrected FP8 and BF16 averages were 85.82 and 86.23.
Across a selection-threshold sweep, density ranged from 5.2% to 23.6%. Corrected FP8 stayed within 2.7 points of full attention at the lowest tested density and within 2.3 points at the default threshold of 0.1. The uncorrected version lost a further 2.7 to 5.4 points.
The correction itself added latency. At 64K, the overhead reached 12.2 milliseconds in the dense limit and measured 3.6 to 4.5 milliseconds at 90% sparsity. Relative overhead was 5% to 14% up to 50% sparsity, rising to 18% in BF16 and 27% in FP8 at 90% sparsity.
In another reported 64K FP8 kernel comparison, V2 was 6% to 7% faster than HPC-BSA across the tested sparsity levels. The dense reference latencies were 123.7 and 133.4 milliseconds.
The evidence remains tied to the test setup
The evidence covers three model configurations and H20-based measurements. The supplied analysis says operator-level results came from a single H20 GPU, while end-to-end tests used four H20 GPUs. The study did not assess other hardware or deployment topologies. Decoding falls back to dense attention, so the reported acceleration mainly concerns prefill rather than the entire response process.
The figures are point estimates: the report provides no confidence intervals, inferential tests or run-to-run uncertainty estimates. The open-loop results used synthetic arrivals rather than real production traces, leaving questions about performance across other models, accelerators, software stacks and serving workloads.
The front matter lists a GitHub repository for the implementation, but the work remains an arXiv preprint. Whether its benchmark-score trade-offs translate to broader models and real downstream systems remains open.
Paper data and sources
Original title: FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving
Authors: Qihang Fan, Huaibo Huang, Zhiying Wu et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text