The ceiling was low
Researchers testing how much attention-cache eviction could improve found that the answer was usually: not much. Across eight model families and the budgets examined, the median share of the remaining gap that a better kept subset could close was between 2.1% and 4.7%. The per-budget 95% intervals stayed within 1.7% and 5.1%. Top-mass selection, the rule that keeps the keys carrying the most original attention weight, already matched the ESpool reference exactly in 39% of cases at s = 4 and 21% at s = 8.
That result concerns a narrow question. In the test, the model's original attention weights were fixed: a selector kept some keys, renormalized those weights and returned their weighted average. The contest was therefore about which keys survived, not about relearning the weights. Under that forced-weight setup, the authors interpret the small ceiling as evidence that top-mass is close to the best attainable subset on the tested data.
A large search, with a built-in boundary
The main frozen-head panel contained 111,744 query-row instances. Six additional arms brought the analysis to ten models, including a 56,448-instance Qwen2.5-7B scale arm and a separate 72,576-instance content-diversity run. These were additional panels, not numbers to be added into one common sample.
To estimate the ceiling, the researchers exhaustively enumerated a restricted set of candidate subsets, then checked those candidates with a mixed-integer second-order-cone feasibility solver. The solver had 120 seconds per case and was run on a 960-instance manifest; 21 cases hit memory limits. Because the search was restricted to that candidate set, the result is an estimate of the available improvement within the tested search space, not a guarantee that no other subset could do better.
A separate signal may help identify the cases where the simple rule matters most. The mass dropped by top-mass selection predicted nonzero error and cases with excess over the candidate optimum with an area under the ROC curve, a ranking measure, of 0.76 to 0.85 on four headline families and 0.76 to 0.89 across ten arms. A one-pass statistic called Phi correlated with ESpool at Spearman correlations from 0.84 to 0.98.
The deployment picture changed when bytes counted
Benchmark deployment tests covered Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, Qwen3-8B and Qwen3-14B, with contexts from 4,096 to 32,768 tokens. They ran RULER and LongBench through NVIDIA's official kvpress pipeline and compared fourteen library baselines: nine uniform-selection methods and five per-head methods at the same per-head budget.
That setup exposed a practical distinction between selecting less and storing less. In needle runs, the nine uniform-selection methods held 0.24% to 4.3% of the full cache. The five per-head methods, including KVzip, held the full cache because their choices were represented as a mask over one equal-slot tensor. ContourKV held 0.15% to 5.3% when its budget was enforced. In other words, a reported selection budget did not automatically mean the pipeline was using less cache.
The cost of enforcing a nominal budget was large in paired tests even when the ranking itself did not change. Scores were lower in 140 of 157 conditions and higher in none. The average LongBench difference was 14.1 points at s = 32, while the difference on RULER reached 62.2 points at s = 128. Those results make the accounting choice part of the comparison, not a footnote.
Question visibility also mattered
Retrieval results offered a second warning. In eight document-paired Llama-3.1-8B conditions at s = 64 and 128, restricting the ranking to document rows changed hit rate by minus 88.2 points, with a reported interval of minus 90.0 to minus 86.4. Moving eviction to the document boundary produced a minus 87.6-point change, with an interval of minus 89.0 to minus 86.3. The difference between the two conditions was only 0.6 points.
Because the rankings were computed after question rows were visible in this test, the near-identical results suggest that much of the apparent retrieval margin coincided with question visibility rather than allocation alone. The test therefore ties interpretation of the margin to the point at which the question became visible.
Promising comparisons, but not a universal winner
ContourKV illustrates why headline win counts need their conditions attached. When its budget was enforced at the byte count used by the budget-enforcing baselines, it beat KVzip in 93 of 160 conditions and lost 22. Without enforcement, it won 146 of 157 paired conditions and lost two. The comparison looked far stronger when the physical-budget constraint was removed.
One audit also reached beyond reconstruction error. At s = 8, the flat-cell balancing result was associated with lower held-out continuation cross-entropy by 0.122 to 0.243 nats in five of eight model families. But across three Qwen3 scales, the cell map retained only 55%, 15% and 31% of the gain available when all cells were considered. That uneven result argues against treating the shortcut as a stable rule across model scales.
The reported intervals were built with a cluster bootstrap over documents, using 1,000 resamples and 95% percentile intervals; paired per-document differences were used when methods shared a batch. That supports caution about the exact size of any one margin, while the broader pattern is straightforward: selection gains were small, and implementation details could be large.
The work is an arXiv v1 preprint dated 25 August 2026, and no funding information is reported in the supplied text or metadata.
Paper data and sources
Original title: Trust the Mass: Forced Weights in KV-Cache Eviction
Authors: Jack Shi, Jerry Gu
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-25
DOI: Not available
Original paper · Full text