The paper reports strong gains on some long-context tasks when fine-tuning used the same sparse key-value cache policy as inference, but the pattern was uneven. On four core tasks, sequence-parallel training was best on NQ and HotpotQA, the base condition led on TriviaQA, and the proposed method led on PopQA.
The policy match
The study asks whether a policy match helps: fine-tuning uses the same key-value cache restrictions that govern sparse inference, rather than changing policies between training and use. The comparison was with sequence- or context-parallel training.
The proposed method combines nested activation checkpointing, CPU offloading, a linear key-value-cache recurrence and autograd saved-tensor packing. The authors say it can handle arbitrary cache policies and arbitrary-length sequences with constant resources.
The benchmark contained 10 Helmet tasks across five capability categories at 64k and 128k token contexts. Each task was divided into non-overlapping development and held-out evaluation partitions.
Proposed-method runs used a 32,768-token cache, 1,024- or 2,048-token chunks, 8-bit cache quantization, four devices and an effective batch size of eight. All proposed-method fine-tuning and sparse-attention inference experiments used the open-source KeysAndValues library.
Strong results, uneven record
On the additional tasks, the authors state that the proposed method strongly outperformed sequence-parallel and base models. In the TREC Coarse row, its scores were 96.0, 96.4 and 96.2 across the sparse policies, compared with 30.0, 23.2 and 77.6 for sequence-parallel training and 28.2, 19.8 and 36.0 for the base checkpoint.
That result did not repeat cleanly across the four core tasks. The authors report mixed, inconclusive results there: sequence-parallel training led two tasks, the base condition led one, and the proposed method led one.
The output-length clue
For the proposed or exact-inference conditions, the output-to-target ratio, R, was about 1 and the fraction of outputs reaching the maximum length, p128, was about 0. With sparse inference on sequence-parallel checkpoints, R was often large and p128 was often close to 100%.
The paper interprets this contrast as a training-inference inconsistency: a checkpoint trained under one cache regime may not behave well when sparse inference applies another. That is a reading of the descriptive pattern, not proof of a causal mechanism.
Metric choice also mattered. SubEM can count an output as successful if it contains the target substring alongside arbitrary nonsense, while stricter extraction requirements such as Accuracy sharply reduce the reported sequence-parallel and base performance.
The cost of matching
The alignment was not free. For Helmet 128k training updates, the proposed method was about 30% more expensive than exact training at a chunk size of 2,048 tokens.
The reported overhead varied by configuration. 1k variants ran 11% to 14% longer than 2k variants, while H2O variants were 2% to 4% slower than the simpler lr and slr policies.
One warning came from the H2O comparisons. Ranking the H2O variants was inconclusive, and h2oor 1k on JSON-KV had R = 3.7 and p128 = 85%, which the authors describe as a hint of a similar output-length failure mode.
A conditional result
Taken together, the preprint offers a conditional result rather than a universal win. It reports strong performance on additional tasks, mixed performance on four core tasks, an output-length warning in one H2O configuration and about 30% higher training-update cost in the 128k comparison. The comparisons are descriptive, with no inferential uncertainty reported, and the H2O ranking remains inconclusive.
Paper data and sources
Original title: Learning how to Forget: Fine-tuning for Long-Context Sparse Attention
Authors: Matthias Seeger, Zeyu Zhang, Vihang Patil et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text