A computational study of large language models suggests that the pattern of an unlearning update may matter more than its overall size when researchers test whether supposedly forgotten information can return. In a pooled analysis of healthy checkpoints, the study’s forget–retain alignment gap, or FRAG, had a Spearman correlation of −0.78 with the change in Extraction Strength (∆ES) after a relearning attack, compared with −0.36 for global L2 distance, a measure of the total weight change. A more negative correlation was associated with less recovery in the tested checkpoints. When FRP checkpoints were excluded, the figures were −0.74 for FRAG and −0.10 for global L2 distance. The result is an association in the tested checkpoints, not proof that FRAG causes stronger unlearning.
The question behind the work is whether update structure—especially its alignment with weights important for forgetting and retaining—can predict relearning robustness better than raw displacement in weight space. The authors hypothesize that robust unlearning should concentrate changes on forget-critical weights while sparing retain-critical ones. They test that idea by exposing an unlearned checkpoint to another round of fine-tuning and measuring how much of the target material returns.
A second training pass as a stress test
FRAG is a training-free score. It is calculated as forget-side cosine alignment minus γ times retain-side cosine alignment, with γ set to 1 in the reported setup. In ordinary terms, the score rewards an update that points toward what is important to forget and penalizes an update that also points toward what is important to retain. The companion method, called FRP, ranks weights using forget-critical importance, a retain-critical penalty and a magnitude prior, then prunes selected weights to create the unlearned model.
The evaluation used OpenUnlearning on three benchmark settings: TOFU with LLaMA-3.2-1B/3B, WMDP-cyber with Qwen2.5-14B-Instruct, and MUSE-News with LLaMA-2-7B. The comparison included GA, GradDiff, NPO, RMU and SP as named baseline methods. In the default relearning attack, the released checkpoint was fine-tuned for one epoch at a learning rate of 10−5 with AdamW, using retain-only, forget-only, or combined forget-and-retain data. The design asks whether a second training pass can restore knowledge that the unlearning step was intended to remove.
Where the pattern showed up
On TOFU, FRP with β=0.00 had the lowest reported average post-attack Extraction Strength and change in that score, while retaining nonzero utility. For the 1B LLaMA-3.2 model, the averages were ES 0.085, ∆ES 0.029 and utility 0.444. For the 3B model, they were ES 0.117, ∆ES 0.049 and utility 0.553. These are benchmark averages, and the study reported no confidence intervals or significance tests for them.
The WMDP-cyber results supplied a direct comparison between update size and update structure. For FRP, unlearned accuracy was 0.429 and MMLU was 0.668. After the retain-only attack, accuracy was 0.417, down from the unlearned value, while MMLU was 0.707. FRP’s global L2 distance was 443.4 and its FRAG score was 9.198; SP had a larger L2 distance of 464.1 but a lower FRAG score of 3.149. In that reported benchmark and attack setting, the larger overall move did not correspond to the more favorable reported relearning profile.
A direction-blind noise control pointed in the same direction. Noise with σ=0.004 matched FRP’s L2 distance, yet its change in accuracy was −0.027, compared with −0.012 for FRP. Both noise settings received a FRAG score of 0.39 on the reported scale, shown as ×100, below FRP’s 9.20. The control indicates that sheer distance can be a poor stand-in for the forget–retain alignment measured by FRAG.
The reported difference persisted in matched comparisons. At equal sparsity, FRP had 2.6–4.0 times lower pre-attack ES and 2.6–3.3 times lower ∆ES. At near-identical update norms, its pre-attack ES was 2.4–3.0 times lower and its ∆ES was 2.2–2.9 times lower. These are descriptive ratios without confidence intervals or significance tests, so they do not establish that selectivity alone caused the differences.
Results beyond the main comparison
The pattern was also examined in another model family and on a news-focused benchmark. In a WMDP-cyber cross-family comparison, FRP’s L2 and FRAG values were 49.9 and 6.311 on Zephyr-7B-β, compared with 443.4 and 9.198 on Qwen2.5-14B-Instruct. On MUSE-News with LLaMA-2-7B under a retain-only attack, FRP had utility 0.379, pre-attack ES 0.059 and ∆ES +0.067, while GradDiff’s utility was 0.048. FRP had the lowest reported ∆ES among the utility-preserving methods listed for that benchmark.
The study does not present FRP as a universal winner. Its reported robustness gains came with a utility trade-off: the MMLU drop for FRP exceeded those of RMU and SP, placing it on a robustness–utility frontier rather than showing dominance on both measures.
What the evidence does not settle
The evidence is limited to TOFU, WMDP-cyber and MUSE-News across several model families and scales. The authors note that FRAG has limited resolution among dense unlearning methods, while FRP was tested as unstructured pruning. Broader benchmarks, multilingual data and larger architectures remain untested.
The predictor analysis was restricted to healthy checkpoints, and no confidence intervals or p-values were reported.
The work is identified as arXiv:2608.25429v1, dated 26 August 2026. It was partly supported by IITP grants funded by the Korean government’s Ministry of Science and ICT, listed as RS-2025-02264029 and RS-2023-00228255. Its narrower conclusion is that forget–retain alignment may be a more useful guide to relearning robustness than raw weight distance in the tested settings.
Paper data and sources
Original title: Distance Is Not Enough: Forget-Retain Alignment Gap Predicts LLM Relearning Robustness
Authors: Yi Chen, Hanna Hsieh, Shuhong Liu et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text