A method called GRACE reported higher accuracy when retrieving the samples selected for a language model to forget than RASLIK in every displayed comparison. It also had the highest reported model-utility score in six of eight algorithm-model combinations. Within each unlearning algorithm, the largest difference in quantitative forget quality across selection methods was 0.012.
The selector starts with a few examples
The paper tackles the problem of building compact forget and retain sets from a mixed training corpus when a request provides only a few examples of undesired behavior. GRACE computes a forget gradient direction from those undesired outputs, then uses non-negative orthogonal matching pursuit, or NNOMP, to choose a compact forget set. It projects out that forget direction before selecting representative retain examples.
The experiments used instruction-tuned LLaMA 3.1 8B and Qwen 2.5 3B models. GRACE was compared with embedding-based retrieval and RASLIK, then paired with four unlearning algorithms: GradDiff, SimNPO, NPO and RMU. The methods were run under a common setup of 200 steps and batches of eight on one 48GB A100.
The tests used two kinds of training data
The tests used two constructed data settings. The heterogeneous MUSE setup combined 100 Harry Potter-related forget samples from MUSE Books with Dolly-15k, which contains approximately 15,000 instruction-response pairs. The domain-specific setup used approximately 1,200 WMDP-Bio biosecurity multiple-choice instruction-answer pairs alongside 20,000 AlpaCare-MedInstruct samples. It randomly selected 200 WMDP-Bio samples as forget data and held out 200 samples for testing in both datasets.
For retrieval, GRACE used 10 MUSE seed samples and 20 WMDP-Bio seeds, then built forget sets of 100 and 200 samples, respectively. Its candidate pool was four times the target forget-set size before the NNOMP selection step. The evaluation separated this retrieval measure, called Forget Retrieval Accuracy, or FRA, from post-unlearning Forget Quality, or FQ, and Model Utility, or MUT. FRA excluded the seed samples.
The biggest differences appeared in utility
On LLaMA, GRACE's FRA was 46.67% versus 28.89% for RASLIK on MUSE, and 82.22% versus 61.67% on WMDP-Bio. On Qwen, the corresponding figures were 43.33% versus 15.56% on MUSE and 87.78% versus 81.11% on WMDP-Bio. The reported GRACE value was higher in every displayed model-dataset comparison.
Utility was highest for GRACE in six of the eight algorithm-model combinations, with the largest reported gain, +0.16, appearing for GradDiff on LLaMA. By comparison, Forget Quality was much less sensitive to the selector: within any one unlearning algorithm, the maximum difference across selection methods was 0.012.
A separate analysis had language models act as judges of six forget and retain dimensions. It used paired-block Friedman tests, Nemenyi comparisons and Benjamini-Hochberg correction; all six metrics had corrected p-values below 10 to the negative 10. GRACE received the highest average ranks for retain-side answer preservation, semantic quality, and coherence and correctness, at 2.170, 2.161 and 2.158. Pairwise comparisons put it ahead of both baselines at p-values below 10 to the negative 16. Agreement between judges was lower for Deviation Quality than for the other judge dimensions.
Additional tests examined the selector's components. The no-projection SimNPO variant reported 0.91 versus 0.95 for complete GRACE on WMDP-Bio, and 0.93 versus 0.96 on MUSE. The no-clustering variant did not improve either metric. The ablation covered a limited model and algorithm set.
In a separate forget-retain swap using SimNPO on LLaMA 3.1 8B, joint GRACE selection had the highest MUT, at 0.57 on WMDP-Bio and 0.48 on MUSE. A mixed setup using RASLIK's forget set and GRACE's retain set scored 0.55 and 0.45, respectively. Forget Quality remained similarly high, and the swap was restricted to one model and one unlearning method.
The method has clear boundaries
Several limits constrain how far the findings can be carried. GRACE assumes that seed examples induce a reasonably coherent forget direction, so requests involving heterogeneous or unrelated phenomena may be harder for it. The experiments also relied on controlled access to ground-truth forget samples, a condition that only partly reflects deployments where those training examples are unknown or inaccessible. The evaluation was not comprehensive across domains, model sizes, languages or request types, and per-sample gradients plus NNOMP add computational and scalability demands.
Paper data and sources
Original title: GRACE:Gradient-guided Coreset Selection for LLM Unlearning
Authors: Praveen Bushipaka, Andrea D'Angelo, Lucia Passaro, Tommaso Cucinotta
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-28
DOI: Not available
Original paper · Full text