An arXiv preprint reports that current methods for AI “unlearning” can reduce harmful memorization, but still struggle to preserve useful responses involving the same concepts. The paper describes a strong trade-off between forgetting and utility and says the methods did not meaningfully improve contextual separation.
The underlying challenge is that some concepts have both harmful and benign applications. ConceptGuard was designed to test whether a model could suppress the harmful use while retaining correct, useful behavior around the same concept.
A benchmark built around dual-use concepts
ConceptGuard contains 5,166 paired instances spanning 68 dual-use concepts. It is split evenly between a harmful forget set, with 2,583 instances, and a benign retain set of the same size. Its construction involved extracting source material, identifying and grouping concepts, and generating complementary benign examples under manual supervision.
Researchers first fine-tuned Qwen-2.5-3B-Instruct and Llama-3.1-8B-Instruct on the combined harmful and benign data. They then applied four unlearning approaches to the harmful set: Gradient Ascent, SimNPO, RMU and UNDIAL.
The evaluation used ROUGE, a text-overlap measure, to track harmful memorization and helpful retention, alongside rubric-based LLM judgments of harmfulness and helpfulness. The benchmark’s central separation score was defined as helpfulness minus harmfulness.
A cost to useful performance
The clearest trade-off appeared in the Gradient Ascent results. On Qwen, its ROUGE scores were 0.11 for harmful memorization, 0.12 for memorization on harmful queries, 0.18 for helpful retention and 0.09 for utility on benign queries. On Llama, the corresponding scores were 0.02, 0.10, 0.05 and 0.13. The paper reports Gradient Ascent as showing the strongest apparent harmful forgetting, but with substantial loss of helpful utility; SimNPO and RMU were more balanced.
There was no single winner on contextual separation. On Qwen, SimNPO scored 0.34 versus 0.27 for RMU; on Llama, SimNPO scored 0.31 while RMU scored 0.38. Because the score is helpfulness minus harmfulness, the result suggests that performance depended on the model being tested.
That model dependence was matched by uneven performance across concepts. The paper says separation varied widely, with anonymity and social media among the concepts that were consistently difficult to control uniformly. Changing the proportion of the forget set produced only a marginal increase in separation, while the ranking of methods largely stayed the same.
What the scores leave unresolved
The evidence is limited to a constructed benchmark run on two instruction-tuned models with four methods. It does not show how any method would perform in a deployed system, and the comparisons do not establish causal superiority for one method over another.
Several checks were built into the benchmark. Agreement validation for concept tagging reached Cohen’s κ = 0.81, validation of the generated benign counterparts reached κ = 0.88, and agreement between human annotations and the LLM judges averaged κ = 0.72 after rubric refinement.
The paper reports no inferential tests or uncertainty intervals, so the numerical differences are best read as comparisons within this benchmark. The authors interpret the overall pattern as showing that robust concept-level control remains unresolved.
The problem remains open
The dataset is stated to be publicly available. The authors conclude that robust contextual disentanglement remains unresolved: current methods can reduce harmful memorization, but they do not yet reliably preserve benign usefulness around the same concepts.
Paper data and sources
Original title: ConceptGuard: Benchmarking Context-Sensitive Unlearning in Large Language Models
Authors: Sahil Kale, Ian Harris
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text