An arXiv preprint reports an association between QLoRA rank and how language models balance new factual learning against retention of existing capabilities. In the main Qwen3-4B experiments, higher ranks were linked to stronger generalization of anonymized facts across reworded questions, but lower retention on unrelated benchmarks. Rank 64 sat in what the study describes as a high-plasticity, low-retention regime.
The contrast was large in the final OOD averages. Full fine-tuning and QLoRA at rank 8 both had average scores of 70.7, with standard deviations of 0.7. The figures were 63.8 with a standard deviation of 1.7 at rank 16, 48.0 with 3.6 at rank 32 and 28.1 with 6.3 at rank 64. These were means and standard deviations across five random seeds.
The benchmark tested fact generalization
The factual task used a controlled, anonymized geographic benchmark derived from OpenStreetMap. It contained 1,938 training examples and 900 held-out evaluation examples. The study asked whether a model could carry a fact across a rephrased question, rather than simply reproduce a training formulation.
Researchers measured acquisition with exact-match accuracy on both the training questions and their paraphrases. Forgetting was defined as the pretrained base model’s average OOD score minus the adapted model’s average score. OOD, or out-of-distribution, refers here to tests of capabilities unrelated to the newly learned facts.
The main comparison used Qwen3-4B, full fine-tuning and QLoRA ranks 8, 16, 32 and 64. Each configuration was repeated over five random seeds, and training loss was applied only to answer tokens.
The trade-off appeared in the scores
At checkpoints selected near a 95% paraphrase target, the higher-rank QLoRA settings generally paired stronger acquisition with less retention. Rank 16 reached 93.3% paraphrase accuracy with 89.9% OOD retention; rank 32 reached 94.6% with 73.3% retention; and rank 64 reached 90.4% with 38.8% retention. Full fine-tuning reached 76.1% on paraphrases while retaining 96.9% of OOD performance.
The loss of OOD performance was broad across the benchmark suite, although TruthfulQA remained comparatively stable.
A smaller control showed a similar pattern
A separate standard-LoRA control pointed in the same qualitative direction. It used unquantized Qwen3-1.7B models at ranks 8, 16 and 32 on the same OSM task and OOD suite, but it changed the model scale from the main Qwen3-4B comparison.
In that control, paraphrase exact match rose from 76% at rank 8 to 79% at rank 16 and 86% at rank 32. Average OOD performance fell from 57.0% to 52.0% and then 40.2%.
Mathematics produced a weaker pattern
The study also ran a separate adaptation experiment on mathematics, using Qwen3-4B and a 94,000-example subset of open-r1-math-220k for one training epoch. It compared full fine-tuning with QLoRA at ranks 16 and 32.
Average math performance was 42.50% with full fine-tuning, 42.60% with rank 16 and 42.03% with rank 32. The corresponding OOD drops were 1.71, 2.23 and 1.58 points. The math comparison showed a much weaker rank-dependent frontier than the OSM test, but it covered only ranks 16 and 32 and one training epoch.
Model diagnostics tracked the losses
To examine what accompanied the benchmark changes, the researchers added diagnostics beyond accuracy. They measured symmetric KL divergence, teacher-forced answer negative log-likelihood, RMS-normalized dense weight drift and SVD-based intruder dimensions.
Configurations with stronger OOD degradation also showed larger model drift. Higher-rank QLoRA had larger KL and dense-update measures, while rank 64 had the largest reported SVD intruder excess. The pattern is an association between rank, drift and retention in the tested models; it does not establish that rank alone caused the loss of retention.
The evidence remains narrow
The central factual benchmark contains 1,938 training examples drawn from 14 small cities. The authors caution that the result may not carry over to larger or more diverse factual corpora.
The standard-LoRA control also changes model scale, using Qwen3-1.7B rather than the Qwen3-4B model in the main comparison. Taken together, the findings describe the reported Qwen configurations and benchmark, not a universal rule about language-model adaptation.
The reported OSM runs put new-fact acquisition and unrelated-capability retention side by side. Lower-rank settings were associated with better OOD retention but weaker same-fact generalization, while higher-rank settings reached stronger paraphrase performance alongside greater model drift.
The work is an arXiv preprint, version v2 dated 27 Aug 2026. The paper states that its code and data are available at https://github.com/zhngstl/new_facts_forgetting.
Paper data and sources
Original title: Learning New Facts with QLoRA: An Acquisition-Retention Frontier
Authors: Estelle Zheng, Sébastien Warichet, Emmanuel Helbert, Christophe Cerisara
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text