An arXiv preprint reports that CoDe-LoRA, a replay-free method for teaching language models a sequence of tasks without revisiting earlier training data, produced higher benchmark scores than comparison systems and showed no measured loss on earlier tasks after later training in its TRACE experiments. On T5, it averaged 80.4 ± 0.2 on Standard CL and 79.1 ± 0.7 on the Large Number of Tasks benchmark. The cited comparison scores were 80.0 for MTL on Standard CL and 72.4 for N-LoRA on the Large Number of Tasks benchmark. The manuscript is dated 6 October 2026.
The problem with strict separation
The result addresses a basic problem in continual learning, where a model takes on tasks one after another. The paper asks whether strict orthogonality, the practice of keeping changes for a new task separate from earlier ones, can limit useful sharing across related tasks. CoDe-LoRA uses two branches: adaptive null-space projection for transferable representation consolidation and prototype-based expert decoupling for task-specific specialization. The design is intended to preserve shared representations while retaining task-specific expertise without replay.
How the model chooses an expert
Routing is task-agnostic: it uses the frozen backbone to construct a prototype for each task from 10 randomly selected training samples. Those prototypes are used to separate or select task-specific experts as the sequence unfolds. The study kept that 10-sample construction fixed throughout the experiments.
Across models and task sequences
The evaluation covered four model backbones and three continual-learning benchmarks. Its primary measures were average performance and backward transfer, a measure of how earlier-task performance changes after later training, with forward transfer analyzed as an additional measure of what later tasks might gain from earlier learning. Results were averaged over three independent runs. CoDe-LoRA outperformed every baseline on every tested backbone and benchmark in the reported comparisons. On Qwen3-0.6B, its margin was 9.0 points on Standard CL, where it scored 79.7 against 70.7, and 3.7 points on the long-sequence benchmark, where it scored 73.0 against 69.3.
Transfer and retention
On TRACE, the generative-task benchmark in the evaluation, CoDe-LoRA's backward-transfer score was 0.0 on all three tested backbones. The strongest baseline lost between 1.3 and 4.3 points, while MoLE-CIE lost as much as 17.8. A separate forward-transfer analysis also favored CoDe-LoRA: it was consistently higher than O-LoRA, including a reported 19.1-point advantage on Yahoo after training on dbpedia and amazon.
What the mechanism check found
The mechanism check addressed the paper's central trade-off. With the orthogonality penalty set at 0.5, prior-subspace energy, a measure of how much new learning occupies directions used earlier, was 0.47% to 0.64%. When the penalty was removed, it was 5.05% to 5.63%. Forward transfer was 43.70 with the penalty and 49.45 without it, a reported difference of 5.75 points. The comparison used one fixed task order, schedule and seed, so it shows an association in that test rather than a general rule.
The branches did not contribute equally
An ablation of the two branches showed that they did not contribute equally. De-LoRA accounted for most of the reported accuracy, while Co-LoRA alone was the weakest branch. The full method added 0.4 to 2.8 points on Standard CL and Long Sequence. On TRACE, however, it matched De-LoRA.
The cost of the extra machinery
The efficiency trade-off was modest in training but less favorable at inference. In the reported table, CoDe-LoRA took 98.4 seconds to train and 43.1 seconds for inference, compared with 108.8 and 30.1 seconds for O-LoRA. N-LoRA's training time was 98.0 seconds, and CoDe-LoRA stored 9.4 million adapter parameters. The table used a single CoDe-LoRA seed, while the other methods' times were means over three seeds, making this comparison less even than the benchmark averages.
Paper data and sources
Original title: CoDe-LoRA: Mitigating the Orthogonality Dilemma in Continual Learning of LLMs via Knowledge Consolidation and Decoupling
Authors: Maoqi Liu, Quan Fang, Yufei He
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-10-06
DOI: Not available
Original paper · Full text