A new machine-learning preprint reports higher average benchmark scores for a sparse model-merging method in two Qwen2.5 experiments. In the 7B setting, the proposed system scored 68.49 on average, versus 67.71 for Task Arithmetic. In a high-conflict four-task test using the 1.5B model, it scored 36.48, compared with 33.70 for DELLA and 29.53 for TIES-Merge. Those are reported averages, and the main comparisons do not include confidence intervals, p-values or formal significance tests.
A selective merge
The pipeline extracts task vectors from expert models and uses GR-ZOO to identify task-critical layers. It merges non-critical layers arithmetically, then applies SAE-based sparse feature projection and fusion to the selected layers. In ordinary terms, the sparse representation keeps many feature positions inactive, allowing task signals to be handled in a larger feature space. The broader proposal is to move task vectors into that higher-dimensional sparse space before fusion, reducing the overlap that may cause one capability to interfere with another.
The paper's theoretical section sets a boundary on what this representation change can guarantee. It states that orthogonal decomposition cannot remove a conflict caused by superposition while preserving both interacting capability directions unless at least one direction is substantially distorted or discarded. The result rests on assumptions about linearization, reachability, sparsity and dictionary coherence, so it does not by itself settle how the method will behave across other architectures or task mixtures.
The benchmark gap
The benchmark work used two model-scale configurations. One used Qwen2.5-7B with three experts for Math, Code and Instruction Following. The other used Qwen2.5-1.5B with four experts for Math, Code, General Knowledge and Safety Alignment. These group sizes refer to task-specific expert models, not independent human observations. For the 7B setup, the supplementary configuration reports common-base full-parameter supervised fine-tuning, with approximately 7,160 Math samples, 8,646 Code tasks and 5,473 Instruction Following samples.
The 7B test also gave a view of the task-by-task profile. The proposed merge scored 85.22 for Math, 74.40 for Code and 45.84 for IFEval, alongside its 68.49 average. These figures describe the reported benchmark profile for that model and task mix.
In the four-task setting, the method's average of 36.48 led DELLA at 33.70 and TIES-Merge at 29.53. The abstract describes the difference from the strongest baseline as a 2.78% improvement. The supplied analysis notes that the paper does not clarify whether this means a relative percentage or a percentage-point difference.
The mechanism is not the whole story
GR-ZOO was tested as the mechanism for deciding where to spend the extra processing. Its selected layers reached recovery rates of 62.31% for Math and 41.46% for Code, close to full-gradient upper bounds of 65.73% and 46.34%. The selector also outperformed random selection, although no uncertainty estimate was reported for these recovery rates.
The ablation results favored the complete SAE design on aggregate. Its reported average was 40.18 versus 39.30 in the three-task setting, and 36.48 versus 35.24 in the four-task setting. Representation diagnostics pointed to less shared activity than in raw task vectors: pairwise active overlap was 0.0983 versus 0.3368, active Jaccard similarity was 0.0518 versus 0.2027, and the task-specific active ratio was 73.84% versus 29.40%. In ordinary language, the SAE representation showed more task-specific activity under these measures.
That pattern was not uniform across individual benchmarks. In the L1 SAE comparison, the complete variant scored 32.32 on HumanEval versus 32.93 for the comparator, and 50.55 on BeaverTail versus 52.97. The averages therefore do not amount to a win on every metric.
Cost and reach
The extra machinery has a cost. In the reported setup, 197 layers were candidates, 16 were selected and 8.12% received SAE processing. Cold GR-ZOO took 467.13 seconds, while training and encoding a single-layer SAE took 366.75 seconds; peak GPU memory was 25,153.93 MB. The authors identify SAE training overhead relative to arithmetic merging and say the expansion factor and cosine-similarity threshold may need calibration across architectures and task scales.
The work relies on publicly available evaluation datasets, collects no new human-subject data, identifies no individuals and reports aggregate benchmark metrics. The experiments described here use the listed Qwen2.5 configurations and selected benchmarks, leaving transfer to other architectures, model scales and broader task mixtures as an open question. The document is an arXiv preprint, version 1, dated 26 Aug 2026.
Paper data and sources
Original title: Escaping Low-Dimensional Overlap: Multi-Task Model Merging via High-Dimensional Sparse Disentanglement
Authors: Yihang Zhang, Shengke Sun, Junjie Wen, Feng Zeng
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text