The pattern depended on the model
A preprint reports that the lightweight Linear Residual Adapter condition had higher average benchmark accuracy than pruned-only models in most tested model-criterion comparisons. The pattern was not uniform: Vicuna-7B-v1.5 showed smaller gains that were occasionally negative, and one random-layer condition recorded lower average accuracy with LRA than with Random alone.
The main no-fine-tuning comparison covered four language models at 25% depth pruning. It averaged zero-shot accuracy across seven benchmarks and compared the Original model, the pruned-only Base version and the +LRA condition. On Qwen2-1.5B and Qwen1.5-7B, the +LRA condition had higher average accuracy under every listed criterion, with reported differences of up to 2.03 and 6.88 percentage points. For Llama-3.1-8B-Instruct, the corresponding differences were 15.74 points under Reverse-order∗ and 15.71 under Taylor.
Vicuna-7B-v1.5 did not follow the same pattern. Its reported differences were smaller and occasionally negative in the main comparisons.
How the correction works
SHIFT-LLM places a Linear Residual Adapter at each pruned layer. The adapter preserves the original identity pathway and fits a lightweight affine residual correction from calibration activations by closed-form regression without gradients. It also supports low-rank factorization, exact merging across consecutive pruned layers and combination with parameter-efficient fine-tuning.
The evaluation used five decoder-only language models: Qwen2-1.5B, Qwen1.5-7B, Qwen2.5-14B, Llama-3.1-8B-Instruct and Vicuna-7B-v1.5. The default adapter rank was 64, and the affine correction was fitted from 256 samples from the C4 validation split. The study tested six layer-selection strategies and seven standard zero-shot benchmarks. It used full benchmark splits without subsampling and kept evaluation data separate from calibration and fine-tuning.
Qwen2.5-14B was reported in a narrower scaling analysis covering two criteria. At 25% depth pruning, the +LRA condition had average zero-shot accuracy of 56.32 versus 53.34 for Base under Block Influence, a difference of 2.98 points. Under Reverse-order∗, the corresponding figures were 52.08 and 46.72, a difference of 5.36 points.
Other tests pointed in the same direction
The separate WikiText-2 language-modeling evaluation reported lower perplexity in the +LRA condition under both tested criteria for Qwen2-1.5B, Qwen1.5-7B and Llama-3.1-8B, and under Block Influence for Vicuna-7B. Vicuna-7B under Reverse-order∗ was the stated exception.
Additional comparisons involved fine-tuning and were separate from the primary training-free setting. During LoRA fine-tuning, the +LRA condition had higher average zero-shot performance by 1.41 points for Qwen2-1.5B, 0.77 for Qwen1.5-7B, 1.14 for Llama-3.1-8B-Instruct and 0.72 for Vicuna-7B-v1.5. With partial-layer fine-tuning, the reported differences were +0.98 for Qwen1.5-7B, +2.58 for Llama-3.1-8B-Instruct and +1.97 for Vicuna-7B-v1.5, while Qwen2-1.5B was lower by 0.58 points.
The measured cost stayed close to pruning alone
The efficiency check was measured on a single V100. A rank-64 adapter required less than 0.25% of a Transformer block's computation. At 25% pruning, measured latency after exact merging was 44.9 milliseconds, close to 44.7 milliseconds for the pruned-only model, while the reported speedup over the original was 1.27×.
In the calibration-size ablation, the +LRA condition was above the pruning-only baseline at every tested sample size. Performance improved only slightly beyond 256 samples and quickly saturated, so the main experiments used 256 samples. Under the same 256-sample C4 budget, SHIFT-LLM had higher average zero-shot accuracy than ReplaceMe (L2) across all evaluated model-pruning combinations.
In a Navigation LLM comparison, average accuracy was 59.08 before LRA and 59.54 with +LRA. The pruning strategy, fine-tuning protocol and hyperparameters were held unchanged.
A negative result remains part of the picture
Random layer selection supplied a negative result. For Qwen2-1.5B at 25% depth pruning, average accuracy was 36.07 with LRA, compared with 38.22 for Random alone. This was a single reported model and pruning condition.
These were descriptive, model-level experiments, and no p-values, confidence intervals or other inferential tests were reported. The evaluation covered a limited set of model families, pruning criteria and benchmark tasks, with the main comparison focused on one pruning ratio. Qwen2.5-14B was examined only in the narrower scaling analysis. Runtime was measured on one V100, and reported per-block FLOPs excluded quadratic attention. The supplied analysis also notes inconsistencies among some reported table averages.
The document is labeled a preprint. Its results describe a lightweight correction associated with higher accuracy in many reported pruned-model conditions while remaining close to the pruned-only latency in the measured test. The evidence does not establish universal accuracy recovery across models and pruning criteria, benefits for untested model families or end-to-end deployment speedups beyond the reported V100 measurement.
Paper data and sources
Original title: SHIFT-LLM: Distribution Shift Correction in Depth-Pruned LLMs
Authors: Ali Bahri, Hang Li, Hongliang Li, Zhitang Chen
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-25
DOI: Not available
Original paper · Full text