A computational comparison of pruning methods found a clear ordering in the main gemma-2-2b test: at 50% sparsity, the MAGNITUDE condition showed the most severe degradation in sparse-autoencoder measures, WANDA was more robust, and SparseGPT stayed closest to the dense baseline across eight metrics and 26 layers.
The sharpest gaps appeared in two intervention measures, Spurious Correlation Removal (SCR) and Targeted Probe Perturbation (TPP). In the worst-affected layers, SCR under MAGNITUDE was up to 60% below the dense baseline and TPP up to 80% below it, while WANDA and SparseGPT remained within 15% to 25% of baseline.
A fixed lens for a pruning experiment
The study asks when a sparse autoencoder trained on a dense language model remains valid after pruning without retraining. In plain terms, the SAE is the fixed feature-analysis model used to examine patterns in the model's internal activations. Its pretrained weights were held fixed to isolate pruning from variation in SAE training.
Four model-SAE pairs were examined: pythia-70m, gemma-2-2b, gemma-2-9b and mistral-7b. Gemma-2-2b received the main layer-by-layer analysis because pretrained SAEs covered layers 0 through 25.
The dense checkpoints were pruned after training with MAGNITUDE, WANDA and SparseGPT, with no retraining or weight updates. WANDA and SparseGPT used 128 OpenWebText samples for calibration. The comparisons used 50% sparsity as the main setting, with 25% and 40% also tested.
On gemma-2-2b, each pruned variant was evaluated on residual outputs from layers 0 through 25 using four SAEBench categories. The eight reported measures covered KL divergence score, CE loss score, explained variance, cosine similarity, MSE, Feature Absorption, SCR and TPP. SCR and TPP used k=10, and the study reported both raw values and percentage changes from the dense baseline.
The broad score could hide a narrower failure
The broad scores did not always reveal what happened in feature-level interventions. In middle layers under MAGNITUDE, KL divergence scores remained at least 0.96, yet SCR and TPP collapsed. The study presents this as a warning that aggregate Core scores alone can miss damage to latent-level interpretability.
Sensitivity was uneven across depth. Middle layers 9-17 showed greater sensitivity than early layers 0-8 or late layers 18-25 at all three tested sparsity levels—25%, 40% and 50%—and the profile rose monotonically as sparsity increased.
An empirical perturbation-energy measure, ε2, followed the same method ranking at 50% sparsity on gemma-2-2b: SparseGPT was lowest, WANDA next and MAGNITUDE highest in early, middle and late layer groups. In that order, the group averages were 26.66, 31.87 and 57.36 for early layers; 20.70, 21.16 and 47.13 for middle layers; and 24.95, 26.45 and 53.69 for late layers.
The theoretical framework treats that energy as part of an upper bound on expected SAE reconstruction degradation. It separates intrinsic SAE error from pruning-related perturbation energy and includes an interaction term; because it is an upper bound rather than an exact equality, it does not by itself predict an exact amount of degradation. No p-values, confidence intervals or formal inferential tests are reported.
Changing where sparsity is placed
The study also tested whether sparsity could be distributed differently across layers. On gemma-2-2b at 50% average sparsity, the layer-wise schedule used 20% sparsity in early layers, rose to 60% at the middle layer and stayed at 60% afterward. Perplexity was lower under this schedule than under uniform sparsity: 148 versus 212 for WANDA, and 98 versus 141 for SparseGPT, with the layer-wise figure first in each pair. This was a preliminary evaluation on one model and two pruning methods, so it does not establish an optimal allocation rule.
What the comparison leaves open
Across models with valid SAE evaluations, the reported Core-metric ordering was SparseGPT, then WANDA, then MAGNITUDE. For pythia-70m, the KL score was 0.926 at baseline, 0.642 under MAGNITUDE, 0.880 under WANDA and 0.882 under SparseGPT. Feature Absorption was not valid for pythia-70m, and mistral-7b had limited layer coverage.
The broader conclusion remains limited. The comparison covered four model-SAE pairs, centered on residual-stream SAEs, and kept SAE weights fixed; it did not test whether retraining an SAE on pruned activations would recover features consistent with those from the dense model. The results therefore do not establish generality beyond the evaluated settings or show that the layer-wise schedule is optimal.
The abstract states that code is publicly available at sae-robustness-under-pruning.
Paper data and sources
Original title: When Pruning Meets Interpretability: Preserving Sparse Autoencoder Robustness in LLMs
Authors: Suchit Gupte, Xueru Zhang, Mohammad Mahdi Khalili
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text