A small, targeted set of neurons in a language model stood out as closely tied to particular skills in suppression tests. In one experiment, the reported top-1% MLP-suppression row, involving units in a feed-forward module, recorded accuracy of 3.17% on the MBPP+ code benchmark and 1.22% on HumanEval+, a separate code test. Several other benchmarks retained much higher scores: 85.41% on MATH-500, 75.11% on AMC, 82.35% on MMLU-Redux and 52.53% on GPQA.
The result comes from a computational arXiv preprint, not a study of people. It tests whether a method called RACE can rank neurons by how consistently their activity aligns with a domain, then compare target and non-target performance after selected activations are suppressed.
How the ranking works
RACE uses Residual-Direction Alignment to generate signed evidence from each observation of a neuron's contribution to the model's residual stream, its running internal signal. Bayesian aggregation then estimates a posterior mean alignment and variance. The experiments also tested CAM-based Bayesian uncertainty scoring and used Reference-Set Filtering with WikiText-2 in the reported code and math experiments.
The experiments covered three language models: Qwen3-4B-it-2507, OLMo-3.1-32B-it and Llama-3.1-8B-it. The code tests used MBPP+ as the target and HumanEval+ for same-domain validation. The math tests used MATH-500 as the target and AMC for the corresponding validation. The study also compared RACE with GxAct and AttnLRP, gradient-based attribution methods, and with an activation-only control.
For the autoregressive audits, generated-token positions were included in the observation set rather than collecting evidence only from the first output token. The reported observation counts were 98,560 token positions for MBPP+ and 705,426 for MATH-500. These were observations used for neuron scoring, not a participant sample.
The pattern also appeared in mathematics
In the math-domain experiment, the reported reference-filtered top-1% MLP-suppression row recorded MATH-500 accuracy of 29.00% and AMC accuracy of 11.94%. Scores on the other listed tests were 70.20% for MBPP+, 67.49% for HumanEval+, 77.79% for MMLU-Redux and 40.40% for GPQA. The reported Intervention Specificity Index was 1.76, compared with 2.65 in the code-domain experiment.
A separate comparison of the model's output distributions reported relative perplexity degradation reaching 77.31% on MBPP+ and 129.04% on MATH-500 after suppression, while the WikiText-2 reference corpus was described as almost entirely unperturbed.
A narrower behavior changed, too
The researchers also examined a specific coding pattern rather than overall benchmark accuracy. Neurons selected using PyComp-1K were associated with a reported 70.9% reduction in comprehension usage on MBPP+ and a 61.1% reduction on HumanEval+, while functional correctness was largely preserved.
RACE was also much cheaper to score than the gradient-attribution comparisons under the study's profiler setup. The reported extra cost was 81.5 GFLOPs and 0.63 times a forward pass for RDA, against 18.77 TFLOPs and 144.29 times for GxAct, and 18.82 TFLOPs and 144.62 times for AttnLRP. The comparison excluded several processing and storage costs, so it is not a complete deployment-cost estimate.
The protocol matters
The striking code result depended heavily on where the model was observed. With the same per-layer top-1% MLP suppression, collecting evidence only from the first generated token produced MBPP+ accuracy of 70.37%, compared with 3.17% when all generated-token positions were included. HumanEval+ showed the same contrast: 76.83% with first-token-only observations versus 1.22% with all generated-token positions.
The rankings became more stable as the scoring set grew. With 100 examples, overlap with the full-data top-1% selection was 0.82 for MLP neurons and 0.93 for attention neurons. At scoring sets of 200 or more examples, all reported overlaps exceeded 0.89 for MLP and 0.94 for attention.
The selected neurons were not equally shared across domains. At the CAM top-1% threshold, all-domain Jaccard overlap, a measure of how much the selected sets shared, was 0.264 for attention neurons and 0.085 for MLP neurons, with standard deviations of 0.099 and 0.052 respectively. The authors report a 3.1-fold gap, indicating more cross-domain reuse in attention channels within this analysis.
What the result does not establish
The evidence is computational and model-specific. The activation-suppression tests show what happened in the tested models and datasets, but they do not establish that the selected neurons are uniquely necessary or sufficient for a capability, or that the findings generalize beyond the tested benchmarks and corpora.
The authors say the linear residual-stream projections used by RACE may miss nonlinear cooperation among neurons and distributed polysemantic features. They also report weaker domain-specific effects in attention modules than in MLPs. Whether nonlinear representations, attention-specific auditing and reference filtering can improve the method remains open, as does whether the findings transfer to permanent pruning or other production interventions.
The work is an arXiv preprint, and the document says code is available through the RACE GitHub repository. Its acknowledgements report support from the National Science Foundation of China, the Key Research and Development Project of Nantong City, and the Key Laboratory of Target Cognition and Application Technology.
Paper data and sources
Original title: RACE: Scalable Statistical Estimation of Functional Consistency in LLM Neurons
Authors: Runyu Wang, Bo Liu, Xiaxin Zhang et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-25
DOI: Not available
Original paper · Full text