A sparse language model can make moral content easy to decode while remaining unusually sensitive to activation noise, according to an arXiv preprint dated 25 August 2026. In a comparison of the base, non-instruct OLMoE-1B-7B mixture-of-experts model and the dense OLMo-2 1B model, both reached 99.0 percent peak accuracy on a moral-versus-neutral sentence task. But the main results text reported a mean critical noise level of 0.92 for OLMoE, compared with 3.81 for the dense model, describing the sparse model as 4.2 times more fragile under the study’s test.
That contrast is the study’s central finding: a high score from a probe did not tell the same story as the representation’s resistance to noise. The researchers drew a deterministic sample of 240 sentence pairs from a 1,200-pair parent dataset, retaining 192 pairs for training and 48 for testing. The probes were binary linear classifiers, meaning they separated the two categories with a simple linear decision rule, and they were evaluated on the held-out test pairs. For the fragility experiment, Gaussian noise was added to hidden states, router logits, individual expert outputs or the final aggregated output. Results at each noise level were averaged over 10 seeds, and the critical level was the first tested point at which mean accuracy fell below 0.6.
Accuracy was not the whole story
At peak probing accuracy, the two models were equally strong: 99.0 percent for each. OLMoE reached that peak at layer 13 and OLMo-2 at layer 12. Both showed decodable information from layer 0, with an encoding breadth of 1.0, but OLMoE was weaker in the first four layers, where its accuracy ranged from 79 to 86 percent versus 94 to 97 percent for OLMo-2. The gap therefore appeared early, while the peak scores later converged.
The robustness measure reversed the emphasis. Here, critical sigma meant the first tested amount of Gaussian noise at which the probe’s mean accuracy dropped below 0.6, so a smaller number marked an earlier loss of performance in this protocol. The main text placed OLMoE at or below 0.3 in nine of 16 layers, while OLMo-2 reached at least 3.0 in 12 of 16 layers. Those figures describe behavior in this experiment, not a general rule about mixture-of-experts systems.
No single expert carried the signal
The analysis then looked inside the sparse model to see whether a small group of experts carried the moral signal. It trained 1,024 expert-layer probes, and 1,020 of them exceeded 75 percent accuracy. At layer 14, all 64 experts cleared 84 percent, with a mean of 93.0 percent and a minimum of 84.4 percent. The reported Gini values, used to summarize how unevenly performance was distributed across experts, ranged from 0.016 to 0.023. The pattern was broad encoding, not a result confined to a few obvious specialists.
Routing showed little moral-versus-neutral separation as well. The largest reported difference in routing probability was 1.8 percent, and the 95th percentile was below 2 percent. At a 0.5 percent detection threshold, detectable preferences involved between three and 19 of the 64 experts. These calculations used mean-pooled inputs, which combine the sequence into a single representation rather than tracking routing at each token.
The component tests offered a clue about where fragility showed up. Noise applied to router logits was most tolerated: its mean critical sigma was 9.56, and eight of 16 layers were capped at the experiment’s maximum value of 10. Expert-output noise produced a mean critical sigma of 1.8, while noise on the final aggregated output produced 0.6. In the measured components, the final aggregation was therefore the least noise-tolerant part of the pipeline.
The authors’ proposed explanation centers on output dilution. OLMoE uses top-eight-of-64 aggregation, so only eight experts contribute for each token, and the routing weights further reduce each contribution. In a separate output-scale comparison using 100 texts, the dense model’s feedforward outputs were 74 times larger on average than the MoE outputs. The dense-to-MoE ratio exceeded 60 times at nine of 16 layers and ranged from 5.3 to 167 times. The scale calculation concerns the feedforward contribution to the residual stream, not the total residual-stream signal.
A clue, not a verdict
A check across training offered no clear sign that a stable moral specialist emerged. Across 11 checkpoints, peak-layer Gini remained between 0.012 and 0.018 while peak accuracy rose from 92.1 percent to 93.7 percent. Adjacent top-five expert lists had a Jaccard similarity of about 0.09, indicating little persistent overlap in which experts ranked highest from one checkpoint to the next. The sampled checkpoints ran from step 5,000, or 20 billion tokens, to step 1.2 million, or 5,033 billion tokens.
The authors interpret the combined results as an architectural output-dilution effect: sparse aggregation preserved moral information that remained easy for a linear probe to decode, while the smaller feedforward signal coincided with lower observed tolerance to raw activation noise. That interpretation is specific to these models and measurements. The study does not establish that other mixture-of-experts architectures will show the same fragility pattern.
Several limits narrow the result. The work studies one MoE family, uses mean-pooled representations and linear probes, compares against a dense model that is not identical in training setup, and uses English data grounded in Moral Foundations Theory. Those choices leave open how the fragility comparison would look with other MoE families, token-level routing analyses, nonlinear probes or a more closely matched dense model.
There is also an unresolved reporting discrepancy. The main text gives the 4.2-fold comparison, with mean critical sigma values of 0.92 and 3.81, while Figure 1 gives a 5.1-fold comparison, with values of 0.84 and 4.25, and places robustness in the final two layers. The direction of the reported difference is consistent, but the precise summary will need to be reconciled with the underlying outputs.
The document is an arXiv version-one preprint dated 25 August 2026. Its disclosure says Anthropic Claude was used for code scaffolding, experimental scripts and prose drafting, while responsibility for the study design, claims and final wording remained with the author.
Paper data and sources
Original title: Output Dilution: Redundant but Fragile Representations in MoE Models
Authors: Orion Reblitz-Richardson
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-25
DOI: Not available
Original paper · Full text