A proposed modification to the Muon optimizer used fewer training tokens to reach a matched validation loss in a series of Transformer pretraining tests, according to a new arXiv preprint. The full version, called SAMuon, reported a 13.3% to 24.0% token-efficiency gain over tuned Muon. A lighter version, SAMuon-lite, reported a 13.3% to 22.1% gain. Both variants also recorded lower final validation loss than tuned Muon in every trained model-and-batch setting.
The study examines how much of an optimizer’s update should be spent in each spectral direction represented by its momentum buffer—roughly, whether different directions can tolerate different step sizes—and asks what that pattern implies for optimizer design. The paper places SGD, Adam and Muon on a single spectral-allocation axis, treating their reported ordering as an interpretation of those measurements.
The shape of an update
To measure the pattern, the researchers estimated the out-of-sample loss-optimal step along each spectral probe with a local-quadratic approximation. In ordinary language, they used local curvature to estimate the best step instead of carrying out an exhaustive line search for every direction. The probes came from analyses of 64M- and 124M-parameter models. Across those analyses, 33 of 6,400 rank probes, or 0.52%, were filtered after failing descent-direction or curvature checks.
The measured profile was highly uneven but stable. Most directions formed a broad, mostly flat bulk with similar tolerated step sizes, while the volatile head showed a sharp drop in the step size it could tolerate. That shape is the basis for the paper’s spectral-allocation comparison, which reports the ordering Muon, then Adam, then SGD in Transformer pretraining.
A targeted reweighting
SAMuon is built around that contrast. It keeps the volatile head at Muon’s scale while amplifying the bulk, retaining Muon’s momentum and whitening, the part of the update that normalizes directions. The full version uses rank-k randomized singular-value decomposition, or randomized SVD, to estimate the leading directions. SAMuon-lite uses a two-level power-iteration approximation instead.
For tuning, both variants used a fixed radius of 50 and tested five values for the tail-boost parameter γ: 3.54, 5, 7.07, 10 and 14.14.
Results across model sizes
The main experiments used 124M-, 300M- and 1B-parameter models trained on the FineWeb 100B-token sample. Batch sizes were 1,024, 2,048 and 4,096 sequences. Within each model-scale and batch-size setting, the optimizer runs used the same initialization and identical data ordering, with a single seed for each configuration.
Across the trained cells, SAMuon’s final validation losses ranged from 2.7114 to 3.1658, while SAMuon-lite’s ranged from 2.7117 to 3.1678. At a matched final loss, the full variant’s token-efficiency improvement over tuned Muon ranged from 13.3% to 24.0%; the lite version ranged from 13.3% to 22.1%.
SAMuon matched or outperformed SAMuon-lite in every reported setting, with a token-efficiency advantage ranging from 0.0 to 2.3 percentage points.
The compute trade-off
In a separate benchmark using a 1B model with a batch size of 1,024, SAMuon added 7.4% to Muon’s iteration time, while SAMuon-lite added 0.5%. The timing test used one GPU configuration and was separate from the training runs.
The paper also gives a mathematical convergence result, but only for idealised exact-whitening updates. Under its stated assumptions, those versions retain Muon’s stated convergence orders: O(T^-1/4) in the stochastic setting and O(T^-1/2) in the deterministic setting. The guarantee does not cover the finite-step Newton–Schulz, randomized-SVD or power-iteration approximations used by the deployed algorithms.
A result with boundaries
The main optimizer configurations used a single seed, and the authors state that the SAMuon variants were under-tuned relative to the baselines. The reported gains should therefore be read as results from the tested comparison, with uncertainty around how they would change under repeated seeds or fuller tuning.
The spectral measurements themselves rely on a local-quadratic approximation, and probes that failed descent or curvature checks were filtered. The convergence guarantee covers only idealised exact whitening, not the finite approximations used in the deployed algorithms. Taken together, the findings are evidence from the tested Transformer pretraining setup, not a universal guarantee.
The work is an arXiv preprint, version 1, dated 26 August 2026. Funding information is not reported in the supplied document.
Paper data and sources
Original title: Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon
Authors: Xiaodong Wu, Wenyi Yu, Chao Zhang, Philip Woodland
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text