A preprint study reports a two-step procedure for carrying learning-rate settings from smaller proxy models to much larger mixture-of-experts (MoE) language models, then extrapolating those settings across training-token budgets. Applied to its proxy experiments, the procedure predicted a learning rate of 3.85 × 10−4 for a 10-trillion-token pretraining run.
The result comes with a clear qualification. The 10-trillion-token target was not put through an exhaustive full-scale learning-rate sweep, which the authors say would be computationally infeasible. Stable training loss and competitive benchmark results support the selected setting, but do not definitively show that it was the best possible one.
Here, the learning rate is simply the training setting being tuned. The method adapted µP to MoE architectures with MLA and the Muon optimizer for the first step, then used a predictive scaling law to extend the transfer across token budgets.
The models were tracked by total and active parameter counts as they grew. The MoE scaling path kept the number of experts active for each token and the expert intermediate dimension fixed while increasing total experts and hidden dimension. That coupled width, active scale and sparsity, rather than changing width in isolation.
The first test: wider models
The width-transfer experiment began with a base proxy whose hidden dimension was 256, with 16 total experts and four attention heads. The models were scaled to 2×, 4× and 8× width, and all were trained for 1.3 billion tokens on a general-knowledge English corpus.
To estimate the preferred setting, the researchers swept discrete learning rates, selected the lowest-loss region and fitted a second-order polynomial to validation loss against log-transformed learning rate. They used exponential moving average, or EMA, checkpoints with α = 0.6; checkpoint updates came at approximately 2-billion-token intervals, with analysis checkpoints every 10 billion tokens.
Under µP, the fitted optimum stayed aligned with the base proxy across the 2×, 4× and 8× width scales. Under standard parameterization, or SP, the comparison approach, the optimum shifted as width increased.
The comparison covered a base model with 0.6 billion total and 0.3 billion active parameters, followed by 2×, 4× and 8× versions with 2.2/0.7, 8/1.5 and 30.7/3.6 billion total/active parameters, respectively.
Following the token curve
The researchers then tested the token dimension. A 5.6-billion-total/1.8-billion-active proxy was compared with a held-out target at twice the proxy width, with 20.7 billion total and 3.8 billion active parameters. The runs extended to 100 billion tokens.
In that experiment, the estimated optimal learning rate declined slightly as the token budget increased, while the fitted loss curves stayed aligned between the proxy and held-out wider model. That supplied the empirical link between width transfer and token-budget extrapolation in the two-step framework.
For the longer extrapolation, the analysis used a 10.8-billion-total/3.3-billion-active proxy and optimal learning-rate estimates from 255 billion to 502 billion tokens. A log-log regression—fitting the relationship after both quantities are put on logarithmic scales—achieved R² = 0.95 and predicted 3.85 × 10−4 for 10 trillion tokens.
Retrospective held-out checks offered a partial test: predicted optima were compared with independently fitted optima and differed by approximately 4.4% on average. The check showed close agreement in the held-out comparison, but it was not a direct test of the final 10-trillion-token optimum.
R² describes how closely the regression fit the available proxy observations; it does not verify the optimum at the 10-trillion-token horizon. The predicted value therefore remains a model-based extrapolation, rather than a result from testing the final system across a full range of learning rates.
What the compute bill shows
The compute accounting shows the trade-off behind the method. Model-scale search added 240.3 ZFLOPs beyond a 64.8-ZFLOP proxy run, while the target computation was approximately 98 times the sum of the proxy runs.
Those figures do not erase the cost of large-model training. The search itself added substantial computation, and the target run remained far larger than the combined proxy work.
A full-scale application
The framework was then applied to a foundation MoE model with 155 billion total and 17 billion active parameters over a 10-trillion-token horizon. Its reported Stage 1 training loss was stable and showed no spikes.
The authors also reported a Pareto-frontier position, meaning a favorable balance in the comparison between benchmark performance and estimated training compute. The model had higher MMLU-Pro accuracy than dots.llm1 and GLM-4.5-Air at comparable or lower estimated training compute.
Exact benchmark scores and detailed matched-comparison figures were not reported in the supplied analysis. The comparison therefore does not establish universal superiority over the two alternatives.
Likewise, the stable loss and competitive scores support the chosen 3.85 × 10−4 setting, but they do not settle whether it was globally optimal. The authors state that exhaustive full-scale verification was computationally infeasible.
Where the evidence stops
The central methodological limit is that sparsity expanded jointly with width in the large-scale recipe. The experiments therefore cannot isolate how much of the observed transfer came from sparsity and how much came from width; the authors call for a controlled sparsity-only study.
The evidence is also tied to a specific implementation path: µP adapted to MoE architectures with MLA and Muon, followed by the token-scaling fit. That makes the results directly relevant to the tested combination, but does not by itself establish the same transfer for every MoE architecture or optimizer.
Taken together, the study supports a narrower conclusion than a universal recipe. The framework produced close retrospective agreement between predicted and independently fitted optima and accompanied a stable large-model run, while the 10-trillion-token learning rate remains a prediction and the width-and-sparsity effects remain entangled.
Paper data and sources
Original title: Let's Scale Step by Step: Compute-Efficient Hyperparameter Transfer for Large-Scale Mixture-of-Experts
Authors: Nayeon Kim, Hojin Lee, Yunju Bak et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text