A modeling preprint reports an orderly signal in neural networks' responses to temporary weight-decay pulses, appearing before their test accuracy visibly improved. Early in the pre-generalization plateau - the long period before visible test improvement - timing responses showed no stable relationship to pulse direction or magnitude. Around 5,000 to 5,500 epochs, that relationship became persistent: stronger increases in weight decay were associated with earlier generalization, while stronger decreases were associated with later generalization, even as test accuracy remained at chance. The authors describe the combination as increasingly constrained function selection with persistent sensitivity to the perturbation dose.
How the test worked
The test was temporary. From the complete saved training state, each branch changed the weight-decay coefficient for 500 epochs, using positive or negative perturbations with absolute sizes of 0.05, 0.10, 0.15, 0.20 or 0.25, then restored the baseline setting. Possible pulse starts were scanned every 25 epochs. Generalization was defined as the first time test accuracy crossed a chosen threshold; the response was the perturbed crossing time minus the baseline crossing time, so negative shifts meant earlier generalization and positive shifts meant later generalization.
The main analysis covered 70 baseline runs: 30 on parity matching, 24 on sparse parity and 16 on factored modular addition. Across those runs, the researchers examined 23,926 pulse starts and 239,260 intervention branches. Each start was paired with ten perturbation magnitudes, corresponding to five positive and five negative doses.
The underlying task sets were small. Task 1 used 304 training entries from 267 distinct pairs and 1,024 test samples. Task 2 used 550 training vectors and 1,000 test vectors, while Task 3 used 570 training examples and 1,708 test examples. Tasks 1 and 2 used four-hidden-layer tanh multilayer perceptrons with 128 units per layer; Task 3 used a shared 67-by-500 embedding matrix.
The signal appeared before the performance change
At the start of the plateau, up to about epoch 5,000, responses showed no stable dose ordering. The researchers analyzed positive and negative weight-decay directions separately across the five dose sizes. A chance-corrected measure of dose-response linearity rose across the 30 Task 1 runs and approached 1 near the end of the interval. Uncertainty was summarized with 95% confidence intervals from 4,000 bootstrap resamples.
The baseline relative weight norm stayed within about 4% of its fit value. Changes in that norm showed near-zero synchrony with changes in dose-response linearity: the median rank-based Spearman correlation was -0.03, with a 95% confidence interval from -0.05 to 0.01.
The qualitative transition was not confined to parity matching. Sparse parity and factored modular addition showed the same move from unordered early responses to dose-ordered timing, and the ordered timing/barrier pattern was reproduced across the 30 Task 1, 24 Task 2 and 16 Task 3 runs. That is a cross-task result within this study's three modeled cohorts, not evidence that the pattern holds across machine-learning problems generally.
Timing outlasted the barrier
To quantify the separation between training outcomes, the study calculated a test-loss barrier between baseline and perturbed checkpoints at 0.95 test accuracy. It defined the barrier as the maximum excess loss above the straight-line interpolation of the endpoint losses, sampled at 51 equally spaced points along the parameter-space path. The reported barriers declined toward zero, while directional and dose-dependent timing shifts remained ordered even when the barrier was close to zero.
After ordering emerged, the researchers compared normalized update directions using principal-component analysis, or PCA, in three clear-grokking Task 1 runs. The baseline and perturbed trajectories approximately overlapped in that projection, with their main difference being timing along a shared trajectory.
The pulse response also appeared earlier than the static measures checked in Task 1. Restricted test binary cross-entropy (BCE) began decreasing only about 50 epochs before held-out test BCE. Parameter movement changed little after early saturation, and the joint-parity Fourier fraction stayed near zero through the plateau. The WD dose structure was established roughly 4,000 epochs before the baseline crossed 0.90 test accuracy.
A useful probe with a narrow reach
The authors call this pattern the canalization of function selection. In their reading, later function selection became more constrained while the dose-ordered timing response persisted. That interpretation is based on the three modeled algorithmic cohorts; whether it extends to larger models or non-algorithmic data remains open.
Interpretation also depends on how baseline runs were chosen. For Task 1, researchers screened 1,000 runs: 177 reached final test accuracy of at least 0.95, 105 met all criteria and the first 30 qualifying runs were used. Task 2 contributed 24 qualifying runs from 500 screened runs. Task 3 used seeds 1 through 16 without additional screening.
Some perturbed branches did not reach the target accuracy within the observation limit and were treated as missing in the barrier analysis; the supplied analysis does not report how many were omitted. The document is an arXiv v1 preprint dated 26 Aug 2026. It offers a computational probe for training dynamics, while replication on broader tasks and a clearer mechanism remain open questions.
Paper data and sources
Original title: Canalization Before Generalization: Grokking as a Dynamical Probe
Authors: Yiming Lin
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text