A preprint reports that FedAvg and SCAFFOLD showed similar sharpness-related training dynamics in controlled experiments. Yet their results diverged by task: in CIFAR-10 settings where Edge-of-Stability dynamics occurred, the algorithms had comparable losses, with FedAvg slightly lower. In MNIST settings without those dynamics, SCAFFOLD consistently outperformed FedAvg and achieved near-zero loss.
The comparison addresses why FedAvg can match or exceed SCAFFOLD in practice despite theoretical expectations that favor SCAFFOLD. The findings point to an association between sharpness and SCAFFOLD’s update behavior, but they do not establish a general ranking between the algorithms or show that high sharpness causes weaker performance for SCAFFOLD.
Following the shape of training
The study investigated Edge-of-Stability dynamics for both methods. In this context, progressive sharpening means that measured local curvature in the loss landscape rises during training; Edge of Stability describes the associated training regime. Both FedAvg and SCAFFOLD exhibited these patterns across the tested architectures and several learning rates.
Sharpness was tracked through the leading eigenvalue of the Hessian, a measure of local curvature. The researchers estimated that value with the Lanczos method, using automatically differentiated Hessian-vector products.
The standard experiments used 5,000 images from each dataset, divided among eight clients. The experiments varied the learning rate, label-skew heterogeneity, communication interval, network depth and dataset size.
Learning rate mattered more than communication timing
The reported equilibrium sharpness—the level at which the measure settled—was inversely proportional to the learning rate. In the comparison described by the authors, raising the learning rate from 0.005 to 0.01 halved the equilibrium value. The study did not derive a closed-form rule that would predict equilibrium sharpness for either algorithm.
The reported association with label-skew heterogeneity differed between the two methods. For FedAvg, equilibrium sharpness was about 550 at h = 0 and about 350 at h = 0.9. SCAFFOLD was less influenced overall, although its behavior became erratic at h = 0.9. The authors said the pattern was not captured by a simple formula in h.
Communication interval showed almost no observed relationship with equilibrium sharpness in the tested experiments. Within those conditions, communication frequency was not associated with a material shift in the sharpness level at which training settled.
The signal that differed between the algorithms
The researchers also introduced update misalignment, a measure of how closely an algorithm’s update matched the ground-truth global gradient. SCAFFOLD’s update misalignment was highly correlated with sharpness, while FedAvg’s was essentially unrelated to sharpness. No confidence interval or p-value was reported for that correlation, so the result is an experimental association rather than a precise estimate of a universal effect.
The authors interpret the pattern as a possible sign that progressive sharpening makes SCAFFOLD’s historical-gradient correction unreliable at high sharpness. They say FedAvg’s update misalignment appeared less affected in the tested settings. This proposed mechanism may help explain the loss comparison, but the experiments do not establish that high sharpness causes weaker performance for SCAFFOLD.
The contrast between the datasets led the authors to propose that dataset complexity and the resulting sharpness may matter more than network depth alone. That interpretation does not show that depth or nonconvexity is irrelevant in every setting, and it does not show that SCAFFOLD generally underperforms FedAvg across federated-learning applications.
What the experiments leave unresolved
The evidence comes from controlled computational experiments using selected image datasets, small MLPs and CNNs, full-batch gradients, label-skew heterogeneity, and the tested FedAvg and SCAFFOLD implementations. The study mainly focused on full-batch FedAvg and SCAFFOLD for CNN image-classification tasks, and it did not determine equilibrium sharpness in closed form.
Stochastic gradients, Transformers, larger image datasets and language tasks were not tested. The researchers used smaller architectures because of hardware constraints, and training runs were generally single trajectories without formal uncertainty estimates. The heterogeneity was induced through an ad hoc procedure controlled by h.
Those limits leave open whether the sharpness-misalignment relationship persists with other architectures, larger datasets and language tasks; which properties of the loss landscape determine equilibrium sharpness; whether a closed-form characterization can be derived; and which changes might preserve SCAFFOLD’s heterogeneity benefits when Edge-of-Stability dynamics occur. The current evidence supports associations in controlled laboratory settings, not a settled rule for federated optimization.
The document is an arXiv version marked v1 and dated 26 August 2026; journal publication was not reported. The paper acknowledges computing support, an internship program, a doctoral scholarship and National Science Foundation grants 2436217, 2425687 and 2601681.
Paper data and sources
Original title: How Edge of Stability Hinders SCAFFOLD in Federated Optimization
Authors: Anant Khandelwal, Michael Crawshaw, Mingrui Liu
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text