An arXiv preprint reports that HSLM had the lowest reported computing time per iteration in three synthetic neural-network comparisons. The authors reported similar final training errors and comparable convergence behavior for HSLM, classical Levenberg-Marquardt and Krylov-subspace Levenberg-Marquardt. But HSLM did not have the lowest iteration count in every comparison. The finding therefore concerns the cost of an individual optimization step, not a demonstrated faster optimization run overall.
The document is an arXiv preprint, version 1, dated 26 August 2026.
A smaller space for each step
HSLM builds a low-dimensional subspace from the gradient, information from recent steps, Krylov-subspace directions and randomized curvature information. It then computes a spectrally damped Levenberg-Marquardt step within that space. A deterministic adequacy monitor enriches the subspace as needed, while Armijo backtracking selects the step length and a reduction ratio updates the damping.
The guarantee comes with conditions
The paper's global convergence analysis is conditional. It assumes that the objective is bounded below on the initial level set and that the gradient is Lipschitz continuous there, meaning its rate of change is controlled. The analysis also requires enforced projected-gradient adequacy and uniformly bounded, positive-definite conditioning in the reduced problem.
Under four stated assumptions, the analysis says HSLM either terminates at a stationary point or has gradient norms whose limit inferior is zero. That conclusion applies only when those assumptions hold.
The local analysis imposes further conditions: a strict local minimizer with positive-definite curvature, subspace and curvature errors that vanish as the iterates approach the solution, vanishing regularization, and a Hessian approximation error that decreases at least linearly. If full steps are eventually accepted, the theorem gives linear convergence when its error measure is eventually bounded by a fixed value strictly below 1, and superlinear convergence when that measure tends to zero.
The result was about step time
The numerical study used neural networks trained on data generated from the Friedman function. The inputs had seven dimensions, the validation set contained 1,000 noise-free samples, and noise added to the training outputs had a standard deviation equal to 5% of the target standard deviation. Three network configurations were each evaluated in 50 trials.
In the primary comparison, HSLM averaged 52.7, 36.9 and 35.4 iterations for Networks 1, 2 and 3, respectively. Its average time per iteration was 0.131, 0.615 and 3.665 seconds. HSLM had the lowest reported time per iteration in all three comparisons, although its iteration count was not lowest for every network.
Relative to classical Levenberg-Marquardt, the reported speedups were approximately 3.7-fold, 3.8-fold and 4.3-fold across the three networks. Relative to Krylov-subspace Levenberg-Marquardt, they were approximately 2.0-fold, 2.8-fold and 3.6-fold. The reported factors increased with network size. The authors also reported similar final training errors for all three methods, approaching the variance of the added noise, and described their convergence behavior as comparable.
The authors reported that HSLM generally met the prescribed projected-gradient adequacy criterion with a substantially smaller subspace than Krylov-subspace Levenberg-Marquardt. In a comparison of basis sources, randomized-curvature-only subspaces struggled to reach the added-noise level. The randomized-curvature-plus-gradient variant showed slower convergence in the reported comparison, while the full hybrid construction reached lower training and validation errors more rapidly and consistently in representative runs.
A narrow test of scale
A separate scaling experiment kept the training set fixed. It reported HSLM speedups over classical Levenberg-Marquardt of approximately 2.3-fold, 2.6-fold and 4.3-fold for the three networks, with additional speedups over Krylov-subspace Levenberg-Marquardt of approximately 2.0-fold, 3.1-fold and 3.6-fold. Because the training set was fixed, those results do not show how the method scales when the number of training samples rises.
The adequacy monitor measures captured projected-gradient information, but that measure does not directly establish that every relevant curvature direction is represented. The local superlinear result also depends on vanishing or very small residuals and additional approximation conditions. The timing comparisons report averages and approximate factors without dispersion measures or uncertainty intervals.
Taken together, the preprint reports a computational advantage in this synthetic neural-network setting, alongside comparable final errors, not a universal win across nonlinear least-squares problems. The fixed-training-set experiment leaves performance with increasing training-set size unresolved.
Paper data and sources
Original title: Adaptive Hybrid Subspace Levenberg Marquardt Algorithm with Adequacy Monitor for Large Scale Least Squares Problems
Authors: M. Duc Hoang, Timothy J. Lewis
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text