The paper's main theorem says a randomly pruned deep neural network can reach the minimax rate — the study's benchmark for estimating a smooth regression function — up to an extra logarithmic factor. The result is an expected integrated squared-error bound and is conditional on the paper's stated assumptions.
The target is m(x)=E{Y|X=x}, the conditional average outcome at a given predictor value. The study measures fit with integrated L2 error, which averages the squared gap between the estimated and target functions over the predictor distribution. In the paper's notation, the main rate is n^(-2p/(2p+d)): n is the number of observations, p describes smoothness and d is the full predictor dimension.
Under a separate theorem, if the support of the predictors lies in an approximate d*-dimensional manifold, the dimension in the main power of n becomes d* rather than d. In plain terms, the theoretical rate is governed by the lower-dimensional structure under that condition, not simply by the full predictor space.
The network is made sparse before training
The proposed estimator starts as a fully connected network with logistic activation. Before training, it is randomly pruned so each hidden neuron retains r incoming connections. Gradient descent then uses data-dependent, non-constant step sizes chosen by an Armijo-type check that requires empirical risk to decrease.
The theoretical argument assumes bounded predictors, an exponential-moment condition on Y, and a (p,C)-smooth regression function. The returned network output is truncated at beta_n = c2 log n.
The numerical evidence comes from a synthetic example
The numerical illustration is a univariate regression simulation. Outcomes were generated by combining a regression function with a noise term; predictor values followed a standard-normal distribution restricted to -1 through 1. The simulations used noise levels of 0.5 and 1 and sample sizes of 100, 200 and 400 observations.
For tuning, the neural-network runs used 80% of each sample for learning and 20% for testing, while the smoothing-spline comparator used generalized cross-validation. The proposed estimator was implemented in R.
In 50 simulations with 100 observations and noise level 1, the lowest listed median L2 error came at c1 = 0.5, at 0.0065. The medians were 0.0095 at c1 = 0.1 and 0.0086 at c1 = 1.0.
In the broader comparison, NNprun's median L2 error was 0.0040, 0.0025 and 0.0018 at sample sizes 100, 200 and 400 when noise was 0.5. At noise level 1, the corresponding medians were 0.0086, 0.0056 and 0.0035. The smoothing-spline medians were slightly lower in every listed condition, from 0.0035 to 0.0016 at noise 0.5 and from 0.0076 to 0.0033 at noise 1.
The evidence remains conditional
The theoretical guarantees depend on boundedness, moment, smoothness and network conditions, while the numerical evidence is limited to a synthetic univariate example. The supplied results therefore leave open whether the reported behavior would persist on real-world or higher-dimensional data.
The simulation summaries report medians and interquartile ranges, not confidence intervals, and the comparison used a learning/testing split rather than an independently reported external test set. These results do not establish universal superiority for NNprun beyond this synthetic exercise.
The constants in the theorem are not numerically specified, and the simulations do not test alternative activations, pruning schemes or step-size rules. The practical distance between the displayed finite-sample errors and the theoretical rates therefore remains uncertain.
Paper data and sources
Original title: Learning of deep neural network regression estimates using gradient descent with pruning
Authors: Michael Kohler, Vincent Molinero Römer, Adam Krzyżak
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text