A preprint reports lower mean squared error (MSE), the study’s measure of prediction error, for a two-stage neural-network strategy than for pooled neural networks in selected simulations and in two dataset-specific tests: Beijing PM2.5 prediction and UTKFace age prediction. The gain was reported under particular assumptions and data settings, not as a general guarantee.
The strategy is a form of transfer learning: it uses data from all groups to estimate an overall mean, then uses each group’s data to estimate an offset—the part that departs from the shared pattern. Adding the two pieces gives a group-level estimate.
A shared pattern, then the differences
The paper’s theory breaks group-level squared error into the first-stage error plus the approximation and estimation errors involved in learning each offset. For dense ReLU networks under assumptions about layered function structure, it derives explicit upper bounds and says the rates can overcome the usual slowdown associated with high-dimensional inputs.
Those statements depend on the paper’s assumptions about the model, overlap, complexity, noise and network architecture. The authors explicitly say the upper bounds are not claimed to be optimal. They identify the most favorable transfer settings as those in which shared or offset functions are simpler and pooled data improve the first-stage estimate; complex group differences or pooled aggregation can produce negative transfer.
The numerical tests
The synthetic experiments used sample sizes of 5,000, 10,000, 30,000 and 50,000, Gaussian noise, a 15% within-group test holdout and 50 Monte Carlo replications.
In low-dimensional Scenario 1, at a signal-to-noise ratio of 5 and a sample size of 50,000, the two-stage neural network had a reported average MSE of 0.27 (0.1), compared with 1.93 (0.3) for the pooled neural network. The cited table caption does not define the parenthetical values.
In the high-dimensional scenarios, with noise variance 1 and 50,000 samples, the two-stage method’s reported MSE was 1.77 versus 2.12 for the pooled method in Scenario 3, and 3.19 versus 3.56 in Scenario 4. The comparisons were based on 50 independent trials.
Two tests on real data
The Beijing evaluation used 43,824 hourly records after missing values were removed. The wind-direction groups contained 4,997 northeast records, 14,150 northwest, 9,387 CV and 15,290 south. The test procedure held out 2014 and was repeated in 20 independent trials.
Across those trials, the two-stage neural network had an overall average test MSE of 0.0519 (0.0007), compared with 0.0555 (0.0023) for the pooled neural network—the lowest reported overall MSE among the listed neural-network strategies. The cited excerpt does not state what the parenthetical figures represent.
The UTKFace evaluation used 23,705 facial images, with ages from 0 to 116. The reported ethnicity groups were White at 43%, Black at 19%, Asian at 14%, Indian at 17% and Other at 7%.
In the main comparison, averaged over 20 independent trials, the two-stage method recorded an MSE of 59.6 (1.9), versus 61.8 (1.7) for pooled training. It was reported as the best method for the White, Black and Asian subgroups and better than pooled training for Indian; the main table excluded samples labeled Other.
A narrower takeaway
These are comparative prediction results, not evidence that transfer learning itself caused the lower errors. The paper tests the method in synthetic settings and two real-data applications, so the reported pattern does not establish performance on unseen datasets.
The framework is also presented for trend filtering and orthogonal-series regression beyond ReLU networks, but those convergence claims remain tied to the assumptions of the particular estimator.
Paper data and sources
Original title: Transfer Learning in Nonparametric Regression with Deep ReLU Networks
Authors: Junpeng Ren, Carlos Misael Madrid Padilla, Yanzhen Chen, Oscar Hernan Madrid Padilla
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text