A preprint reports that power-outage prediction models can look more capable when they are tested with random samples drawn from the same broad data pool. Random-split tests produced an R2 score of approximately 45% for the relative outage target and about 33% for the absolute outage target. But when researchers held out entire states or individual outage events, improvements over a null model - a forecast based on the test set's average - were often small or absent.
The split changed the score
The paper asks whether three routine choices - what features a model sees, how an outage is defined, and how the data are split - change how well it generalizes across places, time and events. Here, generalization means performing on data held back by location or event rather than on cases mixed into the training pool.
The analysis uses publicly available county-day power-outage records from the U.S. East Coast covering 2018 to 2023, combined with ERA5 weather-reanalysis variables, tree-density features and Prithvi WxC embeddings. The XGBoost models varied both their inputs - ERA5, ERA5 plus tree density, or Prithvi WxC - and their targets, which were either absolute outage counts or relative outage percentages.
The cleaning procedure required dates to have at least 40 counties with outlier outages, producing 28 dates. Among the remaining county-date points, 27% represented at least 5% of the county population without power.
The researchers compared three testing approaches. Random testing used an 80-20 train-test split. Spatial testing held out each of 13 states in turn, while temporal testing held out each event date. Each random-split configuration was run 20 times with different test samples. Tuning used a random validation sample for early stopping, and model size was selected with five-fold cross-validation using a 20% validation holdout in each fold. Random-split results were reported with 95% confidence intervals for R2 and mean absolute error.
New states exposed the gap
On the random split, the relative target reached approximately 45% R2, while the absolute target reached about 33%. R2 is a fit score that indicates how closely predictions track the observed data. Those figures describe performance within a mixed data pool, not necessarily performance when a state or event is entirely new.
The spatial holdouts changed the picture. Mean absolute error, or MAE - the average size of the prediction mistake - was 0.061 for the relative target and 2,317 for the absolute target, compared with around 0.06 and 1,800 in the best-case model. In many held-out states, trained models had higher error than the null model. That pattern was consistent for Delaware and Massachusetts, while results for Maryland, New Jersey and Rhode Island were more sensitive to training conditions.
The choice of features did not resolve the spatial problem. Prithvi models were slightly more consistent in the patterns that generalized across states, but overall performance remained poor, and feature-set choice was not identified as the limiting factor. The comparison covered ERA5, ERA5 with tree density and Prithvi WxC embeddings.
Events were harder to transfer
Event-level tests were similarly weak for absolute outage counts. No model meaningfully outperformed the null for that target. For relative outage, only Hurricane events consistently improved over the null; Winter Storms and Other events did not show comparable predictability. Even within the Hurricane subset, Prithvi performed worse than ERA5, and the best-performing feature set varied across events.
A warning for model evaluation
The paper frames the contrast as a warning about data leakage from the training set into the test set and reports poor generalizability for random train-test splits. Under realistic holdouts, improvements over the null were often small or not statistically meaningful.
The conclusion applies to the tested setting: one public U.S. East Coast dataset and XGBoost configurations varying by feature set, target definition and split strategy. The practical implication is that a random split can give a more favorable estimate than a test that asks a model to transfer to an unseen state or event.
The supplied document is an arXiv version 1 preprint dated 25 August 2026. The work was supported by the DOE American Made Digitizing Utilities Prize.
Paper data and sources
Original title: Data Leakage Inflates Generalizability of Power Outage Prediction Models
Authors: Yamil Essus, Ranga Raju Vatsavai, Benjamin Rachunok
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-25
DOI: Not available
Original paper · Full text