A machine-learning analysis of the German Physics Olympiad found only a modest ability to predict which individual students would advance beyond the first round. The researchers trained a classifier on students who took part in the competition, then applied it to students who did not, asking what their chances might have been under the alternative condition. First-round success meant advancing to the second round or beyond; non-advancement was the comparison outcome.
The data came from 282 Physics Olympiad participants and 1,103 non-participants in academic-track secondary schools. Non-participants served as a comparison group and were matched with participants by school type and grade level. The school restriction was used because 96.5% of participants with school-type information attended academic-track schools.
Among the participants, 132, or 46.8%, advanced beyond the first round, while 150, or 53.2%, did not. In the overall population, 406 of 931 participants advanced, a rate of 43.6%. That difference means the participant group in the analysis appeared slightly more positively selected for first-round success than the wider competition population.
What the models found
Researchers compared three machine-learning approaches: elastic-net logistic regression, random forest and gradient boosting. They used 10 repeated rounds of stratified five-fold nested cross-validation, producing 50 outer evaluation folds. The evaluation therefore used repeated folds rather than a single split of the data.
The paper used three summary measures of predictive performance. On average across the 50 outer folds, the elastic net had a Brier score of 0.237, an AUC of 0.605 and Cohen’s kappa of 0.177. The corresponding figures were 0.239, 0.605 and 0.172 for the random forest, and 0.244, 0.610 and 0.171 for gradient boosting. The scores were close, and all three models showed modest individual prediction. Elastic net was chosen for the final analysis because performance was near-equivalent and it was the more parsimonious choice.
The final elastic-net model kept five of 31 input features and reduced the other 26 coefficients to zero. Mathematics skills, physics- and engineering-related skills, and skipping a grade pointed toward higher predicted success, while conventional vocational interests pointed in the opposite direction. These directions describe how the model used the variables; they do not show that any of them caused students to advance.
A useful but narrow signal
The model’s probability estimates did not show significant overall miscalibration, although its calibration curve departed from perfect calibration in a nonlinear way. The reported calibration intercept was 0.03, with a standard error of 0.13; the slope was 1.22, with a standard error of 0.33. A separate global check gave a Spiegelhalter z-statistic of -0.63 and p = .53. The authors interpreted the elastic net as sufficiently calibrated for group-level inference, while its individual predictive power remained modest.
The counterfactual step is therefore narrower than a prediction for every student. It estimates first-round success probabilities for non-participants by transferring relationships learned from participants; it does not reveal whether any particular non-participant would actually have advanced. The reported performance measures were means across the 50 outer folds, and confidence intervals were not reported. That leaves uncertainty around how precisely the model can forecast individual outcomes.
The analysis also examined prior competition experience among the non-participants identified by the model, keeping the question broader than a single contest. But the evidence remains tied to first-round advancement in this competition and to the students and school settings in the dataset. It cannot establish that taking part caused success, nor does it show that any particular non-participant would have advanced. The clearest message is more limited: the model detected some predictive signal, but not a strong individual forecast.
Paper data and sources
Original title: "What If I Had Participated?" A Machine Learning-Based Counterfactual Analysis of Non-Participants' Success Potential in STEM Competitions: The Case of the German Physics Olympiad
Authors: Paul Tschisgale, Knut Neumann
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-09-28
DOI: Not available
Original paper · Full text