A machine-learning model for radioiodine-refractory differentiated thyroid cancer, or RAIR-DTC, ranked patients by their later outcome almost as well at an external center as in its internal test. But it overstated absolute risk: the average predicted risk outside the development setting was 24.1%, compared with an observed rate of 14.8%. In practical terms, the model was better at ordering patients from lower to higher risk than at attaching a trustworthy percentage to each person.
The study included 302 patients in a development cohort and 122 in an external-validation cohort. RAIR-DTC occurred in 46 development patients, or 15.2%, and 18 external patients, or 14.8%, making the observed outcome rate similar in the two groups.
What the model knew
Researchers used the first postoperative radioiodine treatment as the index point. The binary question was whether RAIR-DTC was diagnosed during available follow-up, rather than whether it would occur by a preset time. The development cohort was divided in a 7:3 ratio into 212 training patients, with 32 events, and a 90-patient internal test set, with 14 events.
Candidate inputs were limited to information available at that baseline: sex, age at least 55, histology, tumor size, multifocality, extrathyroidal extension, vascular invasion, lymph-node or distant metastasis, BRAF V600E and TERTp mutations, and the initial pre-radioiodine sTg measure. No information generated after radioiodine treatment was allowed into the model, a safeguard against leakage, in which later information can make a prediction look better than it would have been in real time.
Six classifiers were compared using nested five-fold cross-validation, with tuning and out-of-fold performance estimation repeated 100 times. The final choice was a random-forest model because its out-of-fold ROC AUC was 0.944, nearly tied with XGBoost at 0.945, while its Brier score, a measure of probability accuracy, was lowest at 0.059.
Strong at ranking, weaker at probabilities
On the internal test set, the random forest had an AUC of 0.896 and a Brier score of 0.078. At the external center, its AUC was 0.892, with a precision-recall AUC of 0.597, sensitivity of 77.8% and specificity of 79.8%. AUC measures how well a model separates and ranks people with and without the outcome; the close internal and external figures indicate that this ranking performance held up in the second cohort.
The warning came from calibration, which asks whether predicted probabilities match what actually happens. Externally, the model's mean predicted risk was 24.1%, while the observed rate was 14.8%. Its calibration slope was 0.304 and its observed-to-expected ratio was 0.613, figures that indicate poor agreement between the model's percentages and the cohort's actual rate. The authors therefore considered local recalibration necessary before using the probabilities directly.
One possible explanation offered in the analysis was a shift in the molecular makeup of the patients. BRAF V600E mutations were present in 90.2% of the external cohort, compared with 73.3% of the internal test set. TERTp mutations were present in 23.8% externally versus 2.2% internally. The authors described this as a possible contributor to calibration drift, not a tested cause.
The model's explanation tools pointed to age 55 or older, TERTp mutation, vascular invasion, initial pre-radioiodine sTg, extrathyroidal extension, BRAF V600E and lymph-node metastasis as the leading contributors to its predictions. Older age, TERTp mutation, vascular invasion and higher sTg generally pushed predictions upward. Those are descriptions of fitted-model behavior, not evidence that any of the features causes RAIR-DTC.
Why caution matters
As an exploratory exercise, the researchers divided patients into low-, intermediate- and high-risk groups. In the internal test set, the groups contained 61, 13 and 16 patients and had event rates of 3.3%, 0% and 75%, respectively. In the external cohort, groups of 66, 13 and 43 patients had rates of 0%, 15.4% and 37.2%. The pattern separated observed outcomes, but the groups were not a validated clinical rule and the small numbers make the estimates unstable.
The study's evidence base was narrow. Only 46 development and 18 external RAIR-DTC events were available, the training-set AUC approached 0.98, and the authors cautioned that small strata produced unstable subgroup estimates.
Taken together, the study supports cautious interest in using baseline information for early risk stratification, but the external result does not make the model a ready-to-use percentage calculator. Its ranking performance was strong, while its probabilities were miscalibrated. Local recalibration would be needed before probability-based use, and the fitted-model associations should not be read as proven causes of RAIR-DTC.
Paper data and sources
Original title: An explainable and leakage-aware machine learning model for early risk stratification of radioiodine-refractory differentiated thyroid cancer: development and single-center geographic external validation
Authors: Tan G, Zhang Y, Mao Z et al.
Journal/Repository: Frontiers in endocrinology
Status: Peer-reviewed
First online: 2026-10-19
DOI: Not available
Original paper · Full text