The strongest result from the benchmark is that a score requiring no target labels closely tracked which of six foundation models ranked best on a shifted chest X-ray target. In the MLP-probe setting, AURCC's Spearman rho was 0.943 in every scenario and at each tested probability-bin count, with p < 0.05.
The question is practical: can candidate models be ordered by expected performance in a target domain when source data are labeled but the target domain is unlabeled? The proposed approach addresses it with a single label-free score that aggregates regional pseudo-label discrepancies from SUDO, using the models' temporary label assumptions to compare them.
The test was deliberately specific
The benchmark used filtered chest X-ray datasets for binary pneumonia classification. It included 37,457 samples from MIMIC-CXR, 4,770 from NIH ChestX-ray14, 563 from CheXpert and 88,018 from PadChest. The three source datasets were split 80/20, with PadChest serving as the target.
Positive rates varied across those datasets: 41.6% in MIMIC-CXR, 30.0% in NIH ChestX-ray14, 67.5% in CheXpert and 3.2% in PadChest.
Researchers compared six foundation models in two regimes: zero-shot, or ZS, and MLP-probe, or MP. They also tested three probability-bin counts, K = 5, 10 and 20, to check whether the model ordering changed with the way probabilities were grouped.
AURCC's natural comparator was a ranking based on held-out source AUC. The reference ranking was full-target AUC computed post hoc, allowing the study to ask how closely the label-free ordering matched measured target performance.
For each probability bin, opposing pseudo-label assumptions were evaluated with auxiliary logistic regressions on held-out source data. The SUDO procedure was averaged over 300 stochastic runs before the bin-level discrepancies were aggregated into AURCC.
Where the score held up
Spearman rho is a rank-agreement measure. Against that yardstick, the MP result of 0.943 indicates very close agreement between AURCC and the post-hoc target ranking across the tested scenarios and bin counts.
In the NIH comparison, the held-out-source-AUC baseline had rho 0.829, based on 954 held-out source samples, while AURCC produced a strictly better ranking. In that setting, the reported AURCC ordering was better than the straightforward source-only ordering.
Zero-shot testing was less one-sided. Ranking correlations ranged from 0.714 to 0.943: source AUC was stronger for MIMIC and NIH, while AURCC was stronger for CheXpert. The findings therefore do not show that AURCC always outperforms source AUC.
The small-data test
To examine scarce labeled-source conditions, the authors varied the held-out source size and reported mean Spearman rho over 20 repetitions in the MP regime with K = 10.
For MIMIC, source AUC reached rho 1.000 at full size. At N = 50, however, AURCC's rho was 0.683 versus 0.605 for source AUC.
For NIH and CheXpert, AURCC outperformed source AUC at every tested size. At N = 100 in NIH, the correlations were 0.911 for AURCC and 0.749 for source AUC; at N = 50, AURCC outperformed the baseline in all scenarios.
A narrow claim, for now
The boundary of the evidence is clear. The evaluation covered one binary pneumonia task and one target domain, PadChest; generalization to other tasks, imaging modalities and multiclass settings was not validated.
That limitation matters alongside the benchmark design: the reference outcome was full-target AUC computed post hoc, while the question the method addresses is ranking models when the target domain is unlabeled. The reported result is therefore a statement about rank agreement in this tested setup.
Taken together, the paper supports a strong MP benchmark result and a more mixed zero-shot picture. It leaves open whether the same pattern will hold beyond the tested task and target domain.
Publication note
The document identifies itself as arXiv:2608.25810v1 [cs.CV], dated 26 Aug 2026. The supplied text says the authors have no competing interests relevant to the article and does not report a funding statement.
Paper data and sources
Original title: Label-Free Foundational Model Selection for Medical Image Classification under Distribution Shift via Pseudo Label Discrepancy
Authors: Juan Iñaki Larrea, Lucas Mansilla, Enzo Ferrante
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text