An arXiv preprint testing chest X-ray AI reports a split result. In a matched NIH ChestXray14-to-CheXpert transfer test, ResNet-18 encoders initialized with BYOL scored higher than those initialized with ImageNet. AUC, the study's classification score, was 0.87 versus 0.82 with a frozen-backbone linear probe, and 0.88 versus 0.85 when the final residual block and classifier were partially fine-tuned.
That edge did not carry over to the paper's multimodal retrieval stress test. On held-out OpenI, strict pair-index retrieval, a check for the exact indexed image-text match in the top-ranked results, produced very low scores. ImageNet + Text recorded Recall@1 of 0.0006, Recall@5 of 0.0010 and Recall@10 of 0.0020. BYOL + Text recorded 0.0002, 0.0010 and 0.0022 at the same cutoffs. The processed indices contained 6,800 OpenI candidate entries and 15,000 PadChest entries, counts treated as candidate-pool sizes rather than unique report totals. Because the test uses exact indexed pairs, the figures do not establish broader semantic retrieval.
The tests measured different kinds of performance
The study was designed to make those differences visible across several settings. It used four public chest X-ray datasets with fixed roles: NIH ChestXray14 was the labeled source, CheXpert the external target, PadChest the paired training and in-domain evaluation corpus, and OpenI the external paired stress-test set.
In the source-only transfer condition, trainable components used NIH labels only. CheXpert labels were excluded from parameter updates, hyperparameter selection and early stopping. The evaluation compared a frozen-backbone linear probe with partial fine-tuning of the final residual block and classifier.
For the multimodal experiment, the image encoder was paired with BioClinicalBERT. The two outputs were projected into a normalized shared space, then trained with a symmetric contrastive loss on paired PadChest data. Text inputs were standardized labels or translated free-form text.
Source clues survived the stress tests
A separate diagnostic asked whether the OpenI representations retained clues about their source. Linear probes tried to recover two metadata-derived targets, site_parent and site_folder, from frozen embeddings. The proxy classes could appear in both probe partitions, while patient or study groups were kept disjoint. The proxies remained recoverable, but that result supports source-proxy recoverability, not verified hospital-site identification.
Efforts to reduce the signal exposed a trade-off with downstream utility. InstanceNorm cut proxy accuracy most aggressively and carried the largest utility cost. Adversarial unlearning was intermediate, while CORAL produced a milder change.
The domain-alignment runs were not uniformly stable. These were separate alignment tests using unlabeled target-domain features, rather than the source-only transfer setup. DANN target AUC rose early and then degraded as adversarial pressure increased, while CORAL was less erratic in the observed runs.
The caveats matter
Qualitative checks offered a more nuanced picture. Nearest-neighbor and Grad-CAM analyses showed clinically plausible cross-dataset structure and thoracic attention patterns, but device-heavy and false-positive cases remained ambiguous. These diagnostics do not establish lesion-localization accuracy or causal grounding.
An auxiliary consolidation-classifier check suggested that architecture sensitivity depends on the task. AUCs were close for ResNet-18 at 0.9257, ResNet-50 at 0.9163, DenseNet-121 at 0.9200, EfficientNet-B0 at 0.9203 and ResNet-50 CLIP at 0.9155. ViT-S scored 0.8774, while exploratory Swin-T scored 0.5373 and was unstable. The check was auxiliary and heterogeneous, so it is not directly comparable with the primary NIH-to-CheXpert transfer.
The findings are bounded by the study's scope. It does not estimate epistemic uncertainty or offer a formal account of what the model knows. It is limited to chest radiography, uses unverified OpenI site labels, and does not evaluate calibrated epistemic uncertainty, abstention or formal unknown-unknown detection. The central transfer values are point estimates without seed-level records needed to reconstruct mean and standard deviation, so no statistical-significance claim was made. The manuscript is an arXiv preprint, version 1, dated 26 August 2026.
Paper data and sources
Original title: What Do Medical Vision-Language Models Learn in Radiology? Transfer, Alignment, and Source-Proxy Leakage Under Distribution Shift
Authors: Ayoub Louaye Bouaziz, Lokmane Chebouba, Yassine Himeur
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text