A benchmark of frozen AI image encoders has found a wide gap between performance on familiar white blood cell images and performance on images from other datasets. Models that reached macro-F1 scores of 0.98 to 0.997 in-domain lost 34% to 72% of that score in cross-dataset tests. Macro-F1 is a single score used here to summarize performance across five white blood cell classes.
The result is a warning about relying on a model’s best local score or its confidence as a proxy for transfer. In the benchmark, source-trained probes were almost perfectly calibrated on their home data but much less trustworthy off-domain, where their stated certainty often did not match their accuracy.
A strong home score was not enough
The study used public single-cell WBC image datasets. Acevedo/PBC supplied 10,298 source images, while MLL23/Metafer, Matek-LMU/M8 and Raabin were used as zero-shot targets. The panel contained 15 frozen encoders and a supervised ResNet-18 baseline. The encoders’ features were kept fixed; source-domain logistic regression and one-nearest-neighbor probes were trained, then tested across the full source-to-target matrix at a common 224-by-224 input size and a fixed five-class intersection.
Because patient identifiers were not consistently available, images were split at image level. Probes used a stratified 70% source training split and a held-out 30% source test set, but that in-domain test served only as a clean-ranking proxy. The deployment question in the benchmark was cross-dataset transfer.
The leaderboard changed
That transfer test reshuffled the leaderboard. DinoBloom-L, the best encoder in-domain, fell to 10th among 15 models on MLL23 and trailed RedDino by 0.15 macro-F1. The paired-bootstrap 95% confidence intervals reported for the key MLL23 comparisons excluded zero.
Rankings also depended on how the features were read. Across 12 source-target pairs, one-nearest-neighbor probing had a median Spearman correlation of 0.65 with target rankings, compared with 0.45 for the linear probe. But its Acevedo-to-Raabin correlation was only 0.34, showing that the more stable average was not a universal guide.
Confidence failed to travel
Calibration means the match between a model’s confidence and how often it is correct. On source data, source-trained probes had an expected calibration error of 0.004 and a negative log-likelihood of 0.03. Across 15 encoders and three targets, the off-domain averages rose to 0.35 and 3.2. The pattern indicates that the probes could be confidently wrong after the dataset changed.
A standard source-fitted temperature adjustment did little to repair that problem: target ECE moved from 0.35 to 0.32. An oracle adjustment fitted with target labels reached 0.07, while CBR alone reached 0.29 and CBR combined with source scaling reached 0.25. The oracle result is not a label-free deployment option, and even the better figures left residual miscalibration.
Some fixes made the picture worse
The same caution applied when the target data had skewed proportions of cell types, known as class-prior shift. Across 18 evaluated target-prior scenarios, global target standardization reduced macro-F1 by 0.07 in a realistic clinical setting and by 0.08 in a neutrophil-heavy setting, and it harmed six scenarios overall. SHOT/IM had a mean change of minus 0.029 and harmed 10 of 18 scenarios. BBSE had a mean change of minus 0.035 and harmed all 18.
CBR was the exception in this set of scenarios. It produced positive macro-F1 gains in all 18, averaging +0.059 and ranging from +0.007 to +0.109; no scenario mean was harmed, and a hierarchical bootstrap put the 95% confidence interval at +0.046 to +0.073. The zero-harm count applies to scenario means, not every encoder-by-scenario result.
That improvement was not equally steady at every batch size. In tiny imbalanced batches, CBR’s per-draw harm reached about 46% when a batch contained 16 target images. Average performance remained positive, but the authors recommend batches of at least 32 images.
A separate label-free selection test also weakened the case for choosing a model from target predictions alone. Marginal prediction entropy, a measure of how uncertain those predictions look, had selection regret of 0.25 to 0.37 under every evaluated skewed prior. Regret stayed elevated as the target batch grew from eight to 400 images, while clean accuracy performed better. Agreement-style selectors were not evaluated under skewed priors.
The boundary of the evidence
The analysis also points to a class-specific failure behind the headline averages. The authors identify changes in per-feature means as the dominant measurable shift. Off-domain lymphocyte recall, the share correctly identified, was just 0.16 to 0.20 for DinoBloom-B/L, compared with 0.82 for DINOv2-B, based on five-seed means.
One important limitation is that pretraining exposure was not cleanly separated from dataset shift. The audit reports MLL23 as DinoBloom’s internal cohort and says Acevedo was its only held-out dataset, while Matek, Raabin and MLL23 were in pretraining. As a result, the benchmark had no leakage-free DinoBloom target, so it does not establish that DinoBloom-specific leakage caused the poor MLL23 transfer.
The findings are therefore about model transfer on public WBC images, not clinical performance. The test was limited to a fixed five-class intersection and the listed encoders, probes and target-prior scenarios. Patient identifiers were not consistently available, and the source split was made at image level. Those boundaries mean the results do not establish performance beyond the evaluated datasets and setup.
The authors recommend representative cross-domain validation, disclosure of pretraining overlap, recalibration for each scanner, and class-balanced adaptation instead of relying on global target statistics. They describe CBR as only a partial mitigation. The work is an arXiv preprint accepted as an oral presentation at HemaRAI 2026, with a stated plan to appear in MICCAI 2026 Satellite Events, Lecture Notes in Computer Science, Springer. The authors declare no relevant competing interests.
Paper data and sources
Original title: Can You Trust Frozen Hematology Foundation Models under Acquisition Shift?
Authors: Jai Kumar Sharma, Peeyush Tapadiya
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-25
DOI: Not available
Original paper · Full text