AI Insight
This study evaluated 15 frozen foundation models for blood cell classification and found that while these models achieve near-perfect accuracy on data from their original sources (98-99.7%), their performance drops dramatically (34-72% decrease) when applied to images from different scanners, sites, or preparation methods. The research revealed that top-performing models on original datasets often rank poorly on shifted datasets, model confidence calibration breaks down in new settings, and current adaptation methods fail under realistic conditions where different blood cell types occur at different frequencies than in training data.
Why it matters
Before hematology AI models can be safely deployed in clinical settings across different hospitals and laboratories, they must reliably handle variations in imaging equipment and sample preparation protocols. This work demonstrates that current foundation models are not ready for broad clinical deployment and identifies critical gaps that benchmarks must address, including testing models on truly independent datasets and evaluating performance under realistic shifts in disease prevalence.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Frozen hematology foundation-model (FM) embeddings reach near-saturated in-domain white-blood-cell (WBC) accuracy, but clinical deployment demands reliability across scanners, sites, stains and preparation pipelines. We audit 15 frozen encoders (hematology, pathology, and general vision) across four public single-cell acquisition domains along two axes: accuracy robustness and calibration. In-domain linear-probe macro-F1 is saturated (0.98-0.997), yet cross-dataset macro-F1 drops 34-72% and rankings re-order: DinoBloom-L, the in-domain best, falls to 10th of 15 on the most-shifted target (MLL23) at the benchmark’s shared 224-px input, behind RedDino and several general and pathology encoders. Rank transfer is probe-dependent: 1-NN retrieval is more stable on average than a source-fitted linear head (median $rho$ 0.65 vs 0.45), but neither probe universally predicts target robustness. Calibration also collapses: source-trained probes are nearly calibrated in-domain (expected calibration error, ECE, 0.004) but confidently wrong off-domain (ECE 0.35), and source-fitted temperature scaling transfers poorly. We further audit pretraining exposure and identify MLL23 as DinoBloom’s internal cohort; because DinoBloom’s only held-out dataset is also our source domain, this benchmark cannot isolate exposure from scanner-associated shift. Label-free adaptation and marginal-entropy-based model selection appear safe under balanced evaluation but fail under realistic WBC class-prior shift. Class-Balanced Re-standardization (CBR), a training-free pseudo-label-balanced feature normalization, improves all evaluated target-prior scenario means and partially improves calibration, although encoder-level exceptions and residual miscalibration remain. Hematology FM benchmarks must therefore jointly audit accuracy, calibration, exposure, and class-prior robustness.
Source: Can You Trust Frozen Hematology Foundation Models under Acquisition Shift?