AI Insight
Researchers compared four AI vision models adapted for brain tumor segmentation and found that despite achieving similar performance (78-81% accuracy), the models modified their internal structures in fundamentally different ways. When testing these segmentation-adapted models on Alzheimer's disease classification, one model improved while the other three degraded in performance, revealing that equal task performance does not mean equal internal processing. The study identified that early-to-mid layers of the models are most critical for segmentation, and restricting adaptation to only these layers maintained performance while reducing negative effects on other tasks.
Why it matters
This finding challenges the common practice of selecting AI models based solely on performance metrics, showing that internal mechanisms matter for multi-task medical applications. The research provides a more efficient adaptation strategy that uses fewer parameters and maintains broader applicability across different diagnostic tasks, which could improve AI deployment in clinical settings where models need to handle multiple types of medical image analysis.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Adapting Vision Foundation Models (VFMs) to medical imaging via LoRA has become standard practice, yet model selection is guided almost entirely by downstream task metrics, treating backbones with similar performance as interchangeable. We investigate whether equal performance implies equal internal representational shift. We analyse four ViT-B/16 backbones, DINOv3, SigLIP2, MerMED-FM, and BiomedCLIP, pre-trained with different objectives (self-distillation vs. contrastive vision-language) and on different data domains (natural vs. medical images). We adapt each to brain tumour segmentation and evaluate on six datasets covering both segmentation and Alzheimer’s disease classification. Using attention divergence, weight-change analysis, and activation patching, we find that the four models converge to similar segmentation performance (mean Dice 78-81%) through markedly different internal dynamics, yet diverge in cross-task transfer: segmentation adaptation improves Alzheimer’s disease classification for DINOv3 but degrades it for the other three backbones. We reveal a dissociation between modification magnitude and causal importance: the most heavily modified layers are not the causally relevant ones, with early-to-mid layers identified as sufficient for segmentation recovery. Restricting LoRA to these layers matches or approaches full-layer segmentation while reducing classification degradation in the three affected backbones, providing a mechanistically grounded strategy that is more parameter-efficient and more robust than standard full-layer adaptation.
Source: Convergent Performance, Divergent Mechanisms in LoRA-Adapted Vision Foundation Models for Brain MRI