AI Insight
This study introduces RAU, a framework that combines vision-language models (VLMs) with SAM2 to identify, localize, and segment anatomical structures in medical images using annotated reference images as guides. The system demonstrates that VLMs can learn spatial reasoning between reference and target images through training on moderately sized datasets, enabling accurate anatomical identification and pixel-level segmentation. RAU consistently outperforms baseline methods across multiple datasets and shows strong generalization to unseen medical imaging modalities.
Why it matters
This approach addresses a critical bottleneck in medical image analysis—the scarcity of expert-labeled data—by leveraging reference images to guide interpretation of unlabeled scans. The framework's ability to generalize across different imaging modalities without retraining could significantly accelerate the development of automated clinical workflows and reduce the burden of manual annotation in medical imaging applications.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
-cross
Abstract: Anatomical understanding, which is the ability to identify, localize, or segment anatomical structures, is critical in medical image analysis; however, its progress is constrained by the scarcity of expert-labeled data. A promising remedy is to leverage an annotated reference image to guide the interpretation of an unlabeled target. Although recent vision-language models (VLMs) exhibit non-trivial visual reasoning, their reference-based understanding and fine-grained localization remain limited. We introduce RAU, a framework for reference-based anatomical understanding with VLMs. We first show that a VLM learns to identify anatomical regions through relative spatial reasoning between reference and target images, trained on a moderately sized dataset. We validate this capability through visual question answering (VQA) and bounding box prediction. Next, we demonstrate that the VLM-derived spatial cues can be seamlessly integrated with the fine-grained segmentation capability of SAM2, enabling localization and pixel-level segmentation of small anatomical regions, such as vessel segments. Across two in-distribution and two out-of-distribution datasets, RAU consistently outperforms a SAM2 fine-tuning baseline using the same memory setup, yielding more accurate segmentations and more reliable localization. More importantly, its generalization ability to unseen modalities makes it scalable to unseen datasets, a property crucial for medical image applications. To the best of our knowledge, RAU is the first to explore the capability of VLMs for reference-based identification, localization, and segmentation of anatomical structures in medical images. Its promising performance highlights the potential of VLM-driven approaches for anatomical understanding in automated clinical workflows.
Source: RAU: Reference-based Anatomical Understanding with Vision Language Models