AI Insight
RadMultiBench introduces a standardized benchmark of 6,148 medical images paired with radiological reports across five disease categories to evaluate vision-language models on report generation and disease classification tasks. Testing three different architectures revealed that a classical CNN-RNN model achieved near-perfect classification accuracy (99.5%) and was the only model capable of generating coherent reports, while modern pre-trained vision-language models failed catastrophically at both tasks, with the domain-specific BioViL-T achieving only 11.5% classification accuracy. The benchmark provides a reproducible pipeline for continuous testing and exposes critical limitations in applying general-purpose vision-language models to clinical radiology.
Why it matters
This research demonstrates that state-of-the-art AI models successful in general domains may fail dangerously when applied to high-stakes medical decision-making, potentially preventing deployment of unreliable systems in clinical settings. The reproducible benchmark and pipeline provide essential infrastructure for developing and validating multimodal AI systems before they are used to assist radiologists in diagnosing conditions like cancer, pneumonia, and tuberculosis.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
We present RadMultiBench, a multimodal benchmark of 6,148 medical images paired with expert radiological impressions across five categories–Breast Cancer, Lung Benign, Lung Malignant, Pneumonia, and Tuberculosis–targeting two core clinical tasks: automated report generation and disease classification. To enable reliable, repeatable experimentation, we provide a reproducible pipeline for data ingestion, de-identification, split management, and standardized reporting of text and classification metrics, facilitating dataset versioning and continuous benchmark testing. We compare three architectures: (1) a classical CNN-RNN with Bahdanau attention (ResNet-LSTM), (2) a general-domain vision-language (V-L) prefix-projection model (CLIP-GPT), and (3) a domain-specific V-L prefix-projection model (BioViL-T). Results reveal a sharp performance divide: the ResNet-LSTM attains near-perfect classification (99.5% accuracy) and is the only model to generate coherent reports (BLEU-4 = 0.021), whereas both prefix-projection models catastrophically fail generation (BLEU-4 [≤] 0.0001); moreover, the domain-specific BioViL-T also fails classification (11.5% accuracy). From a data-engineering angle, the pipeline is productionizable with lineage, feature registries, and continuous integration to guard against silent regressions when updating models or data; from a data-science angle, the benchmark provides baselines and analyses that foreground feature alignment and architectural suitability as first-order concerns, inviting ablations on multimodal objectives, retrieval-augmented encoders, and uncertainty-aware decoding. Overall, RadMultiBench highlights that state-of-the-art status in general V-L domains does not translate automatically to high-stakes clinical tasks, and it furnishes the tooling and evidence base needed to develop robust, clinically viable multimodal systems.