AI Insight
This study challenges the assumption that training AI models with synthetic rationales improves clinical predictions. Testing on Alzheimer's disease prediction across 504 experimental configurations, researchers found that "Rationale First" fine-tuning—where models generate explanations before predictions—consistently degraded performance compared to standard label-only training. The problem was not due to poor rationale quality, as expert review confirmed the explanations were medically accurate, but rather stemmed from conditioning predictions on the generated rationales during training.
Why it matters
These findings have significant implications for developing AI systems in high-stakes clinical settings, suggesting that common approaches to making models more interpretable may inadvertently harm their predictive accuracy. The research provides crucial guidance for responsible deployment of language models in healthcare, highlighting that rationales may be more useful at inference time than as training targets.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Supervised fine-tuning with synthetic rationales is widely assumed to improve clinical prediction by teaching models not only what to predict but why. We test this assumption on five-year Alzheimer’s disease and related dementias (ADRD) prediction from longitudinal health histories. In this setting, we generate training rationales using the correct labels, then fine-tune models to output a rationale followed by a label. Across a large-scale controlled experiment of 504 configurations, we find that Rationale First SFT consistently and substantially hurts predictive performance relative to Label Only fine-tuning. The degradation persists across model families and data scales, and is not resolved by using a reasoning-oriented base model. A fixed-protocol cardiovascular disease (CVD) validation reproduces similar findings. Crucially, the failure is not explained by poor rationale quality: human expert annotation confirms that the generated rationales are medically accurate and faithfully grounded in patient-specific evidence, and few-shot experiments show that the same rationales improve performance when used as inference-time demonstrations rather than training targets. A simple label-ordering intervention clarifies the mechanism: placing the prediction before the same rationale restores predictive performance. This suggests that conditioning predictions on generated rationales contributes to the degradation. We hope our work paves the way toward a more precise understanding of when and how rationale-based supervision helps and when it does not, guiding the responsible development of language models for high-stakes clinical prediction.
Source: Rethinking Synthetic Rationale Supervision for Disease Prediction