AI Insight
Automatic speech recognition systems perform poorly on speech from people with dysarthria, and current audio-language models fail to improve accuracy even when provided with clinical context such as diagnosis labels or detailed speech ratings. Researchers tested nine models using the Speech Accessibility Project dataset and found that adding clinical information through prompts produced negligible improvements or sometimes worsened performance, though fine-tuning with LoRA adaptation achieved a 52% reduction in word error rate while maintaining performance without context.
Why it matters
This research highlights a critical accessibility gap in AI speech recognition technology that affects millions of people with speech disabilities. The findings demonstrate that despite advances in AI, current models cannot effectively use clinical context to understand atypical speech, though the successful fine-tuning approach offers a pathway toward more inclusive speech recognition systems.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Automatic speech recognition (ASR) systems remain brittle on dysarthric and other atypical speech. Recent audio-language models raise the possibility of improving performance by conditioning on additional clinical context at inference time, but it is unclear whether these models can make use of such information. We introduce a benchmark built on the Speech Accessibility Project (SAP) dataset that tests whether diagnosis labels, clinician-derived speech ratings, and progressively richer clinical descriptions improve transcription accuracy for dysarthric speech. Across matched comparisons on nine models, we find that current models do not meaningfully use this context: diagnosis-informed and clinically detailed prompts yield negligible improvements and often degrade word error rate. We complement the prompting analysis with context-dependent fine-tuning, showing that LoRA adaptation with a mixture of clinical prompt formats achieves a WER of 0.066, a 52% relative reduction over the frozen baseline, while preserving performance when context is unavailable. Subgroup analyses reveal significant gains for Down syndrome and mild-severity speakers. These results clarify where current models fall short and provide a testbed for measuring progress toward more inclusive ASR.
Source: When Audio-Language Models Fail to Leverage Multimodal Context for Dysarthric Speech Recognition