AI Insight
Researchers tested whether software can automatically score the NIH Stroke Scale (NIHSS), a critical neurological assessment tool, directly from spoken bedside examination transcripts rather than from written notes. Using 112 synthetic stroke cases, they compared a specialized decision model (Jev) against a general large language model (GPT-6-luna), finding that the LLM achieved 99.0% item-level accuracy compared to 94.4% for the decision model. The decision model was significantly faster and cheaper, and its confidence scores could flag uncertain items for human review or LLM verification, with a hybrid approach achieving 98.8% accuracy.
Why it matters
This technology could support ambient AI clinical documentation tools by automatically generating NIHSS scores during time-critical stroke evaluations, potentially reducing scoring errors and documentation burden. The approach may help standardize stroke assessment and accelerate treatment decisions, though validation with real clinical recordings and human expert raters is needed before clinical implementation.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Background. Clinicians score the NIH Stroke Scale (NIHSS) at the bedside, often under time pressure, and ambient AI tools that listen to clinical encounters are now entering practice. It is not known whether software can score the NIHSS from the exam conversation itself, rather than extract a score that has already been written in a note. Methods. We created 112 synthetic code-stroke cases. One hundred were allocated in proportion to real code-stroke presentations, and 12 stress cases tested special scoring rules. For each case, a large language model (Claude Opus 5.5) wrote a reference score form (the answer key) and a speech-to-text-style exam transcript, with main-set transcripts split evenly between a narrated and a telegraphic examiner style. A second model (Claude Fable 5.1), blind to the key, scored every transcript and agreed with the key on 99.2% of items. Each transcript was then scored by Jev, a decision model, and by gpt-6-luna, a general large language model (LLM). The primary outcome was item-level accuracy against the key. Results. The LLM scored 99.0% of items correctly (95% CI 98.4-99.4) and Jev 94.4% (93.2-95.4; McNemar p < 0.001), with exact totals in 101 and 63 of 112 cases. Jev was faster (median 0.15 vs 5.78 s per exam) and cheaper (USD 0.23 vs 0.67 per 1,000 exams). With telegraphic examiners, accuracy fell from 98.0% to 90.9% for Jev and from 100% to 97.7% for the LLM. In a post-hoc analysis that kept only the examiner’s words, accuracy stayed high when the examiner narrated (Jev 96.9%, LLM 98.8%) but dropped to about 80% when the examiner used shorthand. Most of Jev’s errors fell on four items (language, extinction and neglect, LOC questions and visual fields), and on each it tended to repeat the same mistake, for example scoring slurred speech as aphasia. Its confidence flagged most of these errors. Re-checking the 14.8% of items with confidence below 0.70 would have caught 86% of them, and a simulated hybrid reached 98.8% accuracy. Conclusions. Both engines scored most NIHSS items correctly from exam transcripts. The LLM was more accurate, and the decision model was faster and cheaper, with a confidence signal that could route uncertain items to an LLM. Because the cases and the key are synthetic, these findings need confirmation with recorded exams and human raters.