AI Insight
This study evaluated whether Jev, a decision model that processes text and outputs probabilities or classifications at lower cost than large language models (LLMs), could handle structured clinical tasks. Jev initially underperformed compared to five flagship LLMs across mortality prediction, clinical decision-making, and cancer staging tasks, but performance improved substantially when using developer-recommended input modifications, particularly for cancer staging (accuracy rose from 45.4% to 85.8%). A hybrid approach using Jev for high-confidence decisions and LLMs for uncertain cases matched LLM-only performance while reducing costs by 33-44%.
Why it matters
This approach could make AI-assisted clinical decision support more economically feasible by strategically combining cheaper decision models with expensive LLMs, potentially enabling broader implementation in healthcare settings where cost is a barrier to adoption.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Decision models like Jev read text and return probabilities, categories, or ordinal levels, with confidence scores for the latter two, at a fraction of large language models’ (LLMs) cost. We explored how Jev could be integrated into clinical workflows for all these output types: mortality risk estimation, categorical Choosing-Wisely decisions for patient questions, and ordinal colon cancer staging. With standard LLM-style inputs, Jev trailed five flagship LLMs: mortality AUROC was 0.822 versus 0.837-0.846; Choosing-Wisely accuracy, 87.7% versus 98.4-100%; and staging accuracy, 45.4% versus 94.2-100%. Developer-recommended input alterations improved staging accuracy to 75.8% with medical context, 82.1% with task decomposition, and 85.8% with both. None of Jev’s staging errors had full confidence. Using Jev for fully confident staging decisions and LLMs for the rest was non-inferior to each LLM alone and cut flagship LLM costs by 33-44%. Prompted as designed, decision models may efficiently share structured clinical tasks with LLMs.
Source: Decision models can offload structured clinical tasks from large language models