AI Insight
This study evaluates DR. INFO, an AI-based medical assistant using agentic RAG (Retrieval-Augmented Generation) architecture, on HealthBench, a benchmark designed to test clinical question-answering through open-ended scenarios. DR. INFO achieved a score of 0.68 on 1,000 challenging cases, substantially outperforming leading large language models including GPT-5 (0.46), Grok 3 (0.23), Gemini 2.5 Pro (0.19), and Claude 3.7 Sonnet (0.02). The system demonstrated particular strengths in communication, instruction following, and accuracy, while showing room for improvement in contextual awareness and response completeness.
Why it matters
This research demonstrates that specialized medical AI assistants with retrieval-augmented architectures can significantly outperform general-purpose language models on clinical queries, suggesting potential for more reliable AI-enabled clinical decision support. The use of rubric-based evaluation on realistic, open-ended scenarios provides a more meaningful assessment framework than traditional multiple-choice benchmarks for high-stakes medical applications.
Understand the Science
Abstract: Evaluating large language models (LLMs) on their ability to generate high-quality, accurate, situationally aware answers to clinical questions requires going beyond conventional benchmarks to assess how these systems behave in complex, high-stakes clinical scenarios. Traditional evaluations are often limited to multiple-choice questions that fail to capture essential competencies such as contextual reasoning, contextual awareness, and uncertainty handling.
To address these limitations, we evaluate our agentic RAG-based clinical support assistant, DR. INFO, using HealthBench, a rubric-driven benchmark composed of open-ended, expert-annotated health conversations. On the Hard subset of 1,000 challenging examples, DR. INFO achieves a HealthBench Hard score of 0.68, outperforming leading frontier LLMs including the GPT-5 model family (GPT-5: 0.46, GPT-5.2: 0.42, GPT-5.1: 0.40), Grok 3 (0.23), Gemini 2.5 Pro (0.19), and Claude 3.7 Sonnet (0.02) across all behavioral axes (accuracy, completeness, instruction following, etc.). In a separate 100-sample evaluation against similar agentic RAG assistants (OpenEvidence and Pathway.md, now DoxGPT by Doximity), it maintains a performance lead with a HealthBench Hard score of 0.72.
These results highlight the strengths of DR. INFO in communication, instruction following, and accuracy, while also revealing areas for improvement in context awareness and response completeness. Overall, the findings underscore the utility of behavior-level, rubric-based evaluation for building reliable and trustworthy AI-enabled clinical support systems.