AI Insight
Researchers developed a novel method for predicting the difficulty of medical exam questions by using large language models (LLMs) in two capacities: as judges rating question difficulty and as simulated test-takers. When LLMs provided pairwise comparisons of question difficulty and their responses were analyzed for patterns like answer confidence and entropy, the combined approach achieved Spearman correlations of 0.447 on USMLE and 0.494 on PANCE exams, substantially outperforming previous methods and matching the performance of proprietary frontier models like Claude Opus 5.
Why it matters
This approach could help medical educators and exam developers predict how difficult new test questions will be before administering them to actual students, using locally-deployable open-source models rather than expensive proprietary systems. This is particularly valuable for high-security exam materials that cannot be shared with external cloud services.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
-cross
Abstract: Predicting medical item difficulty from item text alone remains challenging. Prior work has largely relied on linguistic features or text representations derived from general-purpose and domain-specific language models, but predictive performance has remained modest. This study takes a different approach by using difficulty-related signals that emerge when open-weight large language models (LLMs) interact with test items. Using USMLE and PANCE medical licensure examination items, we examined LLMs in two roles. As judges, the models provided absolute difficulty ratings and pairwise difficulty comparisons. As simulated examinees, they generated response-based signals including correctness, keyed-option probability, response entropy, and the probability margin between the top two options. We then combined the strongest judgment and response signals with baseline item features in downstream prediction models. Pairwise difficulty judgment was the most consistent individual LLM signal across both datasets and outperformed absolute ratings, and adding simulated-examinee signals yielded only small further gains. The best configurations reached a Spearman correlation of $rho$ = .447 on USMLE, exceeding the best system in the 2024 BEA shared task ($rho$ = .25), and $rho$ = .494 on PANCE. These results were comparable to direct absolute judgments from the proprietary frontier model Claude Opus 5 ($rho$ = .443 and .488, respectively). Overall, judgment and response signals from open-weight LLMs captured difficulty-related information beyond text representations. Because these models can be deployed locally, the approach offers a practical option for predicting the difficulty of unreleased or high-security test items.
Source: LLMs as Judges and Examinees for Medical Item Difficulty Prediction