AI Insight
This study evaluated multiple large language models released between mid-2024 and late-2025 on their ability to solve university-level physics problems and grade student work. Recent models like ChatGPT-5.1 and Gemini 3.0 Pro achieved near-perfect accuracy on text-based physics problems across Classical Mechanics, Electromagnetism, and Quantum Mechanics, with improved multimodal capabilities resolving previous difficulties with diagrams and spatial reasoning. While these models showed significant improvement in aligning with human grading standards and reduced systematic over-marking, they continue to struggle with assigning partial credit to incomplete or flawed student reasoning.
Why it matters
These findings suggest LLMs could provide effective automated support for physics education, both as tutoring tools for students and as grading assistants for instructors. However, their persistent limitations in evaluating nuanced, partially correct reasoning require careful human oversight before full deployment in educational assessment contexts.
Understand the Science
arXiv:2605.23660v2 Announce Type: replace
Abstract: The rapid advancement of Large Language Models (LLMs) has introduced new possibilities and challenges in physics education, necessitating rigorous evaluation of their capabilities as both problem solvers and automated assessors. This paper presents the results of three complementary studies that evaluated frontier models released between mid-2024 and late-2025. Models were assessed on their ability to generate accurate, step-by-step solutions to university-level physics problems in Classical Mechanics, Electromagnetism, and Quantum Mechanics, and subsequently on their reliability in grading student solutions against a formal mark scheme. The results indicate a clear trajectory toward benchmark saturation in text-based reasoning, with recent architectures (such as ChatGPT-5.1 and Gemini 3.0 Pro) achieving near-perfect scores. Furthermore, recent advances in native multimodal integration have resolved previous limitations in spatial geometry and topological interpretation, enabling models to accurately process accompanying diagrams. As automated assessors, newer models demonstrated significant improvements in alignment with human grading, heavily mitigating the systemic over-marking observed in earlier iterations. However, while models reliably evaluate fully correct handwritten work, assigning partial credit to flawed or incomplete reasoning remains a persistent challenge. These findings suggest that as of late 2025, LLMs offer viable support for both independent student learning and instructional automation, provided their limitations in evaluating ambiguous reasoning are actively managed.