AI Insight
NeutronGym is a new benchmark environment that tests whether large language models can actually perform physics-based design tasks rather than simply recall information. The system evaluates AI agents on designing neutron scattering instruments through executable simulations, using a multi-level grading system that assesses syntax, runtime, structure, and scientific accuracy without relying on another LLM as a judge. Reinforcement learning training improved a smaller model's performance from 11% to 77% on held-out test cases, though frontier models still significantly outperformed trained models on the curated benchmark tasks.
Why it matters
This work addresses a critical challenge in evaluating AI capabilities for scientific research by creating verifiable, physics-based tasks that cannot be solved through memorization alone. The approach could help determine whether AI systems can genuinely assist in scientific instrument design and other complex engineering tasks requiring real understanding of physical principles.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Designing a scientific instrument tests whether language-model agents can do physics rather than recall it, provided the grading cannot be argued with. We introduce NeutronGym, to our knowledge the first executable environment for neutron instrument design: agents build instruments through validating tools, McStas ray-traces what they build, and a level-resolved ladder grades syntax, runtime, structure and science with no LLM judge. Procedural families supply unlimited instances of a fixed layout whose design parameters the agent must set, with held-out parameter regimes; a curated slice, McStasBench, adds 16 tasks from published instruments behind memorization probes and a sandbox. Seven models reproduce at most 7 of the 16, none retrieves a reference, and none meets an improvement target. The environment also trains. Reinforcement learning on its reward takes Qwen3-8B from 11% to 77% of held-out instances of a family whose targets come from a hidden design (69% at a second seed), past an untrained Qwen3-32B, and the recipe holds, at one seed each, on three further gated families. The analysis says what that gain is. Without the ladder’s partial credit it collapses by 60 points. From reward alone the trained model reaches what a classical optimizer reaches, at the agent’s simulation budget, only when handed the closed-form physics (77% against 81%, a gap that does not separate at this size), while frontier models still solve 98-99%. Getting a trustworthy result meant failing four task designs that no-model baselines could solve, and we release the probes that found them.
Source: NeutronGym: Physics-Graded Neutron Instrument Design for LLM Agents