AI Insight
This study introduces SciLaws-Bench, a benchmark dataset containing 118 scientific problems across six disciplines designed to test whether large language models can discover scientific laws. The benchmark includes real-world data from 381 papers and evaluates LLMs in two settings: predicting outcomes using fixed data and recovering modified versions of published equations through simulated experiments. The evaluation reveals that LLMs struggle with three key challenges: achieving good predictive accuracy without scientific validity, failing to recover structural components of equations even when matching published formulas, and difficulty in selecting appropriate candidate equations.
Why it matters
This research provides a rigorous framework for evaluating AI systems in scientific discovery, a capability that could accelerate research across multiple disciplines. The findings highlight critical gaps between pattern-fitting and true scientific understanding, which has important implications for how AI tools should be deployed in scientific research.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
-cross
Abstract: Scientific law discovery has long been central to scientific progress, proceeding through iterative cycles of generating hypotheses, testing them against empirical evidence, and refining them under scientific constraints. As large language models (LLMs) become increasingly involved in scientific research, whether they can discover scientific laws and how to evaluate this ability remain open questions. A central evaluation challenge is to move beyond familiar published equations while keeping discovery tasks grounded in scientific data and constraints. We introduce SciLaws-Bench, a curated collection of scientific task packages grounded in the source literature, each linking a scientific problem, supporting data, published reference equations, and scientific-validity rubrics. Through agent-assisted curation and human verification, we assemble 118 problems spanning six disciplines, drawing on 381 papers, 291 candidate laws, and roughly 8M data points. Each problem supports two complementary evaluation settings. SciLaws-Real uses fixed scientific data to evaluate proposed laws for held-out predictive fit and scientific validity. SciLaws-Parallel evaluates recovery of a newly synthesized structural variant of a published equation through active queries to a simulator calibrated to the source data. Our evaluation reveals three limitations: good predictive fit need not imply scientific validity, recovering a published formula does not establish recovery of its new structural terms, and candidate selection remains a bottleneck in scientific law discovery. Project page: https://yiyihum.github.io/SciLaws-Bench
Source: Can LLMs Discover Scientific Laws in Real and Parallel Worlds?