AI Insight
Researchers have developed "Scientist World," a platform that converts published research papers into executable environments where AI systems can attempt to solve the same research problems that human scientists tackled. The system extracts research questions from papers, reproduces baseline methods, and evaluates AI-Scientists against human performance. Testing on 1,000 verified environments showed that current frontier AI-Scientists surpass the original human researchers' results 74% of the time.
Why it matters
This platform provides both a standardized benchmark for evaluating AI research capabilities and a training ground for improving AI-Scientists. The 6,400 hours of recorded research traces could accelerate AI development in scientific discovery, and the demonstrated ability to improve smaller models by 158% in research ideation suggests potential for making AI-assisted research more accessible.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: AI-Scientists arrived, but a world for them to grow in did not. Human scientists take theirs for granted $-$ centuries of open problems and shared ideas, waiting in the scientific literature. To AI-Scientists, that world is readable but not runnable, so we build Scientist World for them: every research paper becomes an executable environment. Our autonomous pipeline extracts each paper’s open-ended research question and reproduces its proposed method as the baseline; an AI-Scientist entering the environment is evaluated against what the human authors achieved. We release the first 1,000 verified environments. With the world as an evaluation, we found that frontier AI-Scientists can already surpass the human researchers 74% of the time. As a record of how AI does science, it contains 6,400 hours of research traces, with every idea, experiment, failure, and success. As a training ground, we are able to raise (+158%) a small model to match frontier models in research ideation. At last, AI-Scientists’ own world has arrived, grown from and growing with science itself.
Source: Scientist World: Grow a Research World for AI-Scientists