AI Insight
Researchers developed Sci-VBench, a new benchmark containing 1,253 expert-annotated examples across 60 scientific subjects to evaluate AI video generation systems on their ability to accurately represent scientific knowledge and reasoning. Testing 16 state-of-the-art AI models revealed that while most systems produce visually realistic videos with similar perceptual quality scores, they vary dramatically in their ability to correctly represent scientific concepts and causal relationships, with proprietary models significantly outperforming open-source alternatives.
Why it matters
This benchmark addresses a critical gap in evaluating whether AI-generated scientific videos are not just visually appealing but also scientifically accurate. The findings suggest current video generation technology may produce misleading scientific content despite appearing realistic, which has important implications for educational materials, scientific communication, and preventing the spread of visually convincing but factually incorrect information.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: We introduce Sci-VBench, a comprehensive benchmark for evaluating knowledge- and reasoning-intensive video generation across scientific domains. It contains 1,253 expert-annotated examples spanning 60 subjects across four core disciplines: Natural Science, Healthcare, Humanities & Social Sciences, and Engineering. Each example requires models to generate temporally rich videos that demand scientific reasoning and knowledge-grounded synthesis, going beyond surface-level visual plausibility. We further establish a rubric-based evaluation protocol. Our analysis shows that, under this protocol, both non-expert human evaluators and MLLM-as-Judge systems can achieve relatively high agreement with expert judgments, supporting reproducible evaluation at scale. We benchmark 16 frontier proprietary and open-source models and find that, while automatic perceptual-quality scores cluster tightly across systems, performance on Prompt Grounding and Scientific and Causal Correctness varies substantially, with a pronounced proprietary-open-source gap. These findings show that advances in visual realism have not yet translated into reliable modeling of scientific and causal dynamics.
Source: Sci-VBench: Evaluating Knowledge- and Reasoning-Intensive Video Generation in Science Domains