AI Insight
Researchers developed DSAEval, a comprehensive benchmark containing 641 real-world data science problems based on 285 diverse datasets to evaluate LLM-based data science agents. The benchmark tests agents across structured and unstructured data (including images and text) using multimodal environment perception, multi-query interactions, and multi-dimensional evaluation metrics. Testing 13 advanced agentic LLMs revealed that Claude-Sonnet-4.5 achieved the strongest overall performance, while multimodal perception improved vision-related task performance by 2.04% to 11.30%, though significant challenges remain in handling unstructured data domains.
Why it matters
This work establishes a standardized framework for evaluating AI agents designed to automate data science tasks, which could accelerate the development of more capable autonomous data analysis tools. The findings highlight current limitations in handling unstructured data, providing clear directions for improving AI systems intended to assist or replace human data scientists in real-world applications.
arXiv:2601.13591v2 Announce Type: replace
Abstract: Recent LLM-based data agents aim to automate data science tasks ranging from data analysis to deep learning. However, the open-ended nature of real-world data science problems, which often span multiple taxonomies and lack standard answers, poses a significant challenge for evaluation. To address this, we introduce DSAEval, a benchmark comprising 641 real-world data science problems grounded in 285 diverse datasets, covering both structured and unstructured data (e.g., image and text). DSAEval incorporates three distinctive features: (1) Multimodal Environment Perception, which enables agents to interpret observations from multiple modalities, including text and vision; (2) Multi-Query Interactions, which mirror the iterative and cumulative nature of real-world data science projects; and (3) Multi-Dimensional Evaluation, which provides a holistic assessment across reasoning, code, and results. We systematically evaluate 13 recent advanced agentic LLMs using DSAEval. Our results show that Claude-Sonnet-4.5 achieves the strongest overall performance, MiMo-V2-Pro and GPT-5.2 lead in duration and step efficiency, respectively, and MiMo-V2-Flash is the most cost-effective. We further demonstrate that multimodal perception consistently improves performance on vision-related tasks, with gains ranging from 2.04% to 11.30%. Overall, while current data science agents perform well on structured data and routine data analysis workflows, substantial challenges remain in unstructured domains. Finally, we offer critical insights and outline future research directions.
Source: DSAEval: Evaluating Data Science Agents on a Wide Range of Real-World Data Science Problems