AI Insight
Researchers developed OmegaUse-OfficeVal, a benchmark to evaluate how well AI language model agents can complete complex office tasks compared to humans. The benchmark includes 100 real-world office tasks that typically require 2.32 hours of human labor each, with economic comparisons built in to measure both cost and quality. Testing showed that while current AI models are faster and cheaper than human workers, they have not yet achieved human-level quality in their deliverables.
Why it matters
This research provides a standardized way to measure AI's readiness to automate knowledge work in real business settings. The economic grounding allows organizations to make informed decisions about when AI assistance is cost-effective versus when human expertise is necessary.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Large language model (LLM) agents are increasingly expected to assist users in completing tasks. However, existing benchmarks provide limited support for evaluating whether agents can carry out office-suite workflows at a reasonable cost. We introduce OmegaUse-OfficeVal, a benchmark for evaluating LLM agents on long-horizon office-suite tasks with task-level economic grounding. The benchmark comprises 100 tasks derived from office-suite requests proposed by practitioners and adapted through a privacy-preserving process. On average, these tasks require 2.32 hours of human labor to complete. An important feature of the benchmark is that each task is paired with two economic signals: human labor time and task price proxy. These signals enable direct comparisons between human costs and LLM inference costs, as well as value-weighted evaluation. To support stable evaluation, we develop code-based verifiers from fine-grained rubrics. We evaluate several frontier LLMs together with a human baseline. Although all evaluated LLMs are substantially cheaper and faster than human workers, they have not yet approached human-level deliverable quality. The code and dataset are fully open-sourced, and more information is available on our project website: https://omegause-officeval.github.io.