AI Insight
This study introduces Ego-MC-Bench, a benchmark for evaluating whether video large language models can provide real-time corrections during tasks like cooking by detecting mistakes as they happen. Researchers found that current state-of-the-art video LLMs struggle with this challenge, largely due to insufficient training data showing mistakes and timely interventions. To address this gap, they created Ego-CoMist, a synthetic dataset that transforms regular cooking videos into training examples with proactive interventions, which improved performance especially for smaller, device-friendly models.
Why it matters
This research advances the development of AI assistants that could provide real-time guidance for everyday tasks, potentially improving how people learn skills from video tutorials. The work has practical applications for edge devices like smartphones or smart glasses that could offer immediate feedback while users perform activities like cooking or home repairs.
Understand the Science
Abstract: Learning everyday skills, like cooking a dish, relies increasingly on instructional media such as online videos. This opens the door to the use of video (and multimodal) large language models (LLMs) as task guidance assistants. A crucial capability for the real-world success of a prospective task guidance assistant is it’s ability to intervene proactively as soon as a mistake is apparent in order to guide the user. To evaluate this crucial capability, we introduce Ego-MC-Bench (Mistake Corrections), a benchmark for evaluating reactive, step-by-step task guidance in realistic cooking scenarios. Extensive experiments show that Ego-MC-Bench is highly challenging for state-of-the-art video LLMs. We argue that a key reason is the limited availability of training data for fine-tuning models on this task. Although there exists a wide range of cooking video datasets, existing datasets lack examples of mistakes along with appropriately timed interventions. To help address this data limitation, we also introduce Ego-CoMist, a counterfactual synthetic dataset created by transforming non -interactive cooking videos into supervised training examples showing proactive interventions. We show that fine-tuning on Ego-CoMist yields performance gains especially for smaller and more efficient video LLMs that are well suited for delivering assistance on edge devices.
Source: Streaming Interventions: Can Video Large Language Models Correct Mistakes as They Occur?