AI Insight
Researchers have developed a new approach for detecting mistakes in instructional videos that can generalize to unseen tasks, rather than being limited to predefined scenarios. They introduce the Mistake Detection Video Question Answering (MD-VQA) benchmark and a novel post-training technique for video-language models that uses a specialized reward function to identify discrepancies between written instructions and corresponding video actions. Their method outperforms existing zero-shot and supervised baselines, showing improvements up to 11.6% on unseen procedures.
Why it matters
This work addresses a critical safety concern across domains like manufacturing, healthcare, and training, where detecting execution errors could prevent accidents and improve quality control. By enabling models to generalize to new tasks without retraining, this approach makes automated mistake detection more practical and scalable for real-world applications.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Human mistakes are inevitable when following instructions, yet they can lead to severe consequences. As such, there has been an increased interest in developing methods for detecting mistakes in videos, with current methods mostly focusing on closed-set protocols. While successful in controlled settings, the closed-set assumption limits their wider applicability, as any changes to the task require collecting new data and re-training models. Instead, we argue that mistake detection methods should learn the general concept of a mistake, rather than overfitting to step-specific details. To reflect this, we introduce the Mistake Detection Video Question Answering (MD-VQA) protocol and accompanying benchmark. MD-VQA tests whether methods can discern if a step was executed correctly with respect to its description, for both seen and unseen actions. To address this important challenge, we propose the first video-language-model post-training technique for mistake detection. Our method uses a tailored reward function to encourage the model to identify discrepancies between an instruction and the corresponding video. Extensive evaluations demonstrate that this approach outperforms zero-shot, supervised fine-tuning, and post-training baselines. Notably, our method generalizes especially well to unseen procedures, for instance, with an improvement of up to 11.6% over the best-performing baseline on EP-VQA, paving the way toward general mistake detection. We release our code and benchmark at https://github.com/FedeSpu/mstk.
Source: Post-Training VLMs for Video Mistake Detection