AI Insight
This paper introduces OMG-Agent, a new framework for generating missing data modalities in multimodal systems through a three-stage workflow that separates semantic reasoning from signal synthesis. The system uses a multimodal language model for semantic planning, retrieves relevant evidence from external knowledge bases, and synthesizes high-fidelity outputs using retrieved features as prompts. Testing on benchmark datasets showed consistent improvements over existing methods, particularly under severe data incompleteness conditions with up to 70% missing data.
Why it matters
The framework addresses a critical limitation in multimodal AI systems where incomplete data undermines reliability, with potential applications in medical diagnostics, autonomous vehicles, and any domain requiring robust handling of missing sensor or data streams. By reducing hallucinations and improving accuracy under extreme data loss, this approach could make multimodal systems more trustworthy in real-world deployment scenarios.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Data incompleteness severely impedes the reliability of multimodal systems. Existing reconstruction methods face distinct bottlenecks: conventional parametric/generative models are prone to hallucinations due to over-reliance on internal memory, while retrieval-augmented frameworks struggle with retrieval rigidity. Critically, these end-to-end architectures are fundamentally constrained by Semantic-Detail Entanglement — a structural conflict between logical reasoning and signal synthesis that compromises fidelity. In this paper, we present textbf{underline{O}}mni-textbf{underline{M}}odality textbf{underline{G}}eneration Agent (textbf{OMG-Agent}), a novel framework that shifts the paradigm from static mapping to a dynamic coarse-to-fine Agentic Workflow. By mimicking a textit{deliberate-then-act} cognitive process, OMG-Agent explicitly decouples the task into three synergistic stages: (1) an MLLM-driven Semantic Planner that resolves input ambiguity via Progressive Contextual Reasoning, creating a deterministic structured semantic plan; (2) a non-parametric Evidence Retriever that grounds abstract semantics in external knowledge; and (3) a Retrieval-Injected Executor that utilizes retrieved evidence as flexible feature prompts to overcome rigidity and synthesize high-fidelity details. Extensive experiments on multiple benchmarks demonstrate that OMG-Agent consistently surpasses state-of-the-art methods, maintaining robustness under extreme missingness, e.g., a $2.6$-point gain on CMU-MOSI at $70$% missing rates.
Source: OMG-Agent: Toward Robust Missing Modality Generation with Decoupled Coarse-to-Fine Agentic Workflows