AI Insight
This paper presents MVCL-DAF++, an enhanced framework for multimodal intent recognition that addresses challenges with weak semantic understanding and poor performance on rare classes. The system introduces prototype-aware contrastive alignment to match instances with class-level prototypes and a coarse-to-fine attention fusion mechanism that combines global and token-level features across modalities. Testing on MIntRec and MIntRec2.0 benchmarks shows state-of-the-art performance with improvements of 1.05% and 4.18% weighted F1 score respectively for rare-class recognition.
Why it matters
Improved multimodal intent recognition has practical applications in human-computer interaction systems, virtual assistants, and conversational AI where understanding user intent from combined text, audio, and visual signals is critical. The enhanced performance on rare classes is particularly valuable for real-world deployment where some intent categories occur infrequently but remain important to recognize accurately.
Understand the Science
-cross
Abstract: Multimodal intent recognition (MMIR) suffers from weak semantic grounding and poor robustness under noisy or rare-class conditions. We propose MVCL-DAF++, which extends MVCL-DAF with two key modules: (1) Prototype-aware contrastive alignment, aligning instances to class-level prototypes to enhance semantic consistency; and (2) Coarse-to-fine attention fusion, integrating global modality summaries with token-level features for hierarchical cross-modal interaction. On MIntRec and MIntRec2.0, MVCL-DAF++ achieves new state-of-the-art results, improving rare-class recognition by +1.05% and +4.18% WF1, respectively. These results demonstrate the effectiveness of prototype-guided learning and coarse-to-fine fusion for robust multimodal understanding. The source code is available at https://github.com/chr1s623/MVCL-DAF-PlusPlus.