AI Insight
HiVe-Mem is a new memory framework for multimodal AI agents that improves how they store and retrieve information from long interaction histories. The system uses two complementary strategies: horizontally expanding memory retrieval by organizing episodes in a relational graph that connects related memories, and vertically composing evidence by letting the system adaptively select from multiple forms of stored information (summaries, captions, images, etc.) for each query. Testing across three benchmarks showed consistent performance improvements while reducing computational token usage by approximately 90% compared to existing methods.
Why it matters
This framework addresses a critical challenge for AI agents that interact over extended periods, such as virtual assistants or embodied robots, by enabling more efficient and contextually appropriate memory retrieval. The dramatic reduction in computational resources while maintaining or improving performance could make sophisticated multimodal agents more practical for real-world deployment.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: As multimodal agents accumulate increasingly long interaction histories, effectively retrieving and utilizing past information becomes critical for long-term interaction. Existing memory systems typically rely on semantic retrieval over compressed memories and expose predefined representations, which may miss relevant memories and overlook heterogeneous evidence needs across queries. We introduce textbf{hivemem}, a multimodal memory framework that interleaves textbf{textsc{H}}ortextbf{textsc{i}}zontal Memory Expansion with textbf{textsc{Ve}}rtical Evidence Composition. For horizontal memory expansion, hivemem organizes memory episodes into a relational graph using textual and visual anchors, expanding semantically retrieved seeds with related neighbors to recover complementary information. For vertical evidence composition, each retrieved memory retains multiple evidence forms, including compact summaries, captions, grounded visual views, raw images, and interaction history. A lightweight trainable selector independently composes an evidence subset for each retrieved memory, enabling query-adaptive utilization of heterogeneous multimodal information. This two-dimensional design jointly broadens memory coverage and adapts the evidence contributed by individual memories. Experiments across three multimodal memory benchmarks show that hivemem consistently improves task performance across base models while reducing token consumption by about 90% compared with the most efficient competing multimodal baselines.