AI Insight
This research introduces LA-RAG, a system for answering questions about multi-hour audio recordings by converting continuous audio into timestamped event records stored in a database, then retrieving relevant segments to generate answers. The system was tested on 24-hour audio benchmarks from home and industrial IoT environments, achieving 76.88% and 71.10% accuracy respectively with sub-0.6 second query latencies. When combined with existing large audio-language models, the structured retrieval approach improved temporal detection performance by 11-17% while reducing processing time.
Why it matters
This work addresses a practical limitation of current AI audio systems that struggle with long recordings due to context length constraints and computational costs. The approach has applications in surveillance, industrial monitoring, and smart home systems where users need to query hours of audio efficiently.
Understand the Science
-cross
Abstract: Answering natural-language questions over multi-hour audio requires both event recognition and temporal grounding. Current large audio-language models perform well on short clips, but are limited by context length, query-time cost, and weak temporal localization. We present LA-RAG (Long Audio-Retrieval Augmented Generation), a structured framework that converts continuous audio into timestamped event records using an open-vocabulary Audio Grounding Model (AGM), stores them in a SQL event database, and answers queries through intent-aware retrieval followed by LLM-based generation. LA-RAG supports offline grounding mode, where long recordings are pre-indexed for low-latency QA, and inference-time grounding mode, where query-conditioned grounding is performed for shorter open-ended clips. We create 24-hour Home-IoT and Industrial-IoT audio benchmarks and augment CASTELLA, a real-world audio moment retrieval dataset with QA pairs. In offline grounding mode, LA-RAG achieves 76.88% overall accuracy on Home-IoT and 71.10% on Industrial-IoT, with average query latencies below 0.6 seconds. In inference-time grounding mode, state-of-the-art LALMs achieve competitive event-detection accuracy on CASTELLA-QA but low temporal detection F1. We further show that LALMs augmented with our structured retrieval metadata achieve consistent temporal detection improvements, with F1 gains of 11-17% across baseline models with improved latency. These results show that explicit timestamped grounding and structured retrieval provide a practical complement to generative audio-language models for deployment-oriented long-audio QA.
Source: Event-Grounded Question Answering over Long Audio via Structured Retrieval