AI Insight
FastSLM introduces a hierarchical temporal abstraction architecture that compresses speech input for multimodal large language models by 97%, reducing the representation to 1.67 tokens per second while maintaining linguistic information quality. The system addresses the computational bottleneck in processing long-form speech by progressively distilling acoustic features across multiple temporal scales, unlike previous models that project high-frame-rate features directly. Experimental results show FastSLM achieves competitive performance on speech-language tasks while requiring substantially fewer computational resources than existing approaches.
Why it matters
This advancement could make real-time speech processing in AI systems significantly more efficient and scalable, enabling practical applications in long-form audio analysis, automated transcription, and voice-based assistants. The dramatic reduction in computational requirements may allow deployment of sophisticated speech-language models on devices with limited resources.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
-cross
Abstract: Scaling Multimodal Large Language Models (MLLMs) to long-form speech is bottlenecked by the explosive growth of input tokens. Existing speech-language models project high-frame-rate acoustic features directly into the LLM input space, making long-context processing computationally prohibitive. Unlike images or videos, speech lacks spatial redundancy, making extreme token compression particularly challenging. To address this limitation, we propose FastSLM, a token-efficient architecture featuring the Hierarchical Temporal Abstractor (HTA), which progressively distills acoustic features across multiple temporal scales. HTA achieves an extreme compression rate of 1.67 tokens per second (97% reduction) while preserving essential linguistic information for downstream speech-language understanding. Experimental results demonstrate that FastSLM achieves competitive performance across diverse speech-language tasks while requiring substantially fewer speech tokens and FLOPs than existing speech-language models. The source code and model checkpoints are available at https://github.com/Lee-junseok1025/FastSLM.
Source: FastSLM: Hierarchical Temporal Abstraction for Efficient Long-Form Speech Adaptation