AI Insight
Researchers developed Nemotron-Labs-Audex-30B-A3B (Audex), a unified large language model that processes both audio and text through a single decoder architecture. The model was trained on 157.4B audio tokens and 320.5B text tokens using multi-stage supervised learning, achieving state-of-the-art performance across multiple audio tasks including speech recognition, translation, text-to-speech, and audio generation. Notably, the system maintains the reasoning and language capabilities of its text-only predecessor without significant performance degradation while adding comprehensive audio intelligence.
Why it matters
This unified approach demonstrates that AI systems can excel at both audio and text processing simultaneously without sacrificing performance in either domain, potentially enabling more capable voice assistants, real-time translation systems, and multimodal AI applications. The release of model checkpoints supports open research and accelerates development in audio-language AI.
Understand the Science
arXiv:2607.05196v2 Announce Type: replace-cross
Abstract: Audio intelligence involves understanding, reasoning about, and generating both audio and speech. In this work, we introduce Nemotron-Labs-Audex-30B-A3B (Audex), a unified audio-text LLM built on Nemotron-Cascade-2-30B-A3B, a strong text-only MoE LLM. Audex adopts a simple unified design with a single Transformer decoder: audio inputs are encoded and projected into the text embedding space, while text tokens and quantized audio output tokens are treated uniformly during generation. This architecture enables strong audio-text fusion, seamless multimodal generation, and compatibility with standard LLM training and inference infrastructure. For training, we meticulously curate audio-text datasets comprising 157.4B audio tokens and 320.5B text tokens. We apply multi-stage supervised training on these datasets, followed by text-only Cascade RL and multi-domain on-policy distillation. Audex delivers state-of-the-art audio understanding, speech recognition and translation, text-to-speech, audio generation, and speech-to-speech generation, while preserving very compelling reasoning, alignment, knowledge, long-context, and agentic capabilities of its text-only LLM backbone with marginal or no regression. We release the model checkpoints to facilitate open research.
Source: Unified Audio Intelligence Without Regressing on Text Intelligence