AI Insight
This paper introduces a self-supervised representation reconstruction (SSRR) loss method for training neural audio codecs that improves speech intelligibility and training efficiency. The proposed JHCodec achieves faster convergence (competitive results after 300k training steps on a single GPU) and better intelligibility scores compared to existing codecs by reconstructing self-supervised representations from codec outputs rather than just optimizing mel-spectrogram reconstruction. The codec operates with zero lookahead, enabling real-time streaming applications while achieving the best word error rate (WER) and character error rate (CER) among evaluated codecs on LibriSpeech test-clean.
Why it matters
This advancement could enable higher quality real-time voice communication and streaming audio applications with lower latency, particularly beneficial for video conferencing, live broadcasting, and assistive technologies that require immediate audio processing. The improved training efficiency also reduces computational costs for developing audio compression systems.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
-cross
Abstract: Neural audio codecs optimized for mel-spectrogram reconstruction often fail to preserve intelligibility. While semantic encoder distillation improves encoded representations, it does not guarantee content preservation in reconstructed speech. In this work, we demonstrate that self-supervised representation reconstruction (SSRR) loss fundamentally improves codec training and performance. First, SSRR significantly accelerates convergence, enabling competitive results after 300k training steps on a single H200 GPU. Second, it enhances intelligibility by reconstructing distilled self-supervised representations from codec outputs. Third, SSRR enables high intelligibility without additional lookahead in streaming Transformer-based codecs, allowing a zero-lookahead architecture for real-time deployment. On LibriSpeech test-clean, JHCodec achieves the best WER and CER among the evaluated codecs while maintaining zero lookahead and low end-to-end latency. We open-source the full implementation, training pipeline, and demo on GitHubh ttps://github.com/jhcodec843/jhcodec.