AI Insight
Researchers developed NeuroCodec, a new system that uses brain signals (EEG) to extract a target speaker's voice from multi-talker environments by operating in the latent space of a neural audio codec rather than using traditional time-frequency masking. The system employs graph convolutions to model EEG data, combines brain and speech signals through attention mechanisms, and uses a six-phase training approach with multiple objectives to generate high-quality speech. Testing on public datasets showed NeuroCodec outperformed existing methods, achieving 82.24% accuracy in listening tests compared to 49.63% for the baseline system, while also enabling high-fidelity 44.1 kHz audio reconstruction up to 20 kHz frequency range.
Why it matters
This technology could significantly improve hearing assistive devices by allowing them to automatically focus on the speaker a user is paying attention to, based on their brain activity. The system's real-time processing capability and high audio quality make it potentially suitable for practical implementation in next-generation hearing aids and cochlear implants.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Driven by recent advancements in hearing assistive devices, target speaker extraction (TSE) in multi-talker environments has drawn significant attention. To address this, hearing technologies identify attention using neural signals recorded via electroencephalography (EEG) to guide TSE, enabling neuro-steered auditory attention decoding. However, most existing techniques rely on time-frequency masking, which suffers from low fidelity in spectro-temporally overlapping regions and is often restricted to narrowband audio. In contrast, we introduce NeuroCodec, which, to the best of our knowledge, is the first generative EEG-guided TSE framework operating in the latent space of a neural audio codec. Specifically, NeuroCodec models spatial EEG dependencies via graph convolutions and fuses cross-modal EEG and speech representations through attention mechanisms and Mamba-based temporal modeling. We further introduce a six-phase training curriculum that progressively incorporates latent reconstruction, multi-scale spectral, and adversarial objectives to ensure stable, high-fidelity speech generation. Experiments on the public Cocktail Party and KUL datasets show NeuroCodec outperforms baselines, across several perceptual metrics. A listening experiment with 20 participants validated that NeuroCodec offers greater perceptual accuracy, achieving a mean score of 82.24 compared with 49.63 for M3ANet. Additionally, the proposed method is seen to enable 44.1 kHz reconstruction with spectral content extending up to 20 kHz, further supporting the perceptual results. Finally, we extend NeuroCodec with a causal streaming formulation suited for real-time processing, providing a basis for hearing assistive devices.
Source: NeuroCodec: EEG-Guided Latent-Domain Generative Model for High-Fidelity Target Speaker Extraction