AI Insight
Researchers investigated how to effectively use synthetic speech from text-to-speech systems to train automatic speech recognition models in privacy-sensitive domains like banking and healthcare. They found that the gap between synthetic and real speech is concentrated in early-to-middle neural network layers and relates to temporal and prosodic features rather than naturalness. By applying room impulse response augmentation and selective layer training, they matched performance of models trained on real data using only 25% real speech (13.6 hours) and exceeded it with higher proportions.
Why it matters
This work addresses a critical challenge in deploying ASR systems in regulated industries where collecting real voice data raises privacy concerns and high costs. The findings provide practical methods to dramatically reduce the amount of real speech data needed for training while maintaining accuracy, potentially enabling better voice recognition systems in healthcare and financial services while protecting user privacy.
Understand the Science
arXiv:2606.29031v2 Announce Type: replace-cross
Abstract: In regulated domains such as banking and healthcare, where privacy constraints make real speech costly to collect and retain, synthetic speech from modern text-to-speech (TTS) is an appealing alternative for training automatic speech recognition (ASR) without exposing sensitive customer recordings. Yet a persistent distributional gap between synthetic and real data limits how far it can replace genuine recordings. Prior work largely treats this gap as a black box to be engineered around, but in our work, we instead examine its origin directly by probing a SLAM-ASR architecture. Then, we localise where its LLM backbone separates real from synthetic speech and find the discriminative signal concentrated in the early-to-middle layers, where temporal and prosodic perturbations disrupt it most. We further show that representation-level separability, help, but does not directly predict downstream ASR gains. On the other hand, convolving synthetic audio with room impulse responses (RIRs) narrows the gap not by making synthetic speech sound cleaner or more natural, but by reproducing the acoustic irregularities of real recordings. Translating these findings into the training procedure, by adding a layer-selection module combined with RIR augmentation matches a fully real-data baseline using only 25% of the real speech (13.6h) and surpasses it at all higher proportions.
Source: How to Leverage Synthetic Speech for LLM-Based ASR Systems?