AI Insight
This study investigated how visible speech articulations help listeners understand speech in noisy environments by examining the distinct contributions of articulatory timing versus shape. Using EEG recordings while participants watched videos of a speaker either with visible mouth movements or with the mouth obscured by a dynamic ellipse that preserved timing but not shape, researchers found that articulatory shape specifically enhances the brain's processing of phonetic features and overall comprehension. The findings demonstrate that the visual benefit of seeing a speaker's mouth during conversation comes primarily from observing the actual shape of mouth movements, not just their timing.
Why it matters
These results could inform the design of communication technologies for hearing-impaired individuals and improve video conferencing systems in noisy environments. Understanding which visual aspects of speech are most critical for comprehension may also guide the development of more effective speech therapy interventions and assistive devices.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
In noisy environments, visible speech articulations improve listening comprehension. The benefit derives from several sources, including articulatory timing and shape. Recent research has shown that visual cortex encodes a categorical representation of articulatory features and that visual speech can benefit both acoustic and phonetic feature processing separately. The present study advances the hypothesis that the shape of the articulators specifically influences the categorization of auditory speech in terms of its phonetic features. We tested this by linearly modeling electroencephalographic responses to natural, continuous speech (in noise) in terms of the acoustic and articulatory features of the speech. We compared the performance of these models in conditions where the speech was accompanied by a natural video of the speaker with their mouth visible, and a video where their mouth was covered by a dynamic ellipse obscuring articulatory shape but preserving dynamics. The dynamic mask reduced comprehension, neural processing of phonetic features, the associated multisensory benefits, and indices of visual-only linguistic processing over occipital scalp. Our findings support substantial visual involvement in speech comprehension, derived largely from the shape of the articulators. They also corroborate several proposals involving audiovisual speech processing hierarchy and the nature of the information contained in visible speech.
Source: Articulatory timing and form support distinct neural benefits during audiovisual speech