AI Insight
Researchers used 7 Tesla fMRI brain imaging from 176 participants watching movies to investigate how the brain processes combined audiovisual information. They found that the brain's response to movies with natural sound and vision together was better predicted by models treating audiovisual information as an integrated whole rather than simply adding separate audio and visual components. The strongest effects were observed in auditory, visual, and attention-related brain regions, suggesting the brain creates emergent audiovisual meaning that exceeds the sum of individual sensory inputs.
Why it matters
This research challenges the traditional assumption that multisensory perception can be fully understood by studying each sense separately and combining results. The findings could inform development of more effective multimedia technologies, artificial intelligence systems that better mimic human perception, and therapeutic approaches for sensory integration disorders.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Natural audiovisual perception may not be fully captured by decomposing movies into auditory and visual streams. I introduce a computational-counterfactual framework that keeps movie viewing intact while varying only AI-derived descriptions of the same clips. Using 7 Tesla movie fMRI imaging data from 176 participants, I tested whether cortical responses were better predicted by native audiovisual semantics than by a dimension-matched additive reconstruction from audio-only and video-only descriptions. The native model outperformed the matched additive baseline under content-aware purged cross-validation, with strongest gains in auditory, visual, and dorsal attention systems. Representational-similarity, feature-replacement, and content-gating analyses showed that the advantage reflected feature- and network-specific routing linked to coherent audiovisual semantic emergence rather than raw auditory-visual discrepancy. The effect survived stronger temporal purging and repeat-content exclusion, suggesting that intact movie viewing evokes cortical structure aligned with native audiovisual meaning beyond additive unimodal semantics.
Source: Computational Counterfactuals Reveal Non-Additive Audiovisual Semantics in Natural Movie Responses