AI Insight
This study investigates whether instruction-tuned multimodal large language models (IT-MLLMs) align with human brain activity more strongly than non-instruction-tuned or unimodal models, using fMRI data recorded while participants watched naturalistic movies. By probing six video and two audio IT-MLLMs with 13 task-specific instructions, the researchers found that instruction-tuned video MLLMs showed higher brain alignment than in-context learning models (approximately 9%), non-instruction-tuned multimodal models (approximately 15%), and unimodal baselines (approximately 20%). Notably, IT models showed weak coupling to instruction-text semantics (r=0.14) compared to in-context learning models (r=0.78), suggesting that instruction-tuning produces task-conditioned representational subspaces rather than simple surface-level semantic matching, and that these subspaces correlate with region-specific brain activity patterns.
Why it matters
These findings advance the understanding of how artificial and biological neural systems process multimodal information, with potential implications for building more brain-aligned AI systems and for developing neuroscientific tools that use AI representations to decode or model human cognitive processing.
arXiv:2506.08277v3 Announce Type: replace
Abstract: Recent voxel-wise multimodal brain encoding studies have shown that multimodal large language models (MLLMs) exhibit a higher degree of brain alignment compared to unimodal models. More recently, instruction-tuned multimodal (IT) models have been shown to generate task-specific representations that align strongly with brain activity, yet most prior evaluations focus on unimodal stimuli or non-instruction-tuned models under multimodal stimuli. We still lack a clear understanding of whether instruction-tuning is associated with IT-MLLMs organizing their representations around functional task demands or if they simply reflect surface semantics. To address this, we estimate brain alignment by predicting fMRI responses recorded during naturalistic movie watching (video with audio) from MLLM representations. Using instruction-specific embeddings from six video and two audio IT-MLLMs, across 13 video task instructions, we find that instruction-tuned video MLLMs show higher brain alignment than in-context learning (ICL) multimodal models (~9%), non-instruction-tuned multimodal models (~15%), and unimodal baselines (~20%). Our evaluation of MLLMs across video and audio tasks, and language-guided probing produces distinct task-specific MLLM representations that vary across brain regions. We also find that ICL models show strong semantic organization (r=0.78), while IT models show weak coupling to instruction-text semantics (r=0.14), consistent with task-conditioned subspaces associated with higher brain alignment. These findings are consistent with an association between task-specific instructions and stronger brain-MLLM alignment, and open new avenues for mapping joint information processing in both systems. We make the code publicly available [https://github.com/subbareddy248/mllm_videos].