AI Insight
Brain-OF is a multimodal foundation model jointly pretrained on three major neuroimaging modalities — fMRI, EEG, and MEG — within a unified framework, addressing the longstanding limitation of modality-specific brain models. The architecture introduces an Any-Resolution Neural Signal Sampler to reconcile the heterogeneous spatiotemporal resolutions across modalities, and combines DINT attention with a Sparse Mixture of Experts to handle both shared and modality-specific neural representations. A dual-domain self-supervised pretraining objective called Masked Temporal-Frequency Modeling reconstructs brain signals simultaneously in the time and frequency domains, enabling richer feature learning across approximately 40 datasets.
Why it matters
A unified brain foundation model capable of processing fMRI, EEG, and MEG could accelerate progress in clinical neuroscience applications such as brain-computer interfaces, neurological disorder diagnosis, and cognitive state decoding by leveraging complementary strengths of each modality rather than relying on siloed single-modality systems.
arXiv:2602.23410v3 Announce Type: replace-cross
Abstract: Brain foundation models have achieved remarkable advances across a wide range of neuroscience tasks. However, most existing models are limited to a single functional modality, restricting their ability to exploit complementary spatiotemporal dynamics and the collective data scale across different neuroimaging techniques. This limitation largely arises from severe semantic heterogeneity and resolution discrepancies among modalities. To address these challenges, we propose Brain-OF, an omnifunctional brain foundation model jointly pretrained on fMRI, EEG and MEG, capable of handling both unimodal and multimodal inputs within a unified framework. To reconcile heterogeneous spatiotemporal resolutions, we introduce the Any-Resolution Neural Signal Sampler, which projects diverse brain signals into a shared semantic space. To further manage semantic shifts, the Brain-OF backbone integrates DINT attention with a Sparse Mixture of Experts, where shared experts capture modality-invariant representations and routed experts specialize in modality-specific semantics. Furthermore, to explicitly internalize the characteristics of neural activity through self-supervised learning, we propose Masked Temporal-Frequency Modeling, a dual-domain pretraining objective that jointly reconstructs brain signals in both the time and frequency domains. Brain-OF is pretrained on a large-scale corpus comprising around 40 datasets and demonstrates superior performance across diverse downstream tasks, highlighting the benefits of joint multimodal integration and dual-domain pretraining.
Source: Brain-OF: An Omnifunctional Foundation Model for fMRI, EEG and MEG