AI Insight
This study analyzes ESM3, a multimodal protein language model that processes multiple types of protein information simultaneously (sequence, 3D structure, secondary structure, solvent accessibility, and functional annotations). The researchers found that physical modalities start in separate computational spaces but progressively merge into a shared representation between layers 25-35 of the 48-layer network, with sequence joining last around layer 28. Notably, functional annotations remain completely separate from physical modalities throughout all layers, and this fusion pattern occurs consistently at the same network depth across 5,555 proteins from diverse organisms spanning all domains of life.
Why it matters
Understanding how protein language models internally organize different types of biological information can guide the development of more efficient and accurate models for predicting protein structure and function. This knowledge may help researchers design better AI tools for drug discovery, protein engineering, and understanding biological systems by revealing which architectural features are essential for multimodal integration.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Protein language models learn general-purpose representations from large collections of protein sequences and structures, and have advanced the prediction of protein structure and function. ESM3 is a multimodal protein language model that ingests a protein through several channels at once, including amino-acid sequence, three-dimensional structure, secondary structure (SS8), solvent accessibility (SASA), and discrete functional annotations, summing their embeddings into a single residual stream. Little is known about whether these modalities occupy separate subspaces and the depth at which they fuse. The present analysis examines ESM3 (esm3-sm-open-v1; 1.4 billion parameters; 48 transformer layers) once per modality in isolation and applies representational-similarity analysis across all 48 layers. The four physical modalities (sequence, structure, SS8, SASA) begin in distinct subspaces, remain maximally separated through roughly the first half of layers, and then fuse into a shared low-dimensional subspace between layers 25 and 35. The fusion is ordered. The structure-derived modalities (structure, SS8, SASA) are mutually aligned from the input, whereas sequence joins last, after layer 28. The functional-annotation modality never fuses; instead, it remains representationally orthogonal to the physical modalities at every layer, and this orthogonality holds whether the annotation is supplied as whole-protein or per-residue, suggesting that it is content-driven rather than a tokenization artifact. The fusion is a learned property, absent in a randomly initialized model of the same architecture, holds at the residue level below the mean-pool, and reorganizes variance, converting between-condition variance into within-condition variance while the stream never approaches isotropy. Fusion depth is independent of protein length but is delayed by structural disorder. The phenomenon is universal across diverse organisms. Across 5,555 proteins from 12 organisms spanning eukaryota, bacteria, and archaea, every superkingdom (and every individual organism) reaches peak modality fusion at the same network depth (layer 35).
Source: A geometric atlas of how ESM3 organizes modalities across depth