AI Insight
This study analyzed 13 protein language models across 15 downstream tasks and 9 datasets, finding that the final layers of these models rarely produce the best embeddings for specific applications. The researchers discovered that optimal performance often comes from intermediate layers, with the best layer depending on the similarity between the downstream task and the model's pre-training objective. They developed PLMSommelier, a tool that automatically identifies the best-performing layer with 98% accuracy and creates truncated models for improved efficiency.
Why it matters
This research enables more efficient and accurate use of protein language models in computational biology applications by identifying which model layers work best for specific tasks. The accompanying software tool will help researchers reduce computational costs and improve prediction accuracy in protein analysis applications.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Protein language models (PLMs) have transferred the latest advances from natural language processing to computational biology. These models, trained on large corpora of protein sequence data, are widely used to translate amino acid sequences into latent-space embeddings, ready for use in diverse downstream tasks (DTs). By consensus, embeddings from the models’ last layers are used, while the models’ internal behavior remains poorly understood. We analyzed 13 PLMs across 15 DTs and 9 datasets to assess the value of embeddings from intermediate PLM layers. We trained probe models on embeddings from each layer, compared their performance, and showed that the last layers of PLMs rarely produced embeddings that led to the best results on downstream tasks. Furthermore, we identified a connection between how models learn a certain DT and the similarity between that DT and the pre-training objective. For example, for residue-level downstream tasks, we observed a steady increase in performance across almost all PLM layers, which we attributed to their similarity to most PLMs’ pre-training objectives. To allow the community to capitalize on our findings, we provide PLMSommelier, a Python package that automatically identifies the best PLM layer for a given DT with ~98% accuracy and creates a truncated model using only the early layers up to the best-performing layer. This will help users save time and memory during inference and yield better predictive performance.
Source: Task- and dataset-specific information in protein language models