AI Insight
This study reveals that the choice of negative examples (non-interacting protein pairs) dramatically affects the performance and ranking of machine learning models for predicting protein-protein interactions. By systematically testing four models across six different methods of constructing negative examples, researchers found that evaluation metrics could shift by up to 0.26 AUPRC depending solely on how test negatives were selected, and that commonly used sophisticated negative sampling strategies often underperformed simple uniform random sampling. The findings demonstrate that model comparisons in PPI prediction are systematically biased by methodological choices in negative example construction, which are frequently unreported or inconsistent across studies.
Why it matters
These findings expose a fundamental problem in how protein interaction prediction models are developed and compared, suggesting that many published performance claims may not be robust or comparable across studies. The released evaluation framework enables more rigorous and transparent model development, which is critical for applications in drug discovery and understanding cellular processes.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Many machine-learning tasks, including representation learning, rely on negative examples that are constructed rather than directly observed. This issue is especially important for protein–protein interaction (PPI) prediction, where experimentally established non-interactions are scarce. PPI models are therefore commonly trained and evaluated against sampled, unobserved protein pairs treated as negatives. We study how this construction affects model comparison. Keeping positives, protein-disjoint splits, class balance, and model selection fixed, we vary training and evaluation negatives independently across six constructions and four sequence-based models, three of which use protein language model (pLM) representations. Changing only the test negatives shifts AUPRC by up to 0.26, and the top-ranked architecture changes with the construction. Negatives matched to true partners in pLM representation space reduce all pLM-based models to near-chance performance, and shuffling partners within each class shows that the original pairing contributes little to AUROC. Constructed training negatives rarely outperform uniform sampling: on external data, they underperform it in 108 of 120 comparisons for the pLM-based models, with gains never exceeding 0.05 AUROC and losses up to 0.23. We release splits, negative-generation procedures, and evaluation code that allow training and evaluation negatives to be varied separately. Our results show that with constructed negatives, model comparisons are not just noisy but systematically sensitive to how negatives are defined, a methodological choice that is often left implicit.
Source: Constructed Negatives Shape the Evaluation of Sequence-Based Protein–Protein Interaction Models