AI Insight
This study investigates how poisoned training data degrades the performance of large language models, finding that performance degradation follows a power law with non-integer exponents rather than the expected quadratic relationship. Through theoretical analysis and experiments with OLMo-style models, researchers demonstrated that this scaling behavior depends on the order of statistical limits and is connected to heavy-tailed data distributions and finite training times. Using solvable ridge regression models, they proved that excess risk scales differently depending on whether high-dimensional or large-sample limits are taken first, with exponents determined by the tail behavior of the data distribution.
Why it matters
Understanding how training data contamination affects AI model performance at scale is critical for ensuring the security and reliability of large language models. This research provides theoretical foundations for predicting vulnerabilities in AI systems and could inform strategies for detecting and mitigating data poisoning attacks during model development.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Pre-training data poisoning of large language models is usually studied using targeted backdoors and their survival through safety post-training, which leaves open a more basic question: how does a model’s clean data performance degrade as the poison rate $varepsilon$ grows? Motivated by our controlled pre-training runs of OLMo-style models, in which the relative clean data validation perplexity increase $Delta$ between poisoned and clean models matched in architecture, token budget, and optimization schedule is well fit by a power law $Deltaapprox Cvarepsilon^{a}$ with a non-integer exponent, we ask what such a law requires theoretically. We first prove an analyticity barrier: whenever the contaminated objective depends analytically on $varepsilon$ around a nondegenerate clean data optimum, $Delta$ is generically quadratic in $varepsilon$, so a generic non-integer exponent is a signature of genuinely singular structure. We then supply that structure in solvable truncated ridge regression with heavy-tailed covariates, controlled by $q_star$, and a label-shift poisoning. Our central result is that the excess risk scaling exponent depending on the order of limits: in the higher dimensional proportional regime it is $epsilon^{q_star/(q_star+2)}$, whereas taking the ample-data limit first gives $epsilon^{2-2/q_star}$, and the limits do not commute. We confirm this prediction through several numerical simulations. Finally, we argue that finite training time plays the role of truncation on the curvature spectrum in local LLM pre-training, deriving the observed scaling law under heavy tailed inverse curvature spectrum as a modeling hypothesis.
Source: A Solvable Theory of Pre-training Data Poisoning: Regime-Dependent Scaling Exponents