AI Insight
This paper investigates training instabilities in large language models, specifically output logit divergence that occurs during pretraining. The researchers identify anisotropic output embeddings as the root cause and propose Output Embedding Centering (OEC) as a solution, which can be implemented either as a deterministic operation (μ-centering) or as a regularization method (μ-loss). Their experiments demonstrate that OEC suppresses logit divergence more effectively than the commonly used z-loss method and performs comparably to logit soft-capping, while also being less sensitive to hyperparameter tuning.
Why it matters
Training large language models is extremely expensive, and instabilities can waste significant computational resources or compromise model quality. This work offers a more principled approach to preventing training failures by addressing the underlying geometric cause rather than just symptoms, potentially leading to more reliable and cost-effective LLM development.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
-cross
Abstract: Pretraining of large language models is not only expensive but also prone to certain training instabilities. A specific instability that often occurs at the end of training is output logit divergence. The most widely used mitigation strategies, z-loss and logit soft-capping, merely address the symptoms rather than the underlying cause of the problem. In this paper, we analyze the instability from the perspective of the output embeddings’ geometry and identify anisotropic embeddings as its source. Based on this, we propose output embedding centering (OEC) as a new mitigation strategy, and demonstrate that it suppresses output logit divergence. OEC can be implemented in two different ways: as a deterministic operation called $mu$-centering, or a regularization method called $mu$-loss. Our experiments show that both variants outperform z-loss in terms of training stability, while being on par with logit soft-capping. This holds true both in the presence and the absence of weight tying. As a secondary result, we find that $mu$-loss is significantly less sensitive to regularization hyperparameter tuning than z-loss.
Source: Output Embedding Centering for Stable LLM Pretraining