AI Insight
This study examines a failure mode in language model fine-tuning where models fail to rank correct tokens above near-synonym alternatives despite decreasing loss. Using density-matrix analysis across five transformer architectures, researchers identify two distinct failure mechanisms: kinematic failure (insufficient commitment to correct tokens) and structural failure (geometric degradation during training). The analysis reveals apparent phase-transition-like jumps that occur entirely in the softmax layer rather than embedding geometry, with dimensionless parameters successfully predicting model behavior across architectures.
Why it matters
These findings reveal a systematic blind spot in language model training that could affect model reliability in contexts requiring precise word choice. The framework provides predictive tools for diagnosing fine-tuning failures and may inform better training strategies, particularly for applications requiring semantic precision such as technical writing or translation.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
-cross
Abstract: Language models fine-tuned where the correct completion must outrank a near-synonym competitor often fail silently. The cross-entropy loss falls monotonically while the correct token never overtakes the competitor in the model’s ranking. We study this across five transformer architectures from two families spanning a sixfold parameter range, on ten contexts whose correct and competing completions share substantial embedding overlap. We build an order parameter combining the predicted distribution with embedding overlap, as a density matrix because that distribution lives over a non-orthogonal basis. It decomposes additively into a signal term tracking commitment to the correct token and a drag term set by how the embedding bulk leaks probability into the score. This isolates two failure modes. In kinematic failure the signal stays too small and the model never commits. In structural failure the drag worsens during fine-tuning, so the model degrades geometrically as its loss falls. The order parameter also shows sharp jumps resembling phase transitions. We test the spontaneous-symmetry-breaking reading by tracking it after every gradient step, and rule it out. The jumps persist under LoRA even though the token embedding matrix never changes. No geometric phase transition is possible when that geometry cannot move, so the discontinuity lies entirely in the softmax readout. A few dimensionless quantities organize the trajectory across architectures. One is consistent across all five models under full fine-tuning. A second sorts architectures into two classes by their bulk embedding distribution and predicts whether LoRA alone can make a sentence commit. As a blind test, the framework predicts a held-out architecture’s critical learning rate to within 2.1% of a later sweep. These results characterize this near-synonym mechanism and need recalibration before extrapolation.
Source: Phantom Transitions in Language Model Fine-Tuning: A Density-Matrix Analysis