AI Insight
This paper introduces S-SPPO, a method for aligning large language models with human preferences that addresses instability issues in existing Self-Play Preference Optimization approaches. The researchers found that current methods can degrade when they assign overly confident win predictions to responses that are semantically very similar, and developed a dual-space calibration framework that adjusts win rate targets based on semantic overlap and maintains diversity between chosen and rejected responses. Testing on Llama-3-8B achieved a 52.19% win rate on AlpacaEval 2.0 without requiring additional human-annotated preference data during training.
Why it matters
This work addresses a critical stability problem in training AI language models to better align with human preferences, potentially enabling more reliable and consistent model behavior. The approach reduces dependence on expensive human-annotated data while preventing performance degradation, which could make advanced language model alignment more practical and cost-effective.
Understand the Science
arXiv:2606.01561v2 Announce Type: replace
Abstract: Aligning Large Language Models (LLMs) with human preferences is often formulated via Direct Preference Optimization (DPO). However, the standard Bradley-Terry instantiation of DPO is limited in modeling common departures from transitivity in human preferences. To address this, recent work has introduced Self-Play Preference Optimization (SPPO), which iteratively refines the policy by training on self-generated win-lose pairs. Our investigation, however, reveals a critical instability in SPPO: the optimization is prone to policy degeneration when the preference oracle assigns overly confident wins to semantically indistinguishable responses. To mitigate this, we propose S-SPPO, a dual-space semantic calibration framework comprising: i) Supervision Calibration via semantic gating, which anneals win rate targets toward the maximum-entropy baseline as semantic overlap increases; and ii) Representation Calibration via latent repulsion to enforce geometric diversity to prevent manifold collapse and maintain latent diversity between chosen and rejected samples. Theoretically, we show that the calibration preserves the constant-sum game structure, facilitating convergence to a Nash Equilibrium. Empirically, S-SPPO avoids the performance degradation seen in prior methods, achieving 52.19% win rate and 47.46% length-controlled win rate on AlpacaEval 2.0 with Llama-3-8B, without using additional human-annotated preferences during training. The code will be available at https://github.com/xiwenc1/s-sppo.
Source: S-SPPO: Semantic-Calibrated Self-Play Preference Optimization