AI Insight
This paper introduces Suan, a new algorithm designed to improve safety alignment in large language models without compromising their usefulness. The method reformulates preference optimization at the gradient level rather than using standard variational approaches, which the authors claim produces more interpretable and stable training. Testing across multiple benchmarks shows that Suan achieves better safety controls than existing methods while maintaining response quality, addressing the common problem of over-refusal in safety-trained models.
Why it matters
As AI language models become more widely deployed, balancing safety with utility is critical. This work offers a potential solution to a key challenge in open-source AI development: making models as safe as proprietary systems without making them overly cautious or less helpful in general use.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Integrating robust safety guardrails into Large Language Models (LLMs) is essential for delivering helpful yet harmless responses. While proprietary systems exhibit reliable safety controls, their underlying methodologies and trade-offs remain largely undisclosed. Achieving comparable security in open-weight models remains a persistent challenge, as post-trained variants frequently suffer from over-refusal and degraded general quality. To overcome these drawbacks, we introduce Suan, a novel preference optimization algorithm. Unlike existing methods, we formulate the optimization objective directly at the gradient level, bypassing the standard variational derivation. As a result, we obtain more interpretable and robust training dynamics. Extensive evaluations across a diverse suite of competitive baselines and benchmarks demonstrate that Suan achieves superior safety alignment while fully preserving response utility.
Source: Suan: Rectifying Direct Preference Safety Alignment in Large Language Models