AI Insight
This paper introduces Suan, a new algorithm designed to improve safety alignment in large language models without compromising their helpfulness. The method formulates its optimization objective directly at the gradient level rather than using standard variational approaches, resulting in more interpretable training dynamics. Testing shows that Suan achieves better safety performance than existing methods while maintaining response quality, addressing the common problem of over-refusal seen in current open-weight safety-aligned models.
Why it matters
As AI language models become more widely deployed, ensuring they provide safe responses without becoming overly cautious or unhelpful is critical for practical applications. This approach could help make open-source models as reliably safe as proprietary systems while maintaining their usefulness across diverse tasks.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Integrating robust safety guardrails into Large Language Models (LLMs) is essential for delivering helpful yet harmless responses. While proprietary systems exhibit reliable safety controls, their underlying methodologies and trade-offs remain largely undisclosed. Achieving comparable security in open-weight models remains a persistent challenge, as post-trained variants frequently suffer from over-refusal and degraded general quality. To overcome these drawbacks, we introduce Suan, a novel preference optimization algorithm. Unlike existing methods, we formulate the optimization objective directly at the gradient level, bypassing the standard variational derivation. As a result, we obtain more interpretable and robust training dynamics. Extensive evaluations across a diverse suite of competitive baselines and benchmarks demonstrate that Suan achieves superior safety alignment while fully preserving response utility.
Source: Suan: Rectifying Direct Preference Safety Alignment in Large Language Models