AI Insight
This paper introduces Activation Watermarking (AWM), a method for monitoring large language models that makes it harder for attackers to find prompts that produce harmful outputs while evading detection. AWM works by fine-tuning models to align hidden states with secret key-derived directions when responses violate policies, then detecting violations through activation similarity tests. At 1% false-positive rates, AWM reduces evasion success by more than half compared to existing methods and can identify which specific policy was violated with 80% accuracy across 20 monitored policies.
Why it matters
This research addresses a critical security gap where attackers can test harmful prompts offline against publicly known monitoring systems before deploying them. The technique could help AI providers better detect misuse of deployed models while maintaining low false-positive rates, and the attribution capability enables providers to understand which types of policies are being violated.
Understand the Science
-cross
Abstract: Providers monitor deployed large language models (LLMs) to detect misuse that they cannot prevent. LLM monitoring is deterministic and often openly available, so $emph{adaptive}$ attackers with a local copy can search offline for prompts that elicit harmful behavior and evade detection. These attacks are especially concerning because providers never observe the misuse and cannot patch their defenses post-hoc. The core challenge is resisting adaptive attackers while preserving detection rates against non-adaptive ones. We propose $emph{Activation Watermarking}$ (AWM), which randomizes monitoring through limited fine-tuning that aligns the LLM’s hidden states with a secret key-derived direction whenever a response violates a policy. Detection is a similarity test on activations the provider already computes, and attackers who know everything but the key must optimize against differently keyed surrogate detectors. At a matched $1%$ false-positive rate, such surrogate-based attacks evade every evaluated baseline at least $79%$ of the time, but AWM more than halves this evasion rate. AWM also achieves the lowest evasion rate on three of four non-adaptive jailbreak families at a small drop in utility measured across seven benchmarks. Because random high-dimensional directions are near-orthogonal, assigning one watermark per policy enables attribution. Across 20 monitored policies, AWM identifies the violated policy with $80%$ accuracy versus a $5%$ chance baseline. AWM is (i) efficient, (ii) substantially more robust against adaptive attackers than related work and (iii) can attribute which policies were violated.
Source: Adaptively Robust LLM Monitoring via Activation Watermarking