AI Insight
This study tested the security of two advanced AI language models (Anthropic's Fable 5 and Opus 4.8) against automated jailbreak attacks designed to elicit harmful responses. Researchers used the HackAgent framework to generate hundreds of thousands of adversarial prompts across nearly 8,000 harmful intents in ten categories, with results verified by a panel of judge models. Despite strong overall defenses, adaptive iterative attacks successfully bypassed safety measures in 6.1% of cases for Fable 5 and 11.5% for Opus 4.8, producing over 700 and 1,600 confirmed harmful outputs respectively, demonstrating that even state-of-the-art models remain vulnerable to sustained automated attacks.
Why it matters
This research reveals that frontier AI models retain exploitable vulnerabilities despite extensive safety testing, with automated tools able to find weaknesses cheaply and quickly without human expertise. The findings suggest that current AI safety measures are insufficient against determined adversarial approaches, highlighting urgent needs for improved defensive architectures and red-teaming protocols.
arXiv:2606.18193v1 Announce Type: cross
Abstract: We evaluate the adversarial robustness of two frontier large language models (LLMs) developed by Anthropic, Fable 5 and Opus 4.8, against four families of automated jailbreak attack across 7 826 harmful intents spanning a ten-category harm taxonomy. Using the HackAgent red-teaming framework, hundreds of thousands of adversarial attempts were generated and every apparent success was independently re-adjudicated by a panel of three judge models (majority vote). Both models resist the majority of attacks, but the residual surface is larger than aggregate framing suggests: it is dominated by adaptive iterative attacks, while static obfuscation is near-fully neutralised. The strongest adaptive search (tree-of-attacks) breaks Opus 4.8 on 11.5% of intents overall, whereas Fable 5 stays in the single digits (6.1% worst-case). Aggregate rates therefore should not be read as reassurance. Even in these hardened configurations, the two models produced 1 620 (Opus 4.8) and 702 (Fable 5) panel-confirmed harmful completions spanning every harm category, located automatically, cheaply, and within the first one or two refinement steps by an attacker model with no human expert in the loop. The reasonable conclusion is that even the best, most-tested frontier models remain reliably breakable under sustained automated pressure.
Source: A Red-Team Study of Anthropic Fable 5 & Opus 4.8 Models