Elamur Research⌕
☰
home / papers / 2025 / Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming

Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming

Paper · 2025 · Mrinank Sharma, Meg Tong, Jesse Mu et al.

Summary

The paper tackles universal jailbreaks: prompting strategies that consistently get past an LLM's safeguards across almost all queries in a domain. In the CBRN domain, such attacks could give non-experts step-by-step help with dangerous procedures.

The authors propose Constitutional Classifiers. A natural-language constitution lists allowed and forbidden content categories, and an LLM uses it to generate synthetic training data. Data augmentation and pools of benign examples are added to keep false positives low. Two kinds of classifiers guard the model:

  • an input classifier screens prompts;
  • a streaming output classifier scores the response token by token, so generation can stop as soon as harm appears.

Prototype classifiers built on Claude 3.5 Sonnet went through a large bug-bounty red-teaming effort, and no participant found a reliable universal jailbreak. Later classifiers built on Claude 3.5 Haiku used a chemical-weapons constitution and blocked most unseen jailbreak attempts, with only modest extra refusals and compute.

The work matters because it shows strong jailbreak defense can still be cheap enough to deploy. Changing the constitution also lets the system adapt to new threats.

Key points

  • Red teaming ran through HackerOne with 405 participants and bounties of up to $15K. Effort was estimated at over 3,000 hours (mean estimate 4,720; 90% CI 3,242–7,417).
  • Red teamers targeted ten CBRN questions. No report answered all ten at even half the detail of an unguarded model. Under the stricter criterion (detail comparable to an unguarded model), nobody answered more than six.
  • On automated tests, the improved Claude 3.5 Haiku classifiers refused over 95% of held-out jailbreaks. The unguarded system refused only 14%.
  • The deployment cost was small: refusals on production Claude.ai traffic rose by 0.38% in absolute terms, and inference overhead was 23.7%.
  • One apparent universal jailbreak was actually an infrastructure bug: up to 128 tokens slipped through after the output classifier had flagged harm. This shows the deployment pipeline needs testing too, not just the classifiers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢AnthropicSan Francisco · United States6 works · 1,536 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Bias, fairness & safety · 517 works

Cite

Cite as Mrinank Sharma, Meg Tong, Jesse Mu, Jerry Wei, Jorrit Kruthoff, Scott Goodfriend, Euan Ong, Alwin Peng, Raj Agarwal, Cem Anil, Amanda Askell, Nathan Bailey, Joe Benton, Emma Bluemke, Samuel R. Bowman, Eric Christiansen, Hoagy Cunningham, Andy Dau, Anjali Gopal, Rob Gilson, Logan Graham, Logan Howard, Nimit Kalra, Taesung Lee, Kevin Lin, Peter Lofgren, Francesco Mosconi, Clare O'Hara, Catherine Olsson, Linda Petrini, Samir Rajani, Nikhil Saxena, Alex Silverstein, Tanya Singh, Theodore Sumers, Leonard Tang, Kevin K. Troy, Constantin Weisser, Ruiqi Zhong, Giulio Zhou, Jan Leike, Jared Kaplan and Ethan Perez. 2025. Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming. arXiv:2501.18837.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2501.18837 · arXiv 2501.18837

@misc{sharma2025constitutional,
  title = {{Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming}},
  author = {Mrinank Sharma, Meg Tong, Jesse Mu, Jerry Wei, Jorrit Kruthoff, Scott Goodfriend, Euan Ong, Alwin Peng, Raj Agarwal, Cem Anil, Amanda Askell, Nathan Bailey, Joe Benton, Emma Bluemke, Samuel R. Bowman, Eric Christiansen, Hoagy Cunningham, Andy Dau, Anjali Gopal, Rob Gilson, Logan Graham, Logan Howard, Nimit Kalra, Taesung Lee, Kevin Lin, Peter Lofgren, Francesco Mosconi, Clare O'Hara, Catherine Olsson, Linda Petrini, Samir Rajani, Nikhil Saxena, Alex Silverstein, Tanya Singh, Theodore Sumers, Leonard Tang, Kevin K. Troy, Constantin Weisser, Ruiqi Zhong, Giulio Zhou, Jan Leike, Jared Kaplan and Ethan Perez},
  year = {2025},
  doi = {10.48550/arxiv.2501.18837},
  eprint = {2501.18837},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2501.18837}
}

References

47 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint, 2016. URL https://arxiv.org/abs/1610.01644. Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Jailbreaking leading safety-aligned LLMs with simple adaptive attacks. arXi… ↗
  2. Anthropic. Anthropic’s responsible scaling policy, 2023a. URL https://www-cdn.anthropic.com/1adf000c8f675958c2ee23805d91aaade1cd4613/responsible-scaling-policy.pdf. ↗
  3. Anthropic. Model card and evaluations for claude models, 2023b. URL https://www-cdn.anthropic.com/bd2a28d2535bfb0494cc8e2a3bf135d2e7523226/Model-Card-Claude-2. pdf. ↗
  4. Anthropic. Claude’s character, claude-character. 2024a. find
  5. URL https://www.anthropic.com/news/Anthropic. The Claude 3 model family: Opus, Sonnet, Haiku, 2024b. URL https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_ Card_Claude_3.pdf. ↗
  6. Stephen Casper, Lennart Schulze, Oam Patel, and Dylan Hadfield-Menell. Defending against unforeseen failure modes with latent adversarial training. arXiv preprint, 2024. URL https://arxiv.org/abs/2403.05030. — [arXiv:2403.05030](https://arxiv.org/abs/2403.05030) ↗
  7. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries, 2024. URL https://arxiv.org/abs/2310.08419. ↗
  8. Jianfeng Chi, Ujjwal Karn, Hongyuan Zhan, Eric Smith, Javier Rando, Yiming Zhang, Kate Plawiak, Zacharie Delpierre Coudert, Kartikeya Upasani, and Mahesh Pasupuleti. Llama guard 3 vision: Safeguarding human–AI image understanding conversations. arXiv preprint, 2024. URL https://arxiv.org/abs/2411.1… — [arXiv:2411.10414](https://arxiv.org/abs/2411.10414) ↗
  9. Justin Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui Hsieh. OR-Bench: An over-refusal benchmark for large language models. arXiv preprint, 2024. URL https://arxiv.org/abs/2405.20947. ↗
  10. Thomas J DiCiccio and Bradley Efron. Bootstrap confidence intervals. Statistical science, 1996. URL https://projecteuclid.org/journals/statistical-science/volume-11/issue-3/ Bootstrap-confidence-intervals/10.1214/ss/1032280214.full. — [doi:10.1214/ss/1032280214.full](https://doi.org/10.1214/ss/1032280214.full) ↗
  11. Moussa Koulako Bala Doumbouya, Ananjan Nandi, Gabriel Poesia, Davide Ghilardi, Anna Goldie, Federico Bianchi, Dan Jurafsky, and Christopher D Manning. h4rm3l: A dynamic benchmark of composable jailbreak attacks for LLM safety assessment. arXiv preprint, 2024. URL https://arxiv.org/abs/2408.04811. ↗
  12. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 herd of models. arXiv preprint, 2024. URL https://arxiv.org/abs/2407.21783. ↗
  13. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan… — Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
  14. Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan, and Fabien Roger. AI control: Improving safety despite intentional subversion. arXiv preprint, 2023. URL https://arxiv.org/abs/2312. 06942. ↗
  15. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR), 2020. URL https://arxiv.org/abs/2009.03300. ↗
  16. Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ML safety. arXiv preprint, 2021. URL https://arxiv.org/abs/2109.13916. ↗
  17. John Hughes, Sara Price, Aengus Lynch, Rylan Schaeffer, Fazl Barez, Sanmi Koyejo, Henry Sleight, Erik Jones, Ethan Perez, and Mrinank Sharma. Best-of-N jailbreaking. arXiv preprint, 2024. URL https://arxiv.org/abs/2412.03556. — [arXiv:2412.03556](https://arxiv.org/abs/2412.03556) ↗
  18. Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama guard: LLM-based input-output safeguard for human-AI conversations. arXiv preprint, 2023. URL https://arxiv.org/abs/2312.06674. ↗
  19. Taeyoun Kim, Suhas Kotha, and Aditi Raghunathan. Testing the limits of jailbreaking defenses with the purple problem. arXiv preprint, 2024. URL https://arxiv.org/abs/2403.14725. — [arXiv:2403.14725](https://arxiv.org/abs/2403.14725) ↗
  20. Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew B. Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, Rishub T… find
  21. Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. AutoDAN: Generating stealthy jailbreak prompts on aligned large language models. In International Conference on Learning Representations (ICLR), 2023. URL https://arxiv.org/abs/2310.04451. ↗
  22. Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield-Menell. Eight methods to evaluate robust unlearning in LLMs. arXiv preprint, 2024. URL https://arxiv.org/ abs/2402.16835. — [arXiv:2402.16835](https://arxiv.org/abs/2402.16835) ↗
  23. Todor Markov, Chong Zhang, Sandhini Agarwal, Tyna Eloundou, Teddy Lee, Steven Adler, Angela Jiang, and Lilian Weng. A holistic approach to undesired content detection in the real world. In Conference of the Association for the Advancement of Artificial Intelligence (AAAI), 2023. URL https://arxiv.or… ↗
  24. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. HarmBench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint, 2024. URL https://arxiv.org/abs/24… ↗
  25. Ninareh Mehrabi, Palash Goyal, Christophe Dupuy, Qian Hu, Shalini Ghosh, Richard Zemel, Kai-Wei Chang, Aram Galstyan, and Rahul Gupta. FLIRT: Feedback loop in-context red teaming. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024. URL https://arxiv.org/abs/2308.04265. ↗
  26. OpenAI. OpenAI preparedness framework (beta), 2023. openai-preparedness-framework-beta.pdf. find
  27. URL https://cdn.openai.com/Nedjma Ousidhoum, Xinran Zhao, Tianqing Fang, Yangqiu Song, and Dit-Yan Yeung. Probing toxic content in large pre-trained language models. In Proceedings of the Association for Computational Linguistics (ACL), 2021. URL https://aclanthology.org/2021.acl-long.329/. ↗
  28. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Conference on Neural Information Processing Systems (NeurIPS), 2022. URL https://…
  29. Alwin Peng, Julian Michael, Henry Sleight, Ethan Perez, and Mrinank Sharma. Rapid response: Mitigating LLM jailbreaks with a few examples. arXiv preprint, 2024. URL https://arxiv.org/abs/2411.07494. — [arXiv:2411.07494](https://arxiv.org/abs/2411.07494) ↗
  30. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. arXiv preprint, 2022. URL https://arxiv.org/abs/2202.03286. ↗
  31. Pliny. L1b3rt4s, 2025. URL https://github.com/elder-plinius/L1B3RT4S. ↗
  32. Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Peter Henderson, Mengdi Wang, and Prateek Mittal. Visual adversarial examples jailbreak aligned large language models. In Conference of the Association for the Advancement of Artificial Intelligence (AAAI), 2024. URL https://arxiv.org/abs/2306.13213. ↗
  33. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018. URL https://cdn.openai.com/research-covers/ language-unsupervised/language_understanding_paper.pdf. ↗
  34. James Reason. Human error. Cambridge University Press, 1990. find
  35. Traian Rebedea, Razvan Dinu, Makesh Narsimhan Sreedhar, Christopher Parisien, and Jonathan Cohen. NeMo guardrails: A toolkit for controllable and safe LLM applications with programmable rails. In Conference on Empirical Methods in Natural Language Processing (EMNLP): System Demonstrations, 2023. URL… find
  36. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. GPQA: A graduate-level Google-proof Q&A benchmark. arXiv preprint, 2023. URL https://arxiv.org/abs/2311.12022. ↗
  37. Sophie Rose, Richard Moulange, James Smith, and Cassidy Nelson. The near-term impact of AI on biological misuse. The Centre for Long-Term Resilience, 2024. URL https://www.longtermresilience.org/wp-content/uploads/2024/07/ CLTR-Report-The-near-term-impact-of-AI-on-biological-misuse-July-2024-1.pdf. ↗
  38. Mikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Hambro, Aram H. Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Foerster, Tim Rocktäschel, and Roberta Raileanu. Rainbow teaming: Open-ended generation of diverse adversarial prompts. In Conference on Neural I… — [arXiv:2402.16822](https://arxiv.org/abs/2402.16822) find
  39. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. “Do anything now”: Characterizing and evaluating in-the-wild jailbreak prompts on large language models. arXiv preprint, 2023. URL https://arxiv.org/abs/2308.03825. ↗
  40. Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. arXiv preprint, 2023. URL https://arxiv.org/abs/2310.16789. ↗
  41. Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, and Sam Toyer. A StrongREJECT for empty jailbreaks. arXiv preprint, 2024. URL https://arxiv.org/abs/2402.10260. ↗
  42. Tony T. Wang, John Hughes, Henry Sleight, Rylan Schaeffer, Rajashree Agrawal, Fazl Barez, Mrinank Sharma, Jesse Mu, Nir Shavit, and Ethan Perez. Jailbreak defense in a narrow domain: Limitations of existing methods and a new transcript-classifier approach, 2024. URL https://arxiv.org/abs/2412.02159… — [arXiv:2412.02159](https://arxiv.org/abs/2412.02159) ↗
  43. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Conference on Neural Information Processing Systems (NeurIPS), 2022. URL https://arxiv.org/abs/2201.11903. ↗
  44. Wojciech Zaremba, Evgenia Nitishinskaya, Boaz Barak, Stephanie Lin, Sam Toyer, Yaodong Yu, Rachel Dias, Eric Wallace, Kai Xiao, Johannes Heidecke, and Amelia Glaese. Trading inference-time compute for adversarial robustness, 2024. URL https://openai.com/research/ trading-inference-time-compute-for-a… ↗
  45. Zhexin Zhang, Junxiao Yang, Pei Ke, Shiyao Cui, Chujie Zheng, Hongning Wang, and Minlie Huang. Safe unlearning: A surprisingly effective and generalizable solution to defend against jailbreak attacks. arXiv preprint, 2024. URL https://arxiv.org/abs/2407.02855. — [arXiv:2407.02855](https://arxiv.org/abs/2407.02855) ↗
  46. Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: 1M ChatGPT interaction logs in the wild. In International Conference on Learning Representations (ICLR), 2024. URL https://arxiv.org/abs/2405.01470. ↗
  47. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint, 2023. URL https://arxiv.org/abs/2307. 15043. Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolt… ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sharma-2025-constitutional-classifiers-defending-against-universal/