Elamur Research
☰
home / papers / 2023 / Jailbroken: How Does LLM Safety Training Fail?

Jailbroken: How Does LLM Safety Training Fail?

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2307.02483

Summary

The paper asks why chat models that have gone through safety training and red-teaming can still be pushed into producing content they were taught to refuse, and how such jailbreaks can be built on purpose. The authors propose two explanations. In competing objectives, the model's drive to follow instructions or continue text plausibly conflicts with its refusal behaviour. In mismatched generalization, an input falls outside what safety training covered but is still something the model can handle thanks to pretraining, such as Base64-encoded text. Using these two ideas, they design many black-box attacks. They test them, along with jailbreaks circulating online, on GPT-4, Claude v1.3 and GPT-3.5 Turbo. Combined attacks get on-topic harmful answers for more than 96% of prompts, and for every prompt in the curated red-teaming set. The authors conclude that more scale alone will not close these gaps. They argue for safety-capability parity, meaning safeguards need to be as capable as the model they protect.

Key points

  • Evaluation data: 32 harmful prompts adapted from OpenAI and Anthropic red-teaming material, plus a held-out set of 317 prompts that the authors did not see until data collection was finished.
  • Scoring: each response is labelled GOOD BOT (refusal), BAD BOT (on-topic answer to the harmful request) or UNCLEAR. Decoding used temperature 0, and data for each model was gathered within a 6-day window.
  • Scale and capabilities: GPT-4 can follow Base64 instructions but with weaker safeguards, while GPT-3.5 Turbo cannot follow them at all. The authors read this as evidence that new capabilities widen the attack surface.
  • LLM-assisted attacks: some attacks were partly automated with LLMs.
    • auto_payload_splitting used GPT-4 to flag sensitive words.
    • With auto_obfuscation, Claude rediscovered Base64 and GPT-4 came up with Spanish leetspeak.
  • Responsible disclosure: results were shared with OpenAI and Anthropic before release, and the exact prompts for the strongest attacks were withheld.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

443 papers of our library cite it (of the 140673 whose bibliographies we read); 30 of them have a page here, 413 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2307.02483 · arXiv 2307.02483

@misc{wei2023jailbroken,
  title = {{Jailbroken: How Does LLM Safety Training Fail?}},
  author = {Wei, Alexander and Haghtalab, Nika and Steinhardt, Jacob},
  year = {2023},
  doi = {10.48550/arxiv.2307.02483},
  eprint = {2307.02483},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.02483}
}

References

58 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adversa. Universal LLM jailbreak: ChatGPT, GPT-4, Bard, Bing, Anthropic, and beyond. Adversa Blog, 2023. URL https://adversa.ai/blog/universal-llm-jailbreak-chatg pt-gpt-4-bard-bing-anthropic-and-beyond/. ↗
  2. Alex Albert. Jailbreak Chat. https://www.jailbreakchat.com/, 2023. ↗
  3. Alex Albert. Jailbreak Chat. https://web.archive.org/web/20230413032954/https: //www.jailbreakchat.com/, 2023. ↗
  4. Anthropic. Anthropic API reference. https://console.anthropic.com/docs/api/refer ence, 2023. ↗
  5. Anthropic. “We are offering a new version of our model, Claude-v1.3, that is safer and less susceptible to adversarial attacks.”. https://twitter.com/AnthropicAI/status/1648353 600350060545, 2023. ↗
  6. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. find
  7. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08073, 2022. find our card
  8. Boaz Barak. “Another jailbreak for GPT4: Talk to it in Morse code”. https://twitter.com/boazbaraktcs/status/1637657623100096513, 2023. ↗
  9. Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021. find our card
  10. Greg Brockman. “Deploying GPT-4 subject to adversarial pressures of real world has been a great practice run for practical AI alignment. Just getting started, but encouraged by degree of alignment we’ve achieved so far (and the engineering process we’ve been maturing to improve issues).”. https://tw… find
  11. Greg Brockman, Atty Eleti, Elie Georges, Joanne Jang, Logan Kilpatrick, Rachel Lim, Luke Miller, and Michelle Pokrass. Introducing ChatGPT and Whisper APIs. OpenAI Blog, 2023. URL https://openai.com/blog/introducing-chatgpt-and-whisper-apis. ↗
  12. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877–1901, 2020. find
  13. Matt Burgess. The hacking of ChatGPT is just getting started. Wired, 2023. find
  14. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In USENIX Security Symposium, volume 6, 2021. find our card
  15. Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopadhyay. Adversarial attacks and defences: A survey. arXiv preprint arXiv:1810.00069, 2018. — [arXiv:1810.00069](https://arxiv.org/abs/1810.00069) find
  16. W.C. Christensen and F.A. Manuele. Safety Through Design. American Society of Mechanical Engineers, 1999. find
  17. Jon Christian. Amazing “jailbreak” bypasses ChatGPT’s ethics safeguards. Futurism, 2023. find
  18. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. find
  19. Cleo Nardo. The Waluigi effect (mega-post). https://www.lesswrong.com/posts/D7Pume YTDPfBTp3i7/the-waluigi-effect-mega-post, 2023. ↗
  20. El-Mahdi El-Mhamdi, Sadegh Farhadkhani, Rachid Guerraoui, Nirupam Gupta, Lê-Nguyên Hoang, Rafael Pinot, Sébastien Rouault, and John Stephan. On the impossible safety of large AI models. arXiv preprint arXiv:2209.15259, 2022. — [arXiv:2209.15259](https://arxiv.org/abs/2209.15259) find
  21. Dan Elton. “(humble brag) I’ve had alpha access to Anthropic’s competitor to chatGPT the past 2 weeks. The media embargo was just lifted an hour ago. I’ll share some comparisons w chatGPT in thread. This summary I’m QT’ing aligns w/ my experience. See also screenshot of doc from Anthropic...”. https… find
  22. Colin Fraser. “Master thread of ways I have discovered to get ChatGPT to output text that it’s not supposed to, including bigotry, URLs and personal information, and more.”. https: //twitter.com/colin\_fraser/status/1630763219450212355, 2023. find
  23. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858, 2022. find our card
  24. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. Real-ToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, 2020. find
  25. Josh A Goldstein, Girish Sastry, Micah Musser, Renee DiResta, Matthew Gentzel, and Katerina Sedova. Generative language models and automated influence operations: Emerging threats and potential mitigations. arXiv preprint arXiv:2301.04246, 2023. — [arXiv:2301.04246](https://arxiv.org/abs/2301.04246) find
  26. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. More than you’ve asked for: A comprehensive analysis of novel prompt injection threats to application-integrated large language models. arXiv preprint arXiv:2302.12173, 2023. find our card
  27. Alexey Guzey. A two sentence jailbreak for GPT-4 and Claude & why nobody knows how to fix it. https://guzey.com/ai/two-sentence-universal-jailbreak/, 2023. ↗
  28. Julian Hazell. Large language models can be used to effectively scale spear phishing campaigns. arXiv preprint arXiv:2305.06972, 2023. — [arXiv:2305.06972](https://arxiv.org/abs/2305.06972) find
  29. Erik Jones, Anca Dragan, Aditi Raghunathan, and Jacob Steinhardt. Automatically auditing large language models via discrete optimization. arXiv preprint arXiv:2303.04381, 2023. find
  30. Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto. Exploiting programmatic behavior of LLMs: Dual-use through standard security attacks. arXiv preprint arXiv:2302.05733, 2023. — [arXiv:2302.05733](https://arxiv.org/abs/2302.05733) find
  31. Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. arXiv preprint arXiv:2302.08582, 2023. find
  32. Sarah Kreps, R. Miles McCain, and Miles Brundage. All the news that’s fit to fabricate: Ai-generated text as a tool of media misinformation. Journal of Experimental Political Science, 9 (1):104–117, 2022. find
  33. Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, and Yangqiu Song. Multi-step jailbreaking privacy attacks on ChatGPT. arXiv preprint arXiv:2304.05197, 2023. find our card
  34. Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella-Béguelin. Analyzing leakage of personally identifiable information in language models. arXiv preprint arXiv:2302.00539, 2023. find
  35. Zvi Mowshowitz. Jailbreaking ChatGPT on release day. https://www.lesswrong.com/post s/RYcoJdvmoBbi5Nax7/jailbreaking-chatgpt-on-release-day, 2023. ↗
  36. Neel Nanda. Mechanistic interpretability quickstart guide. https://www.neelnanda.io/mec hanistic-interpretability/quickstart, 2023. ↗
  37. Nin\_kat. “New jailbreak based on virtual functions smuggle”. https://old.reddit.com/r /ChatGPT/comments/10urbdj/new\_jailbreak\_based\_on\_virtual\_functions\_smuggle/, 2023. ↗
  38. OpenAI. GPT-4 technical report. arXiv preprint 2303.08774, 2023. find our card
  39. OpenAI. Models. OpenAI API Documentation, 2023. URL https://platform.openai.co m/docs/models/. ↗
  40. OpenAI. Our approach to AI safety. https://openai.com/blog/our-approach-to-ai-s afety, 2023. ↗
  41. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35, 2022. find
  42. Ethan Perez, Saffron Huang, H. Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3419–3448, 2022. find our card
  43. Roman Semenov. “The new jailbreak is so fun”. https://twitter.com/semenov\_roman\_/s tatus/1621465137025613825, 2023. ↗
  44. Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang. On second thought, let’s not think step by step! Bias and toxicity in zero-shot reasoning. arXiv preprint arXiv:2212.08061, 2022. find our card
  45. Irene Solaiman and Christy Dennison. Process for adapting language models to society (PALMS) with values-targeted datasets. Advances in Neural Information Processing Systems, 34:5861– 5873, 2021. find
  46. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020. find
  47. Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. arXiv preprint arXiv:2305.03047, 2023. find our card
  48. walkerspider. DAN is my new friend. https://old.reddit.com/r/ChatGPT/comments/zl cyr9/dan\_is\_my\_new\_friend/, 2022. ↗
  49. Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for attacking and analyzing NLP. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing,… find
  50. Boxin Wang, Wei Ping, Chaowei Xiao, Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Bo Li, Anima Anandkumar, and Bryan Catanzaro. Exploring the limits of domain-adaptive training for detoxifying large-scale language models. In Advances in Neural Information Processing Systems, 2022. find
  51. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022. find our card
  52. Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, and Po-Sen Huang. Challenges in detoxifying language models. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2447–2469, 20… find
  53. WitchBOT. You can use GPT-4 to create prompt injections against GPT-4. https://www.less wrong.com/posts/bNCDexejSZpkuu3yz/you-can-use-gpt-4-to-create-prompt-injec tions-against-gpt-4, 2023. ↗
  54. Zack Witten. “Thread of known ChatGPT jailbreaks”. https://twitter.com/zswitten/sta tus/1598380220943593472, 2022. ↗
  55. Yotam Wolf, Noam Wies, Yoav Levine, and Amnon Shashua. Fundamental limitations of alignment in large language models. arXiv preprint arXiv:2304.11082, 2023. find
  56. Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. Recipes for safety in open-domain chatbots. arXiv preprint arXiv:2010.07079, 2020. find
  57. Wei Emma Zhang, Quan Z Sheng, Ahoud Alhazmi, and Chenliang Li. Adversarial attacks on deep-learning models in natural language processing: A survey. ACM Transactions on Intelligent Systems and Technology (TIST), 11(3):1–41, 2020. find
  58. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. Jailbroken: How Does LLM Safety Training Fail? find