Elamur Research⌕
☰
home / papers / 2024 / Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack

Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack

Paper · 2024 · Mark Russinovich, Ahmed Salem and Ronen Eldan

Summary

Safety-aligned LLMs are usually tested against single-turn jailbreaks. Those attacks tend to contain recognizable malicious text that input filters can block once the attack becomes known. The authors present Crescendo, a black-box attack that spreads the jailbreak across a conversation. It opens with a harmless, general question related to the target task. Each later turn builds on the model's own previous answers, so the dialogue escalates step by step without any explicitly harmful prompt. The authors compare this to the "foot-in-the-door" persuasion effect.

They first ran Crescendo manually against major open and closed chat models. They then built Crescendomation, an LLM-driven tool that automates the attack using feedback on refusals and output quality. On the AdvBench subset, Crescendomation clearly beat existing jailbreak methods.

The work matters because it exposes a gap in current alignment and benchmarking practice, which focus on single-turn attacks. The authors propose their tool for multi-turn red-teaming and for generating alignment data.

Key points

  • Models tested: ChatGPT, GPT-3.5 Turbo, GPT-4, Gemini-Pro, Gemini-Ultra, Claude-2, Claude-3, LLaMA-2 70b and LLaMA-3 70b. Manual Crescendo bypassed safety alignment on nearly all tasks, and jailbroken multimodal systems could also be pushed into generating images.
  • Benchmark results: On the AdvBench subset, Crescendomation outperformed MSJ, CIA, PAIR and CoA by 29–61% on GPT-4 and by 49–71% on Gemini-Pro.
  • Evaluation pipeline: Outputs were scored in several ways:
    • a GPT-4 "Judge";
    • a GPT-4 "Secondary Judge" that reviews the first verdict to reduce false negatives;
    • manual inspection of top results;
    • external moderation services, namely Google's Perspective API and the content filters offered in Microsoft's Azure.
  • Defenses: The attack was also tested against Self-Reminder and Goal Prioritization. Suggested mitigations include alignment training on Crescendo-style data and filtering both inputs and outputs.
  • Release and disclosure: Crescendomation is open-sourced in PyRIT (github.com/Azure/PyRIT). Before publication, the findings were disclosed to OpenAI, Microsoft, Meta, Google and Anthropic with a three-month embargo.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States378 works · 161,630 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksBias, fairness & safety · 587 works

Cite

Cite as Mark Russinovich, Ahmed Salem and Ronen Eldan. 2024. Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack. arXiv:2404.01833.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2404.01833 · arXiv 2404.01833

@misc{russinovich2024great,
  title = {{Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack}},
  author = {Mark Russinovich, Ahmed Salem and Ronen Eldan},
  year = {2024},
  doi = {10.48550/arxiv.2404.01833},
  eprint = {2404.01833},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.01833}
}

References

25 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. [12] Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu. Jailbreaker: Automated Jailbreak Across Multiple Large Language Model Chatbots. CoRR abs/2307.08715, 2023. — [arXiv:2307.08715](https://arxiv.org/abs/2307.08715) find
  2. [1] — Multilingual Jailbreak Challenges in Large Language Models https://chat.openai.com/share/ 31708d66-c735-46e4-94fd-41f436d4d3e9. [2] https://gemini.google.com/share/ 35f0817c3a03. [13] Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. Multilingual Jailbreak Challenges in Large Language Models. CoRR abs/2310.06474, 2023. ↗
  3. [23] Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella-Béguelin. Analyzing leakage of personally identifiable information in language models, 2023.
  4. [14] Suyu Ge, Chunting Zhou, Rui Hou, Madian Khabsa, Yi-Chia Wang, Qifan Wang, Jiawei Han, and Yuning Mao. Mart: Improving llm safety with multi-round automatic red-teaming, 2023.
  5. [24] Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal, 2024.
  6. [15] Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Campbell-Gillingham, Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig, Charlie Chen, Do… — Improving alignment of dialogue agents via targeted human judgements
  7. [25] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Traini… — Training language models to follow instructions with human feedback
  8. [26] Ahmed Salem, Andrew Paverd, and Boris Köpf. Maatphor: Automated variant analysis for prompt injection attacks, 2023. find
  9. [16] Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation. CoRR abs/2310.06987, 2023.
  10. [27] Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. Do Anything Now: Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models. CoRR abs/2308.03825, 2023.
  11. [17] Nikhil Kandpal, Krishna Pillutla, Alina Oprea, Peter Kairouz, Christopher A. Choquette-Choo, and Zheng Xu. User inference attacks on large language models, 2023. find
  12. [28] Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How Does LLM Safety Training Fail? CoRR abs/2307.02483, 2023.
  13. [18] Tadayoshi Kohno, Yasemin Acar, and Wulf Loh. Ethical frameworks and computer security trolley problems: Foundations for conversations. In 32nd USENIX Security Symposium (USENIX Security 23), pages 5145– 5162, Anaheim, CA, August 2023. USENIX Association. find
  14. [29] Rui Wen, Tianhao Wang, Michael Backes, Yang Zhang, and Ahmed Salem. Last one standing: A comparative analysis of security and privacy of soft prompt tuning, lora, and in-context learning, 2023. find
  15. [30] Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. Defending chatgpt against jailbreak attack via self-reminders. Nature Machine Intelligence, 5:1486–1496, 2023. find
  16. [19] Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L. Buckley, Jason Phang, Samuel R. Bowman, and Ethan Perez. Pretraining language models with human preferences, 2023.
  17. [20] Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, Jie Huang, Fanpu Meng, and Yangqiu Song. Multi-step jailbreaking privacy attacks on chatgpt, 2023.
  18. [31] Liang Xu, Kangkang Zhao, Lei Zhu, and Hang Xue. Sc-safety: A multi-round open-ended question adversarial safety benchmark for large language models in chinese, 2023. find
  19. [21] Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models. CoRR abs/2310.04451, 2023.
  20. [32] Xikang Yang, Xuehai Tang, Songlin Hu, and Jizhong Han. Chain of attack: a semantic-driven contextual multi-turn attacker for llm, 2024. find
  21. [22] Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study. CoRR abs/2305.13860, 2023.
  22. [33] Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts. CoRR abs/2309.10253, 2023.
  23. [34] Zhexin Zhang, Junxiao Yang, Pei Ke, Fei Mi, Hongning Wang, and Minlie Huang. Defending large language models against jailbreaking attacks through goal prioritization, 2024.
  24. [35] Zhuo Zhang, Guangyu Shen, Guanhong Tao, Siyuan Cheng, and Xiangyu Zhang. Make them spill the beans! coercive knowledge extraction from (production) llms, 2023. find
  25. [36] Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and Transferable Adversarial Attacks on Aligned Language Models. CoRR abs/2307.15043, 2023.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/russinovich-2024-great-now-write-article-about/