Elamur Research⌕
☰
home / papers / 2024 / Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks

Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks

Paper · 2024 · Maksym Andriushchenko, Francesco Croce and Nicolas Flammarion

Summary

The paper asks whether today's safety-aligned LLMs can withstand jailbreaks that are tailored to each target model rather than applied uniformly. The authors build "adaptive" attacks from a few simple components. The first is a hand-crafted prompt template, fixed per model and not per request. The second, used where logprobs are at least partly exposed, is a random search over an adversarial suffix that pushes up the probability of an affirmative first token such as "Sure". Some models also get restarts and self-transfer of suffixes found on easier requests. Where the API allows it, the attacks use interface-specific weaknesses, for example response prefilling on Claude. For evaluation, the authors rely on a set of 50 harmful behaviors that Chao et al. selected out of AdvBench, and an attempt counts as successful only if a GPT-4 judge assigns it 10/10. With this setup the attacks reach a 100% success rate on every open and proprietary model tested, including the adversarially trained R2D2. The main lesson is that robustness claims need model-specific adaptive attacks, because no single attack works across all targets.

Key points

  • Models broken at 100% ASR (GPT-4 judge):
    • Open and adversarially trained models: Vicuna-13B, Mistral-7B, Phi-3-Mini, Nemotron-4-340B, Llama-2-Chat-7B/13B/70B, Llama-3-Instruct-8B, Gemma-7B and R2D2-7B (HarmBench).
    • Proprietary models: GPT-3.5 Turbo, GPT-4o and all Claude models, including Claude 3 Opus and Claude 3.5 Sonnet.
  • Effect of random search: The template alone often does little. On Llama-2-Chat-7B and Llama-3-Instruct-8B it scores 0%, but adding random search and self-transfer lifts both to 100%. Earlier attacks such as PAIR, TAP and GCG scored between 4% and 54% on Llama-2-Chat-7B.
  • Model-specific weak spots:
    • R2D2 gives way most easily to in-context-learning prompts.
    • Claude, which hides logprobs, is jailbroken through prefilling or through transfer from GPT-4.
    • On Claude 2.1 the GPT-4 judge produces roughly 20% false positives.
  • Trojan detection: Random search restricted to a limited token set won first place in the SaTML'24 Trojan Detection Competition.
  • Released resources: Code, logs and jailbreak artifacts are published in JailbreakBench format at github.com/tml-epfl/llm-adaptive-attacks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓École Polytechnique Fédérale de LausanneLausanne · Switzerland48 works · 7,352 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksBias, fairness & safety · 587 works

Cite

Cite as Maksym Andriushchenko, Francesco Croce and Nicolas Flammarion. 2024. Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks. arXiv:2404.02151.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2404.02151 · arXiv 2404.02151

@misc{andriushchenko2024jailbreaking,
  title = {{Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks}},
  author = {Maksym Andriushchenko, Francesco Croce and Nicolas Flammarion},
  year = {2024},
  doi = {10.48550/arxiv.2404.02151},
  eprint = {2404.02151},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.02151}
}

References

62 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024.
  2. AI@Meta. Llama 3 model card. GitHub, 2024. URL https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md. ↗
  3. Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: a query-efficient black-box adversarial attack via random search. In ECCV, 2020. find
  4. Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, Zico Kolter, Matt Fredrikson, et al. Agentharm: A benchmark for measuring harmfulness of llm agents. ICLR, 2025. find
  5. Cem Anil, Esin Durmus, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Nina Rimsky, Meg Tong, Jesse Mu, Daniel Ford, et al. Many-shot jailbreaking. Anthropic, April, 2024. find
  6. Anthropic. Prefill claude’s response for greater output control, 2024a. URL https://docs.ant hropic.com/claude/docs/prefill-claudes-response. Accessed: 2024-09-11. ↗
  7. Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024b. find
  8. Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. ICML, 2018. find
  9. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. find
  10. Battista Biggio and Fabio Roli. Wild patterns: ten years after the rise of adversarial machine learning. Pattern Recognition, 2018. find
  11. Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2013, Prague, Czech Republic,… find
  12. Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. arXiv preprint arXiv:1902.06705, 2019. — [arXiv:1902.06705](https://arxiv.org/abs/1902.06705) find
  13. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419, 2023.
  14. Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramèr, Hamed Hassani, and Eric Wong. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. In Neur…
  15. Yixin Cheng, Markos Georgopoulos, Volkan Cevher, and Grigorios G Chrysos. Leveraging the context through multi-round interactions for jailbreaking attacks. arXiv preprint arXiv:2402.09177, 2024. — [arXiv:2402.09177](https://arxiv.org/abs/2402.09177) find
  16. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna… ↗
  17. Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: a standardized adversarial robustness benchmark. In NeurIPS Datasets and Benchmarks Track, 2021. find
  18. Francesco Croce, Maksym Andriushchenko, Naman D Singh, Nicolas Flammarion, and Matthias Hein. Sparse-rs: a versatile framework for query-efficient sparse black-box adversarial attacks. In AAAI, 2022. find
  19. Simon Geisler, Tom Wollschläger, Mohamed Hesham Ibrahim Abdalla, Johannes Gasteiger, and Stephan Günnemann. Attacking large language models with projected gradient descent. arXiv preprint arXiv:2402.09154, 2024. — [arXiv:2402.09154](https://arxiv.org/abs/2402.09154) find
  20. Gemini Team Google. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
  21. Jonathan Hayase, Ema Borevkovic, Nicholas Carlini, Florian Tramèr, and Milad Nasr. Query-based adversarial prompt generation. arXiv preprint arXiv:2402.12329, 2024. — [arXiv:2402.12329](https://arxiv.org/abs/2402.12329) find
  22. Brian RY Huang, Maximilian Li, and Leonard Tang. Endless jailbreaks with bijection learning. arXiv preprint arXiv:2410.01294, 2024. — [arXiv:2410.01294](https://arxiv.org/abs/2410.01294) find
  23. Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama guard: Llm-based input-output safeguard for human-ai conversations, 2023.
  24. Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. Baseline defenses for adversarial attacks against aligned language models. arXiv preprint arXiv:2309.00614, 2023.
  25. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. find
  26. Priyanshu Kumar, Elaine Lau, Saranya Vijayakumar, Tu Trinh, Scale Red Team, Elaine Chang, Vaughn Robinson, Sean Hendryx, Shuyan Zhou, Matt Fredrikson, et al. Refusal-trained llms are easily jailbroken as browser agents. ICLR, 2025. find
  27. Raz Lapid, Ron Langberg, and Moshe Sipper. Open sesame! universal black box jailbreaking of large language models. arXiv preprint arXiv:2309.01446, 2023. — [arXiv:2309.01446](https://arxiv.org/abs/2309.01446) find
  28. Zeyi Liao and Huan Sun. Amplegcg: Learning a universal and transferable generative model of adversarial suffixes for jailbreaking both open and closed llms. arXiv preprint arXiv:2404.07921, 2024. — [arXiv:2404.07921](https://arxiv.org/abs/2404.07921) find
  29. Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451, 2023.
  30. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. ICLR, 2018.
  31. Neal Mangaokar, Ashish Hooda, Jihye Choi, Shreyas Chandrashekaran, Kassem Fawaz, Somesh Jha, and Atul Prakash. Prp: Propagating universal perturbations to attack large language model guard-rails. arXiv preprint arXiv:2402.15911, 2024. find
  32. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. In ICML, 2024.
  33. Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. Tree of attacks: Jailbreaking black-box llms automatically. arXiv preprint arXiv:2312.02119, 2023.
  34. Zvi Mowshowitz. Jailbreaking chatgpt on release day. https://www.lesswrong.com/po sts/RYcoJdvmoBbi5Nax7/jailbreaking-chatgpt-on-release-day, 2022. Accessed: 2024-02-25. ↗
  35. Nvidia team. Nemotron-4 340b technical report. Technical Report, 2024. URL https://d1qx 31qr3h6wln.cloudfront.net/publications/Nemotron_4_340B_8T_0.pdf.
  36. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730–27744, 2022.
  37. Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693, 2023.
  38. Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. Safety alignment should be made more than just a few tokens deep. ICLR, 2025.
  39. Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. Certified defenses against adversarial examples. ICLR, 2018. find
  40. Javier Rando and Florian Tramèr. Universal jailbreak backdoors from poisoned human feedback. arXiv preprint arXiv:2311.14455, 2023. — [arXiv:2311.14455](https://arxiv.org/abs/2311.14455) find
  41. Javier Rando and Florian Tramèr. Find the trojan: Universal backdoor detection in aligned llms. https://github.com/ethz-spylab/rlhf_trojan_competition, 2024. ↗
  42. Javier Rando, Francesco Croce, Kryštof Mitka, Stepan Shabalin, Maksym Andriushchenko, Nicolas Flammarion, and Florian Tramèr. Competition report: Finding universal jailbreak backdoors in aligned llms. arXiv preprint arXiv:2404.14461, 2024. — [arXiv:2404.14461](https://arxiv.org/abs/2404.14461) find
  43. Leonard Rastrigin. The convergence of the random search method in the extremal control of a many parameter system. Automaton & Remote Control, 24:1337–1342, 1963. find
  44. Alexander Robey, Eric Wong, Hamed Hassani, and George J Pappas. Smoothllm: Defending large language models against jailbreaking attacks. arXiv preprint arXiv:2310.03684, 1(10), 2023.
  45. Mark Russinovich, Ahmed Salem, and Ronen Eldan. Great, now write an article about that: The crescendo multi-turn llm jailbreak attack. arXiv preprint arXiv:2404.01833, 2024.
  46. Rusheb Shah, Soroush Pour, Arush Tagade, Stephen Casper, Javier Rando, et al. Scalable and transferable black-box jailbreaks for language models via persona modulation. arXiv preprint arXiv:2311.03348, 2023. — [arXiv:2311.03348](https://arxiv.org/abs/2311.03348) find
  47. Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980, 2020.
  48. Chawin Sitawarin, Norman Mu, David Wagner, and Alexandre Araujo. Pal: Proxy-guided black-box attack on large language models. arXiv preprint arXiv:2402.09674, 2024. — [arXiv:2402.09674](https://arxiv.org/abs/2402.09674) find
  49. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. ICLR, 2014.
  50. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  51. Florian Tramèr, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. In NeurIPS, 2020. find
  52. Jason Vega, Isha Chaudhary, Changming Xu, and Gagandeep Singh. Bypassing the safety training of open-source llms with priming attacks. arXiv preprint arXiv:2312.12321, 2023. — [arXiv:2312.12321](https://arxiv.org/abs/2312.12321) find
  53. Bertie Vidgen, Hannah Rose Kirk, Rebecca Qian, Nino Scherrer, Anand Kannappan, Scott A Hale, and Paul Röttger. Simplesafetytests: a test suite for identifying critical safety risks in large language models. arXiv preprint arXiv:2311.08370, 2023.
  54. Zhenhua Wang, Wei Xie, Baosheng Wang, Enze Wang, Zhiwen Gui, Shuoyoucheng Ma, and Kai Chen. Foot in the door: Understanding large language model jailbreaking via cognitive psychology. arXiv preprint arXiv:2402.15690, 2024. — [arXiv:2402.15690](https://arxiv.org/abs/2402.15690) find
  55. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail? NeurIPS, 2023a.
  56. Zeming Wei, Yifei Wang, and Yisen Wang. Jailbreak and guard aligned language models with only few in-context demonstrations. arXiv preprint arXiv:2310.06387, 2023b.
  57. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019.
  58. Jiahao Yu, Xingwei Lin, and Xinyu Xing. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts. arXiv preprint arXiv:2309.10253, 2023.
  59. Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms. arXiv preprint arXiv:2401.06373, 2024.
  60. Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. Autodan: Automatic and interpretable adversarial attacks on large language models. arXiv preprint arXiv:2310.15140, 2023.
  61. Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023.
  62. Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, J Zico Kolter, Matt Fredrikson, and Dan Hendrycks. Improving alignment and robustness with circuit breakers. In NeurIPS, 2024.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/andriushchenko-2024-jailbreaking-leading-safety-aligned-llms/