Elamur Research
☰
home / papers / 2023 / AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models

AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2310.04451

Summary

Jailbreak attacks push safety-aligned LLMs into answering requests they should refuse. Earlier approaches fall into two camps. Hand-written prompts, such as the "Do-Anything-Now (DAN)" family, scale poorly and lose effectiveness as models are updated. Gradient-guided token search, such as GCG, produces gibberish suffixes that simple perplexity filters catch easily. The authors introduce AutoDAN, which automatically builds fluent, human-readable jailbreak prompts. It frames prompt construction as discrete optimization and solves it with a hierarchical genetic algorithm. The initial population is seeded from a handcrafted DAN-style prototype that an LLM rewrites into diverse variants. Candidates are then ranked by a fitness score and evolved through paragraph-level crossover, sentence- and word-level operations guided by momentum-based word scoring, and occasional LLM-driven mutation. On AdvBench Behaviour requests against models such as Llama2 and Vicuna, AutoDAN beats GCG in attack strength, cross-model transfer and cross-sample universality, and it evades perplexity-based detection. For red-teaming, the takeaway is that defenses which only flag unnatural text cannot stop semantically coherent attacks.

Key points

  • Two variants are compared: AutoDAN-GA and AutoDAN-HGA (the hierarchical version). The authors report roughly 60% higher attack strength than the GCG baseline while remaining immune to the perplexity defense. Code is available at github.com/SheltonLiu-N/AutoDAN.
  • Transfer to commercial APIs: on GPT-3.5-turbo-0301 (prompts optimized on Vicuna), AutoDAN-HGA reaches a keyword ASR of 0.7077 versus 0.0730 for GCG. On GPT-4-0613 both methods stay near zero (0.0077 vs 0.0004).
  • Against paraphrasing with GPT-3.5-turbo as the paraphraser, protecting Vicuna-7b, GCG falls to 0.06 ASR while AutoDAN-HGA keeps 0.68. Adversarial training barely affects either attack (0.94 vs 0.93).
  • LLM-based mutation is applied with probability 0.01 per sample. It improves attack effectiveness by about 35% and slightly shortens runtime (722.6s vs 766.6s).
  • LLM-based diverse initialization outperforms WordNet synonym replacement on Llama2, with keyword ASR of 0.51 versus 0.33.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

305 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 296 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.04451 · arXiv 2310.04451

@misc{liu2023autodan,
  title = {{AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models}},
  author = {Liu, Xiaogeng and Xu, Nan and Chen, Muhao and Xiao, Chaowei},
  year = {2023},
  doi = {10.48550/arxiv.2310.04451},
  eprint = {2310.04451},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.04451}
}

References

38 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alex Albert. https://www.jailbreakchat.com/, 2023. Accessed: 2023-09-28. ↗
  2. Gabriel Alon and Michael Kamfonas. Detecting language model attacks with perplexity, 2023. find our card
  3. Dogu Araci. Finbert: Financial sentiment analysis with pre-trained language models. arXiv preprint arXiv:1908.10063, 2019. — [arXiv:1908.10063](https://arxiv.org/abs/1908.10063) find
  4. Stephen H. Bach, Victor Sanh, Zheng-Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-David, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Alan Fries, Maged S. Al-shaib… find our card
  5. Matt Burgess. The hacking of chatgpt is just getting started. Wired, 2023. find
  6. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https: //lmsys.org/blog/2023-03-30-vicuna… find
  7. Jon Christian. Amazing “jailbreak” bypasses chatgpt’s ethics safeguards. Futurism, February, 4: 2023, 2023. find
  8. Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu. Jailbreaker: Automated jailbreak across multiple large language model chatbots, 2023. find
  9. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314, 2023. find
  10. Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. In International Conference on Machine Learning, pp. 5988–6008. PMLR, 2022. find
  11. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858, 2022. find our card
  12. Josh A Goldstein, Girish Sastry, Micah Musser, Renee DiResta, Matthew Gentzel, and Katerina Sedova. Generative language models and automated influence operations: Emerging threats and potential mitigations. arXiv preprint arXiv:2301.04246, 2023. — [arXiv:2301.04246](https://arxiv.org/abs/2301.04246) find
  13. Alex Havrilla. https://huggingface.co/datasets/Dahoas/synthetic-instruct-gptj-pairwise, 2023. Accessed: 2023-09-28. ↗
  14. Julian Hazell. Large language models can be used to effectively scale spear phishing campaigns. arXiv preprint arXiv:2305.06972, 2023. — [arXiv:2305.06972](https://arxiv.org/abs/2305.06972) find
  15. Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. Baseline defenses for adversarial attacks against aligned language models, 2023. find our card
  16. Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto. Exploiting programmatic behavior of llms: Dual-use through standard security attacks. arXiv preprint arXiv:2302.05733, 2023. — [arXiv:2302.05733](https://arxiv.org/abs/2302.05733) find
  17. Raz Lapid, Ron Langberg, and Moshe Sipper. Open sesame! universal black box jailbreaking of large language models, 2023. find
  18. Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, Jie Huang, Fanpu Meng, and Yangqiu Song. Multi-step jailbreaking privacy attacks on chatgpt, 2023. find our card
  19. Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. Jailbreaking chatgpt via prompt engineering: An empirical study, 2023. find our card
  20. Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics, 23(6), 2022. find
  21. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal, 2024. find our card
  22. John X Morris, Eli Lifland, Jack Lanchantin, Yangfeng Ji, and Yanjun Qi. Reevaluating adversarial examples in natural language. arXiv preprint arXiv:2004.14174, 2020. find
  23. AJ ONeal. https://gist.github.com/coolaj86/6f4f7b30129b0251f61fa7baaa881516, 2023. Accessed: 2023-09-28. ↗
  24. OpenAI. Snapshot of gpt-3.5-turbo from march 1st 2023. https://openai.com/blog/chatgpt, 2023a. Accessed: 2023-08-30. ↗
  25. OpenAI. Gpt-4 technical report, 2023b. find our card
  26. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730–27744, 2022. find
  27. Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint arXiv:1605.07277, 2016. — [arXiv:1605.07277](https://arxiv.org/abs/1605.07277) find
  28. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. arXiv preprint arXiv:2202.03286, 2022. find our card
  29. Robert Tinn, Hao Cheng, Yu Gu, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Fine-tuning large neural language models for biomedical natural language processing. Patterns, 4(4), 2023. find
  30. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. walkerspider. https://old.reddit.com/r/ChatGP… ↗
  31. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022a. find
  32. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. arXiv preprint arXiv:2204.07705,… find
  33. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail? arXiv preprint arXiv:2307.02483, 2023. find our card
  34. Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109.10862, 2021. find our card
  35. Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher. arXiv preprint arXiv:2308.06463, 2023. find our card
  36. Weikang Zhou, Xiao Wang, Limao Xiong, Han Xia, Yingshuang Gu, Mingxu Chai, Fukang Zhu, Caishuang Huang, Shihan Dou, Zhiheng Xi, Rui Zheng, Songyang Gao, Yicheng Zou, Hang Yan, Yifan Le, Ruohui Wang, Lijun Li, Jing Shao, Tao Gui, Qi Zhang, and Xuanjing Huang. Easyjailbreak: A unified framework for ja… find
  37. Terry Yue Zhuo, Yujin Huang, Chunyang Chen, and Zhenchang Xing. Red teaming chatgpt via jailbreaking: Bias, robustness, reliability and toxicity. arXiv preprint arXiv:2301.12867, pp. 12– 2, 2023. find
  38. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023. find our card