Elamur Research⌕
☰
home / papers / 2023 / A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily

A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily

Paper · 2023 · Peng Ding, Jun Kuang, Dan Ma et al.

Summary

Safety-aligned LLMs can still be manipulated into giving harmful answers through jailbreak prompts. Existing attacks either depend on laborious hand-written templates (e.g., DAN) or on slow gradient search over white-box models (e.g., GCG). GCG produces nonsensical suffixes that are easy to filter and work poorly on commercial systems. The authors argue that most jailbreaks come down to two moves, and they build ReNeLLM, an automatic, training-free framework in which LLMs generate attacks against other LLMs. First, a harmful request is altered by a random chain of meaning-preserving edits, and an LLM classifier confirms the harmful intent is still present. Second, the altered request is hidden inside a harmless-looking task, and attempts repeat until an evaluator LLM judges the target's reply harmful. Experiments show higher attack success at much lower time cost than baselines, transfer across open and commercial models, and weak existing defenses. Attention analysis suggests nesting changes which part of a prompt the model prioritizes, which motivates priority-based defenses. The work offers a practical red-teaming tool and a warning for safety alignment research.

Key points

  • The attack is framed as a search over a finite strategy space. The goal is to maximize the harmfulness score that an evaluator LLM assigns to the target model's response.
  • Six rewriting operations are used: shortening, restructuring sentences, misspelling sensitive words, inserting meaningless characters, partial translation and style change. These feed into three nesting scenarios: code completion, text continuation and table filling.
  • Supervised fine-tuning of llama-2-chat-13b on code-completion attack data lowered table-filling ASR from 100 to 0. Text continuation still reached an ASR of 88.1, so the defense did not generalize.
  • Used as a harmfulness classifier, GPT-3.5 cut GCG's ASR by 86.0 but caught only 17.7 for ReNeLLM. GPT-4 was stronger, yet ReNeLLM kept about 10% effectiveness while the other methods fell near zero.
  • The listed limitations are fixed nesting scenarios, English-only datasets, random selection of strategies, and reliance on costly online models (GPT-3.5, GPT-4, Claude2). Code is available at github.com/NJUNLP/ReNeLLM.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Nanjing UniversityNanjing · China99 works · 8,512 citations🏢MeituanBeijing · China26 works · 3,953 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksBias, fairness & safety · 587 works

Cite

Cite as Peng Ding, Jun Kuang, Dan Ma, Xuezhi Cao, Yunsen Xian, Jiajun Chen and Shujian Huang. 2023. A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily. arXiv:2311.08268.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2311.08268 · arXiv 2311.08268

@misc{ding2023wolf,
  title = {{A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily}},
  author = {Peng Ding, Jun Kuang, Dan Ma, Xuezhi Cao, Yunsen Xian, Jiajun Chen and Shujian Huang},
  year = {2023},
  doi = {10.48550/arxiv.2311.08268},
  eprint = {2311.08268},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2311.08268}
}

References

39 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Albert. 2023. https://www.jailbreakchat.com/. ↗
  2. Anthropic. 2023. Model card and evaluations for claude models, https://www-files.anthropic.com/production/images/Model-Card-Claude-2. pdf. ↗
  3. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. find
  4. Bochuan Cao, Yuanpu Cao, Lu Lin, and Jinghui Chen. 2023. Defending against alignment-breaking attacks via robustly aligned llm. arXiv preprint arXiv:2309.14348.
  5. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. 2023. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419.
  6. Noam Chomsky. 2002. Syntactic structures. Mouton de Gruyter. find
  7. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30. find
  8. Hao Fu, Yao; Peng and Tushar Khot. 2022. How does gpt obtain its ability? tracing emergent abilities of language models to their sources. Yao Fu’s Notion. find
  9. Josh A Goldstein, Girish Sastry, Micah Musser, Renee DiResta, Matthew Gentzel, and Katerina Sedova. 2023. Generative language models and automated influence operations: Emerging threats and potential mitigations. arXiv preprint arXiv:2301.04246. — [arXiv:2301.04246](https://arxiv.org/abs/2301.04246) find
  10. Julian Hazell. 2023. Large language models can be used to effectively scale spear phishing campaigns. arXiv preprint arXiv:2305.06972. — [arXiv:2305.06972](https://arxiv.org/abs/2305.06972) find
  11. Xingwei He, Zhenghao Lin, Yeyun Gong, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, Weizhu Chen, et al. 2023. Annollm: Making large language models to be better crowdsourced annotators. arXiv preprint arXiv:2303.16854.
  12. Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. 2023. Baseline defenses for adversarial attacks against aligned language models. arXiv preprint arXiv:2309.00614.
  13. Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto. 2023. Exploiting programmatic behavior of llms: Dual-use through standard security attacks. arXiv preprint arXiv:2302.05733. find
  14. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199– 22213.
  15. Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. 2023. Pretraining language models with human preferences. In International Conference on Machine Learning, pages 17506–17533. PMLR.
  16. Raz Lapid, Ron Langberg, and Moshe Sipper. 2023. Open sesame! universal black box jailbreaking of large language models. arXiv preprint arXiv:2309.01446. — [arXiv:2309.01446](https://arxiv.org/abs/2309.01446) find
  17. June M Liu, Donghao Li, He Cao, Tianhe Ren, Zeyi Liao, and Jiamin Wu. 2023a. Chatcounselor: A large language models for mental health support. arXiv preprint arXiv:2309.15461. — [arXiv:2309.15461](https://arxiv.org/abs/2309.15461) find
  18. Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2023b. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451.
  19. Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. 2023c. Jailbreaking chatgpt via prompt engineering: An empirical study. arXiv preprint arXiv:2305.13860.
  20. Zhaoyang Liu, Zeqiang Lai, Zhangwei Gao, Erfei Cui, Xizhou Zhu, Lewei Lu, Qifeng Chen, Yu Qiao, Jifeng Dai, and Wenhai Wang. 2023d. Controlllm: Augment language models with tools by searching on graphs. arXiv preprint arXiv:2310.17796. — [arXiv:2310.17796](https://arxiv.org/abs/2310.17796) find
  21. Todor Markov, Chong Zhang, Sandhini Agarwal, Florentine Eloundou Nekoul, Theodore Lee, Steven Adler, Angela Jiang, and Lilian Weng. 2023. A holistic approach to undesired content detection in the real world. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 15009–150…
  22. ONeal. 2023. Chatgpt-dan-jailbreak, https://gist.github.com/coolaj86/ 6f4f7b30129b0251f61fa7baaa881516. ↗
  23. ChatGPT, https://openai.com/ ↗
  24. OpenAI. 2023b. GPT-4 technical report, https://cdn.openai.com/papers/gpt-4.pdf. ↗
  25. OpenAI. 2023c. https://openai.com/policies/ usage-policies. ↗
  26. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
  27. Fábio Perez and Ian Ribeiro. 2022. Ignore previous prompt: Attack techniques for language models. arXiv preprint arXiv:2211.09527.
  28. Abhinav Rao, Sachin Vashistha, Atharva Naik, Somak Aditya, and Monojit Choudhury. 2023. Tricking llms into disobedience: Understanding, analyzing, and preventing jailbreaks. arXiv preprint arXiv:2305.14965. — [arXiv:2305.14965](https://arxiv.org/abs/2305.14965) find
  29. Gaurav Sahu, Olga Vechtomova, Dzmitry Bahdanau, and Issam H Laradji. 2023. Promptmix: A class boundary augmentation method for large language model distillation. arXiv preprint arXiv:2310.14192. find
  30. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2023. " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. arXiv preprint arXiv:2308.03825.
  31. Irene Solaiman and Christy Dennison. 2021. Process for adapting language models to society (palms) with values-targeted datasets. Advances in Neural Information Processing Systems, 34:5861–5873.
  32. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008– 3021. find
  33. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. walkerspider. 2022. DAN is my new friend., ht…
  34. Boxin Wang, Wei Ping, Chaowei Xiao, Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Bo Li, Anima Anandkumar, and Bryan Catanzaro. 2022a. Exploring the limits of domain-adaptive training for detoxifying large-scale language models. Advances in Neural Information Processing Systems, 35:35811– 35824. find
  35. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022b. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171.
  36. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593. find
  37. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How does llm safety training fail? arXiv preprint arXiv:2307.02483.
  38. Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.
  39. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ding-2023-wolf-sheep-s-clothing-generalized/