Elamur Research
☰
home / papers / 2023 / Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations

Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2310.06387

Summary

Aligned LLMs can still be manipulated into producing harmful output. Existing jailbreaks are either expensive, relying on iterative optimization as in GCG or AutoDAN, or rigid hand-written templates. Wei et al. show that in-context learning alone can push a model's safety behaviour in either direction.

Their In-Context Attack (ICA) prepends a few query–response pairs in which malicious requests are answered compliantly. In-Context Defense (ICD) instead prepends pairs where such requests are refused. The demonstrations do not need to match the target query and can be reused across prompts. ICA needs only one forward pass, and because its prompts are ordinary natural language, perplexity filters struggle to flag them.

A theoretical framework explains the effect: demonstrations bias the generation distribution toward a harmful or safe target distribution. Experiments on AdvBench with Vicuna, Llama-2, QWen and GPT-4 show high attack success for ICA and sharp ASR reductions under ICD, at negligible cost to general ability. This gives a cheap, training-free tool for both red-teaming and guarding LLMs.

Key points

  • ICA reaches an 81% Attack Success Rate (ASR) on GPT-4 using the AdvBench dataset.
  • With 1–2 safe demonstrations, ICD cuts Llama-2's ASR against transferable GCG from 21% to 0%. It brings most black-box cases (GCG-T, PAIR) close to zero and outperforms the Self-reminder baseline.
  • Against white-box adaptive attacks (GCG, AutoDAN), 2-shot ICD lowers average GCG ASR from 65% to 40%.
  • Natural performance on GLUE and MT-Bench stays comparable, with GPT-4 even improving on MT-Bench (9.24 vs 8.89). Inference overhead is about 2% for Vicuna and under 1% for GPT-4.
  • When ICA and ICD are combined on Vicuna-7b, a 1-shot attack is almost neutralised. However, 10 harmful shots still reach 56–59% ASR against ICD, compared with 87% without any defense.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

125 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 123 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.06387 · arXiv 2310.06387

@misc{wei2023jailbreak,
  title = {{Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations}},
  author = {Wei, Zeming and Wang, Yifei and Li, Ang and Mo, Yichuan and Wang, Yisen},
  year = {2023},
  doi = {10.48550/arxiv.2310.06387},
  eprint = {2310.06387},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.06387}
}

References

61 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gabriel Alon and Michael Kamfonas. Detecting language model attacks with perplexity, 2023. 2, 3, 4, 5 find our card
  2. Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Jailbreaking leading safety-aligned llms with simple adaptive attacks. arXiv preprint arXiv:2404.02151, 2024. 1 find
  3. Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, et al. Foundational challenges in assuring alignment and safety of large language models. arXiv preprint arXiv:2404.09932, 2024. 1 find
  4. Jinze Bai et al. Qwen technical report, 2023. 6 find
  5. Yuntao Bai et al. Constitutional ai: Harmlessness from ai feedback, 2022. 1, 3 find our card
  6. Tom B. Brown et al. Language models are few-shot learners, 2020. 1, 3 find
  7. Bochuan Cao, Yuanpu Cao, Lu Lin, and Jinghui Chen. Defending against alignment-breaking attacks via robustly aligned llm, 2023. 2, 3 find
  8. Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods, 2017. 3 find
  9. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419, 2023. 1, 3, 6, 7 find our card
  10. Canyu Chen and Kai Shu. Combating misinformation in the age of llms: Opportunities and challenges. arXiv preprint arXiv:2311.05656, 2023. 1 find
  11. Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. Multilingual jailbreak challenges in large language models, 2023. 3 find our card
  12. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. A survey on in-context learning, 2023. 3 find our card
  13. Michael Feffer, Anusha Sinha, Zachary C Lipton, and Hoda Heidari. Red-teaming for generative ai: Silver bullet or security theater? arXiv preprint arXiv:2401.15897, 2024. 1 — [arXiv:2401.15897](https://arxiv.org/abs/2401.15897) find
  14. Or Honovich, Uri Shaham, Samuel R. Bowman, and Omer Levy. Instruction induction: From few examples to natural language task descriptions, 2022. 3 find
  15. Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. Catastrophic jailbreak of open-source llms via exploiting generation, 2023. 1 find our card
  16. Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. Baseline defenses for adversarial attacks against aligned language models, 2023. 2, 3, 4, 5 find our card
  17. Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Juntao Dai, and Yaodong Yang. Aligner: Achieving efficient alignment through weak-to-strong correction, 2024. 3 find
  18. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  19. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… find
  20. Fengqing Jiang, Zhangchen Xu, Luyao Niu, Zhen Xiang, Bhaskar Ramasubramanian, Bo Li, and Radha Poovendran. Artprompt: Ascii art-based jailbreak attacks against aligned llms. arXiv preprint arXiv:2402.11753, 2024. 3 find
  21. Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L. Buckley, Jason Phang, Samuel R. Bowman, and Ethan Perez. Pretraining language models with human preferences, 2023. 1 find
  22. Aounon Kumar, Chirag Agarwal, Suraj Srinivas, Soheil Feizi, and Hima Lakkaraju. Certifying llm safety against adversarial prompting, 2023. 3, 5 find
  23. Raz Lapid, Ron Langberg, and Moshe Sipper. Open sesame! universal black box jailbreaking of large language models, 2023. 3 find
  24. Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han. Deepinception: Hypnotize large language model to be jailbreaker, 2023. 1, 3, 6 find our card
  25. Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. Rain: Your language models can align themselves without finetuning, 2023. 5 find
  26. Zeyi Liao and Huan Sun. Amplegcg: Learning a universal and transferable generative model of adversarial suffixes for jailbreaking both open and closed llms, 2024. 3 find
  27. Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models, 2023. 1, 3, 6, 7, 8, 16 find our card
  28. Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. Jailbreaking chatgpt via prompt engineering: An empirical study, 2023. 3, 6 find our card
  29. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017. 3 find our card
  30. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. 2024. 6, 7 find our card
  31. Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. Tree of attacks: Jailbreaking black-box llms automatically, 2024. 1, 3, 7 find our card
  32. Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Noisy channel language model prompting for few-shot text classification. In ACL, 2022. 3 find our card
  33. OpenAI. Gpt-4 technical report, 2024. 2, 6 find our card
  34. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training la… find
  35. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models, 2022. 3 find our card
  36. Mansi Phute, Alec Helbling, Matthew Hull, ShengYun Peng, Sebastian Szyller, Cory Cornelius, and Duen Horng Chau. Llm self defense: By self examination, llms know they are being tricked, 2023. 2 find
  37. Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Peter Henderson, Mengdi Wang, and Prateek Mittal. Visual adversarial examples jailbreak aligned large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 21527–21536, 2024. 1 find
  38. Qibing Ren, Chang Gao, Jing Shao, Junchi Yan, Xin Tan, Wai Lam, and Lizhuang Ma. Exploring safety generalization challenges of large language models via code. In The 62nd Annual Meeting of the Association for Computational Linguistics, 2024. 3 find
  39. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. "do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models, 2023. 3 find our card
  40. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV au2, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts, 2020. 3 find our card
  41. Hugo Touvron et al. Llama 2: Open foundation and fine-tuned chat models, 2023. 2, 6 find
  42. Neeraj Varshney, Pavel Dolin, Agastya Seth, and Chitta Baral. The art of defending: A systematic evaluation and analysis of llm defense strategies on safety and over-defensiveness, 2023. 3 find
  43. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, 2019. 5, 9 find our card
  44. Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning. In Workshop on Efficient Systems for Foundation Models, 2023. 3 find
  45. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions, 2023. 3 find
  46. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail?, 2023. 1, 3 find our card
  47. Noam Wies, Yoav Levine, and Amnon Shashua. The learnability of in-context learning, 2023. 5 find
  48. Yotam Wolf, Noam Wies, Oshri Avnery, Yoav Levine, and Amnon Shashua. Fundamental limitations of alignment in large language models, 2023. 5 find
  49. Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. Defending chatgpt against jailbreak attack via self-reminders. Nature Machine Intelligence, pages 1–11, 2023. 2, 3, 5, 8, 9 find
  50. Benfeng Xu, Quan Wang, Zhendong Mao, Yajuan Lyu, Qiaoqiao She, and Yongdong Zhang. $k$NN prompting: Beyond-context learning with calibration-free nearest neighbor inference. In ICLR, 2023. 3 find
  51. Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. A survey on large language model (llm) security and privacy: The good, the bad, and the ugly. High-Confidence Computing, page 100211, 2024. 1 find our card
  52. Zheng-Xin Yong, Cristina Menghini, and Stephen H. Bach. Low-resource languages jailbreak gpt-4, 2023. 3 find our card
  53. Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. GPT-4 is too smart to be safe: Stealthy chat with LLMs via cipher. In ICLR, 2024. 3 find our card
  54. Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms, 2024. 1 find
  55. Yihao Zhang and Zeming Wei. Boosting jailbreak attack with momentum. In ICLR 2024 Workshop on Reliable and Responsible Foundation Models, 2024. 3 find
  56. Yihao Zhang, Zeming Wei, Jun Sun, and Meng Sun. Towards general conceptual model editing via adversarial representation engineering. arXiv preprint, 2024. 1 find
  57. Yiming Zhang, Shi Feng, and Chenhao Tan. Active example selection for in-context learning, 2022. 3 find our card
  58. Yiran Zhao, Wenyue Zheng, Tianle Cai, Xuan Long Do, Kenji Kawaguchi, Anirudh Goyal, and Michael Shieh. Accelerating greedy coordinate gradient via probe sampling. arXiv preprint arXiv:2403.01251, 2024. 3 — [arXiv:2403.01251](https://arxiv.org/abs/2403.01251) find
  59. Lianmin Zheng et al. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. 5, 6, 7, 9 find
  60. Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. Autodan: Interpretable gradient-based adversarial attacks on large language models, 2023. 4, 6 find
  61. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023. 1, 2, 3, 4, 6, 7 find our card