Baseline Defenses for Adversarial Attacks Against Aligned Language Models
Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2309.00614
Summary
Optimization-based jailbreaks, especially the universal, transferable suffix attack (GCG) of Zou et al. (2023), can automatically generate prompts that bypass the safety alignment of chat LLMs. These prompts can be crafted on open models and then used against API-only systems. The authors ask whether standard defense ideas from adversarial machine learning in computer vision work in this setting. They do not propose new methods. Instead, they benchmark three representative baseline families:
- perplexity-based detection,
- input preprocessing through paraphrasing or retokenization,
- adversarial training.
Each defense is tested under white-box and gray-box assumptions, and the authors measure how much it costs in normal model quality. They also propose a threat model that limits the attacker's compute budget, counted in model evaluations, rather than the size of the perturbation. Simple filtering and paraphrasing hold up surprisingly well, because discrete text optimization is weak and costly, which makes adaptive attacks harder than in vision. Adversarial training remains an open problem. The authors conclude that LLM security may not follow the lessons learned in computer vision.
Key points
- GCG suffixes reach high attack success rates on some models: 0.79 on Vicuna-7B, 0.7 on Falcon-7B-Inst. and 0.96 on Guanaco-7B. Success is much lower on ChatGLM-6B (0.04) and MPT-7B-Chat (0.12). Yet almost none of these prompts get past a basic or windowed perplexity filter (pass rates 0.00, with a maximum of 0.01).
- GCG is estimated to cost 5–6 orders of magnitude more than typical vision attacks. This motivates measuring attacker strength by compute budget.
- Following Zou et al., an attack counts as successful when the model's output contains none of a fixed list of refusal phrases.
- Paraphrasing is not invulnerable. An adaptive attack made Vicuna-7B, acting as the paraphraser, output a harmful request together with an exact adversarial suffix.
- To measure the cost to model utility, Alpaca was fine-tuned for 3 epochs (batch 128, learning rate 5e-5, 4 RTX A5000 GPUs, about half a day). It was then scored with AlpacaEval against Text-Davinci-003 references, reaching roughly a 48% win rate versus about 44% with a learning rate of 2e-5.
Cited by
- Detecting Language Model Attacks with Perplexity
- Jailbreaking Black Box Large Language Models in Twenty Queries
- JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models
- DeepInception: Hypnotize Large Language Model to Be Jailbreaker
- AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
- HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
- SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks
- Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations
- A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly
Cite
@misc{jain2023baseline,
title = {{Baseline Defenses for Adversarial Attacks Against Aligned Language Models}},
author = {Jain, Neel and Schwarzschild, Avi and Wen, Yuxin and Somepalli, Gowthami and Kirchenbauer, John and Chiang, Ping-yeh and Goldblum, Micah and Saha, Aniruddha and Geiping, Jonas and Goldstein, Tom},
year = {2023},
doi = {10.48550/arxiv.2309.00614},
eprint = {2309.00614},
archivePrefix = {arXiv},
url = {https://doi.org/10.48550/arxiv.2309.00614}
}References
- Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples. In Proceedings of the 35th International Conference on Machine Learning, pp. 274–283. PMLR, July 2018. URL https: //proceedings.mlr.press/v80/athaly… find
- Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022a. find
- Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022b. find our card
- Arjun Nitin Bhagoji, Daniel Cullina, Chawin Sitawarin, and Prateek Mittal. Enhancing robustness of machine learning systems via data transformations. In 2018 52nd Annual Conference on Information Sciences and Systems (CISS), pp. 1–5. find
- Nicholas Carlini and David Wagner. Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, AISec ’17, pp. 3–14, New York, NY, USA, November 2017. Association for Computing Machinery. ISBN 978-1-45… — [doi:10.1145/3128572.3140444](https://doi.org/10.1145/3128572.3140444) find
- Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On Evaluating Adversarial Robustness. arxiv:1902.06705[cs, stat], February 2019. doi: 10.48550/arXiv.1902.06705. URL http://arxiv.org/abs/1902.067… — [doi:10.48550/arxiv.1902.06705](https://doi.org/10.48550/arxiv.1902.06705) ↗
- Nicholas Carlini, Milad Nasr, Christopher A. Choquette-Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, and Ludwig Schmidt. Are aligned neural networks adversarially aligned? arxiv:2306.15447[cs], June 2023. doi: 10.48550/arXiv.2306.154… find
- Stephen Casper, Jason Lin, Joe Kwon, Gatlen Culp, and Dylan Hadfield-Menell. Explore, Establish, Exploit: Red Teaming Language Models from Scratch. arxiv:2306.09442[cs], June 2023. doi: 10.48550/arXiv.2306.09442. URL http://arxiv.org/abs/2306.09442. ↗
- Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https: //lmsys.org/blog/2023-03-30-vicuna… find
- Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314, 2023. find
- Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305.14387, 2023. find our card
- Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. HotFlip: White-box adversarial examples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 31–36, Melbourne, Australia, July 2018. Association for Com… find our card
- Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858, 2022. find our card
- Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. Black-box generation of adversarial text sequences to evade deep learning classifiers. In 2018 IEEE Security and Privacy Workshops (SPW), pp. 50–56. find our card
- Yue Gao, I. Shumailov, Kassem Fawaz, and Nicolas Papernot. On the Limitations of Stochastic Pre-processing Defenses. In Advances in Neural Information Processing Systems, volume 35, pp. 24280–24294, December 2022. URL https://proceedings.neurips.cc/paper\_files/paper/2022/hash/997089469acbeb410405e43… ↗
- Amin Ghiasi, Ali Shafahi, and Tom Goldstein. Breaking certified defenses: Semantic adversarial examples with spoofed robustness certificates. In International Conference on Learning Representations, 2019. find
- Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375, 2022. find our card
- Micah Goldblum, Avi Schwarzschild, Ankit Patel, and Tom Goldstein. Adversarial attacks on machine learning systems for high-frequency trading. In Proceedings of the Second ACM International Conference on AI in Finance, pp. 1–9, 2021. find
- Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014. find our card
- Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. arxiv:2302.12173[cs], May 2023. doi: 10.48550/arXiv.2302.12173. URL http://arxiv.org/ab… ↗ our card
- Kathrin Grosse, Praveen Manoharan, Nicolas Papernot, Michael Backes, and Patrick McDaniel. On the (Statistical) Detection of Adversarial Examples. arxiv:1702.06280[cs, stat], October 2017. doi: 10.48550/arXiv.1702.06280. URL http://arxiv.org/abs/1702.06280. — [doi:10.48550/arxiv.1702.06280](https://doi.org/10.48550/arxiv.1702.06280) ↗
- Shixiang Gu and Luca Rigazio. Towards deep neural network architectures robust to adversarial examples. arXiv preprint arXiv:1412.5068, 2014. — [arXiv:1412.5068](https://arxiv.org/abs/1412.5068) find
- Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. Gradient-based Adversarial Attacks against Text Transformers. arxiv:2104.13733[cs], April 2021. doi: 10.48550/arXiv.2104.13733. URL http://arxiv.org/abs/2104.13733. ↗
- Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved Problems in ML Safety. arxiv:2109.13916[cs], June 2022. doi: 10.48550/arXiv.2109.13916. URL http: //arxiv.org/abs/2109.13916. find
- Hossein Hosseini and Radha Poovendran. Semantic adversarial examples. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1614–1619, 2018. find
- Neel Jain, Khalid Saifullah, Yuxin Wen, John Kirchenbauer, Manli Shu, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Bring your own data! self-supervised evaluation for large language models. arXiv preprint arXiv:2306.13651, 2023. — [arXiv:2306.13651](https://arxiv.org/abs/2306.13651) find
- Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto. Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks. arxiv:2302.05733[cs], February 2023. doi: 10.48550/arXiv.2302.05733. URL http://arxiv.org/abs/2302.05733. — [doi:10.48550/arxiv.2302.05733](https://doi.org/10.48550/arxiv.2302.05733) ↗
- John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum, and Tom Goldstein. On the reliability of watermarks for large language models. arXiv preprint arXiv:2306.04634, 2023. find
- Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. In International Conference on Machine Learning, pp. 17506–17533. PMLR, 2023. find
- Taku Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 66–75, Melbourne, Australia, July 2018. Association for Computation… find
- Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, Jie Huang, Fanpu Meng, and Yangqiu Song. Multi-step Jailbreaking Privacy Attacks on ChatGPT. arxiv:2304.05197[cs], May 2023. doi: 10.48550/arXiv.2304.05197. URL http://arxiv.org/abs/2304.05197. ↗ our card
- Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. Textbugger: Generating adversarial text against real-world applications. arXiv preprint arXiv:1812.05271, 2018. find our card
- Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. BERT-ATTACK: Adversarial attack against BERT using BERT. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6193–6202, Online, November 2020. Association for Computational Linguisti… find our card
- Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017. find our card
- Ninareh Mehrabi, Palash Goyal, Christophe Dupuy, Qian Hu, Shalini Ghosh, Richard Zemel, Kai-Wei Chang, Aram Galstyan, and Rahul Gupta. FLIRT: Feedback Loop In-context Red Teaming. arxiv:2308.04265[cs], August 2023. doi: 10.48550/arXiv.2308.04265. URL http://arxiv. org/abs/2308.04265. find
- Dongyu Meng and Hao Chen. Magnet: a two-pronged defense against adversarial examples. In Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, pp. 135–147, 2017a. find
- Dongyu Meng and Hao Chen. MagNet: A Two-Pronged Defense against Adversarial Examples. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, CCS ’17, pp. 135–147, New York, NY, USA, October 2017b. Association for Computing Machinery. ISBN 978-1-4503-4946-8. doi: 10… — [doi:10.1145/3133956.3134057](https://doi.org/10.1145/3133956.3134057) find
- Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. arXiv preprint arXiv:1702.04267, 2017. — [arXiv:1702.04267](https://arxiv.org/abs/1702.04267) find
- Han Cheol Moon, Shafiq Joty, Ruochen Zhao, Megh Thakkar, and Xu Chi. Randomized smoothing with masked inference for adversarially robust text classifications. arXiv preprint arXiv:2305.06522, 2023. find
- John X. Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP. arXiv:2005.05909 [cs], October 2020. URL http://arxiv.org/abs/2005.05909. arXiv: 2005.05909. ↗ our card
- Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Animashree Anandkumar. Diffusion Models for Adversarial Purification. In Proceedings of the 39th International Conference on Machine Learning, pp. 16805–16827. PMLR, June 2022. URL https: //proceedings.mlr.press/v162/nie22a.html. find
- Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730–27744, 2022. find
- Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116… find
- Fábio Perez and Ian Ribeiro. Ignore Previous Prompt: Attack Techniques For Language Models. arxiv:2211.09527[cs], November 2022. doi: 10.48550/arXiv.2211.09527. URL http: //arxiv.org/abs/2211.09527. find our card
- Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. Bpe-dropout: Simple and effective subword regularization. arXiv preprint arXiv:1910.13267, 2019. find our card
- Xiangyu Qi, Kaixuan Huang, Ashwinee Panda, Mengdi Wang, and Prateek Mittal. Visual Adversarial Examples Jailbreak Aligned Large Language Models. In The Second Workshop on New Frontiers in Adversarial Machine Learning, August 2023. URL https://openreview. net/forum?id=cZ4j7L6oui. find
- Sylvestre-Alvise Rebuffi, Sven Gowal, Dan Andrei Calian, Florian Stimberg, Olivia Wiles, and Timothy A Mann. Data Augmentation Can Improve Robustness. In Advances in Neural Information Processing Systems, volume 34, pp. 29935–29948. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/p… ↗
- Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense-gan: Protecting classifiers against adversarial attacks using generative models. arXiv preprint arXiv:1805.06605, 2018. — [arXiv:1805.06605](https://arxiv.org/abs/1805.06605) find
- Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips… ↗
- Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. ” do anything now”: Characterizing and evaluating in-the-wild jailbreak prompts on large language models. arXiv preprint arXiv:2308.03825, 2023. find our card
- Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980, 2020. find our card
- Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013. find our card
- Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford\_alpaca, 2023. ↗
- MosaicML NLP Team. Introducing mpt-7b: A new standard for open-source, ly usable llms, 2023. URL www.mosaicml.com/blog/mpt-7b. find
- Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. find
- Florian Tramer. Detecting Adversarial Examples Is (Nearly) As Hard As Classifying Them. In Proceedings of the 39th International Conference on Machine Learning, pp. 21692–21702. PMLR, June 2022. URL https://proceedings.mlr.press/v162/tramer22a.html. ↗
- Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On Adaptive Attacks to Adversarial Example Defenses. In Advances in Neural Information Processing Systems, volume 33, pp. 1633–1645. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/hash/11f38f8ecd71… ↗
- Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for attacking and analyzing NLP. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing… find
- Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022. find
- Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery. arXiv preprint arXiv:2302.03668, February 2023. URL https://arxiv.org/abs/2302.03668v1. ↗
- Zuxuan Wu, Ser-Nam Lim, Larry S Davis, and Tom Goldstein. Making an invisibility cloak: Real world adversarial attacks on object detectors. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16, pp. 1–17. find
- Xiaoyong Yuan, Pan He, Qile Zhu, and Xiaolin Li. Adversarial examples: Attacks and defenses for deep learning. IEEE transactions on neural networks and learning systems, 30(9):2805–2824, 2019. find
- Zhen Zhang, Guanhua Zhang, Bairu Hou, Wenqi Fan, Qing Li, Sijia Liu, Yang Zhang, and Shiyu Chang. Certified robustness for large language models with self-denoising. arXiv preprint arXiv:2307.07171, 2023. — [arXiv:2307.07171](https://arxiv.org/abs/2307.07171) find
- Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. Freelb: Enhanced adversarial training for natural language understanding. arXiv preprint arXiv:1909.11764, 2019. find our card
- Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Neil Zhenqiang Gong, Yue Zhang, and Xing Xie. PromptBench: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts. arxiv:2306.04528[cs], June 2023. doi: 10.48550/arXiv.2306.0452… find our card
- Roland S. Zimmermann, Wieland Brendel, Florian Tramer, and Nicholas Carlini. Increasing Confidence in Adversarial Robustness Evaluations. In Advances in Neural Information Processing Systems, volume 35, pp. 13174–13189, December 2022. URL https://proceedings.neurips.cc/paper\_files/paper/2022/hash/55… ↗
- Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and Transferable Adversarial Attacks on Aligned Language Models. arxiv:2307.15043[cs], July 2023. doi: 10.48550/arXiv.2307.15043. URL http://arxiv.org/abs/2307.15043. ↗ our card