Elamur Research⌕
☰
home / papers / 2023 / Certifying LLM Safety against Adversarial Prompting

Certifying LLM Safety against Adversarial Prompting

Summary

Safety-aligned LLMs can still be jailbroken when an attacker attaches crafted adversarial tokens to a harmful request. Automated methods such as the Greedy Coordinate Gradient (GCG) attack make this easy. Existing defenses like perplexity filtering offer no formal guarantees, and newer attacks can evade them. The authors propose erase-and-check, which they describe as the first defense with provable safety guarantees. It deletes up to d tokens in various patterns, runs every resulting subsequence and the original prompt through a safety filter, and flags the input as harmful if any check fails. As a result, any harmful prompt the filter catches stays caught after adversarial additions of bounded size.

The filter is built in two ways: by prompting Llama 2, and by fine-tuning DistilBERT on harmful prompts from AdvBench plus safe prompts written by the authors. The method covers three threat models of increasing generality: suffix, insertion and infusion. Certified detection of harmful prompts is high, and accuracy on benign inputs remains good. The paper also introduces faster heuristic variants. This work shows that verifiable LLM safety is achievable, at a computational cost.

Key points

  • With the Llama 2 filter and 20-token adversarial suffixes, erase-and-check certifiably flags 92% of harmful prompts and classifies 97% of safe prompts correctly. The DistilBERT filter scores even higher and runs much faster.
  • The three attack modes are:
    • suffix: tokens appended at the end;
    • insertion: a contiguous block placed anywhere in the prompt;
    • infusion: tokens scattered at arbitrary positions.
  • In suffix mode, the count of subsequences that must be evaluated scales as O(d). Infusion becomes impractical for long adversarial sequences, which is the main limitation.
  • The three empirical variants are RandEC (random subsampling), GreedyEC (greedy erasure that maximizes the harmful-class softmax) and GradEC (gradient-guided erasure). They were tested against GCG prompts. Detection by the gradient-based variant rises from 0 to 76% as iterations increase to 100, with under 0.4 s per prompt.
  • The paper was published at COLM 2024. Code is available at github.com/aounon/certified-llm-safety.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Cited by

86 papers of our library cite it (of the 140884 whose bibliographies we read); 9 of them have a page here, 77 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2309.02705 · arXiv 2309.02705

@misc{kumar2023certifying,
  title = {{Certifying LLM Safety against Adversarial Prompting}},
  author = {Kumar, Aounon and Agarwal, Chirag and Srinivas, Suraj and Li, Aaron Jiaxun and Feizi, Soheil and Lakkaraju, Himabindu},
  year = {2023},
  doi = {10.48550/arxiv.2309.02705},
  eprint = {2309.02705},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2309.02705}
}

References

55 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Train… find
  2. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared… find our card
  3. Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin J. Chadwick, Phoebe Thacker, Lucy Campbell-Gillingham, Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug… find our card
  4. Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher L. Buckley, Jason Phang, Samuel R. Bowman, and Ethan Perez. Pretraining language models with human preferences. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, ed… find our card
  5. Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. Recipes for safety in open-domain chatbots. CoRR, abs/2010.07079, 2020. URL https://arxiv.org/abs/2010.07079. ↗
  6. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does LLM safety training fail? CoRR, abs/2307.02483, 2023. https://doi.org/10.48550/arXiv.2307.02483. URL https://doi.org/10.48550/ arXiv.2307.02483. Return to Table of Contents ↗ our card
  7. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023. find our card
  8. Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. Baseline defenses for adversarial attacks against aligned language models, 2023. find our card
  9. Gabriel Alon and Michael Kamfonas. Detecting language model attacks with perplexity, 2023. find our card
  10. Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models, 2023. find our card
  11. Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. Autodan: Automatic and interpretable adversarial attacks on large language models, 2023. find
  12. Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machin… find
  13. Florian Tramèr, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Confer… find
  14. Yunrui Yu, Xitong Gao, and Cheng-Zhong Xu. LAFEAT: piercing through adversarial defenses with latent features. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 5735–5745. Computer Vision Foundation / IEEE, 2021. https://doi.org/10.1109/CVPR46… — [doi:10.1109/cvpr46437.2021.00568](https://doi.org/10.1109/cvpr46437.2021.00568) ↗
  15. Nicholas Carlini and David A. Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, AISec@CCS 2017, Dallas, TX, USA, November 3, 2017, pages 3–14, 2017. https://doi.org/10.1145/3128572.3… — [doi:10.1145/3128572.3140444](https://doi.org/10.1145/3128572.3140444) ↗
  16. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. find
  17. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01108, 2019. URL http://arxiv.org/abs/1910.01108. ↗
  18. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 201… find our card
  19. Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Srndic, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Hendrik Blockeel, Kristian Kersting, Siegfried Nijssen, and Filip Zelezný, editors, Machine Learning and Knowledge Dis… — [doi:10.1007/978-3-642-40994-3_25](https://doi.org/10.1007/978-3-642-40994-3_25) find
  20. Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://… find our card
  21. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings… find our card
  22. Yangyi Chen, Hongcheng Gao, Ganqu Cui, Fanchao Qi, Longtao Huang, Zhiyuan Liu, and Maosong Sun. Why should adversarial perturbations be imperceptible? rethink the research paradigm in adversarial NLP. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on… find
  23. Jacob Buckman, Aurko Roy, Colin Raffel, and Ian J. Goodfellow. Thermometer encoding: One hot way to resist adversarial examples. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings, 2018. URL https://open… find
  24. Chuan Guo, Mayank Rana, Moustapha Cissé, and Laurens van der Maaten. Countering adversarial images using input transformations. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings, 2018. URL https://openr… find
  25. Guneet S. Dhillon, Kamyar Azizzadenesheli, Zachary C. Lipton, Jeremy Bernstein, Jean Kossaifi, Aran Khanna, and Animashree Anandkumar. Stochastic activation pruning for robust adversarial defense. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30… find
  26. Xin Li and Fuxin Li. Adversarial examples detection in deep networks with convolutional filter statistics. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 5775–5783, 2017. https://doi.org/10.1109/ICCV.2017.615. URL https://doi.org/10.1109/ICC… — [doi:10.1109/iccv.2017.615](https://doi.org/10.1109/iccv.2017.615) ↗
  27. Kathrin Grosse, Praveen Manoharan, Nicolas Papernot, Michael Backes, and Patrick D. McDaniel. On the (statistical) detection of adversarial examples. CoRR, abs/1702.06280, 2017. URL http://arxiv.org/abs/1702.06280. Return to Table of Contents — [arXiv:1702.06280](https://arxiv.org/abs/1702.06280) ↗
  28. Zhitao Gong, Wenlu Wang, and Wei-Shinn Ku. Adversarial and clean data are not twins. CoRR, abs/1704.04960, 2017. URL http://arxiv.org/abs/1704.04960. — [arXiv:1704.04960](https://arxiv.org/abs/1704.04960) ↗
  29. Dang Nguyen Minh and Anh Tuan Luu. Textual manifold-based defense against natural language adversarial examples. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6612–6625, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational L… find
  30. KiYoon Yoo, Jangho Kim, Jiho Jang, and Nojun Kwak. Detection of adversarial examples in text classification: Benchmark and baseline via robust density estimation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3656–3672, Dublin, Ireland, May 2022. Association for Compu… find
  31. Lukas Huber, Marc Alexander Kühn, Edoardo Mosca, and Georg Groh. Detecting word-level adversarial text attacks via SHapley additive exPlanations. In Proceedings of the 7th Workshop on Representation Learning for NLP, pages 156–166, Dublin, Ireland, May 2022. Association for Computational Linguistics… find
  32. Jonathan Uesato, Brendan O’Donoghue, Pushmeet Kohli, and Aäron van den Oord. Adversarial risk and the dangers of evaluating against weak attacks. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, pages 5032–5041… find
  33. Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Uesato, Relja Arandjelovic, Timothy Mann, and Pushmeet Kohli. On the effectiveness of interval bound propagation for training verifiably robust models, 2018. find
  34. Po-Sen Huang, Robert Stanforth, Johannes Welbl, Chris Dyer, Dani Yogatama, Sven Gowal, Krishnamurthy Dvijotham, and Pushmeet Kohli. Achieving verified robustness to symbol substitutions via interval bound propagation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Pro… find
  35. Krishnamurthy Dvijotham, Sven Gowal, Robert Stanforth, Relja Arandjelovic, Brendan O’Donoghue, Jonathan Uesato, and Pushmeet Kohli. Training verified learners with learned verifiers, 2018. find
  36. Matthew Mirman, Timon Gehr, and Martin Vechev. Differentiable abstract interpretation for provably robust neural networks. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 35… find
  37. Eric Wong and J. Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, pages 5283–5292, 2018. find
  38. Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. Semidefinite relaxations for certifying robustness to adversarial examples. In Proceedings of the 32nd International Return to Table of Contents Conference on Neural Information Processing Systems, NIPS’18, page 10900–10910, Red Hook, NY, USA, 20… find
  39. Sahil Singla and Soheil Feizi. Second-order provable defenses against adversarial attacks. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 8981–8991. PMLR, 2020. URL http… find
  40. Sahil Singla and Soheil Feizi. Skew orthogonal convolutions. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 9756–9766. PMLR, 2021.… find
  41. Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 1310–1… find
  42. Mathias Lécuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness to adversarial examples with differential privacy. In 2019 IEEE Symposium on Security and Privacy, SP 2019, San Francisco, CA, USA, May 19-23, 2019, pages 656–672, 2019. find
  43. Bai Li, Changyou Chen, Wenlin Wang, and Lawrence Carin. Certified adversarial robustness with additive noise. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pages 9459–… find
  44. Hadi Salman, Jerry Li, Ilya P. Razenshteyn, Pengchuan Zhang, Huan Zhang, Sébastien Bubeck, and Greg Yang. Provably robust deep learning via adversarially trained smoothed classifiers. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems… find
  45. Mao Ye, Chengyue Gong, and Qiang Liu. SAFER: A structure-free approach for certified robustness to adversarial word substitutions. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, A… find
  46. Haiteng Zhao, Chang Ma, Xinshuai Dong, Anh Tuan Luu, Zhi-Hong Deng, and Hanwang Zhang. Certified robustness against natural language attacks by causal intervention. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Ma… find
  47. Zhen Zhang, Guanhua Zhang, Bairu Hou, Wenqi Fan, Qing Li, Sijia Liu, Yang Zhang, and Shiyu Chang. Certified robustness for large language models with self-denoising. CoRR, abs/2307.07171, 2023. https://doi.org/10.48550/arXiv.2307.07171. URL https://doi.org/10.48550/arXiv.2307.07171. Return to Table… — [doi:10.48550/arxiv.2307.07171](https://doi.org/10.48550/arxiv.2307.07171) ↗
  48. Zhuoqun Huang, Neil G Marchant, Keane Lucas, Lujo Bauer, Olga Ohrimenko, and Benjamin I. P. Rubinstein. RS-del: Edit distance robustness certificates for sequence classifiers via randomized deletion. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.… ↗
  49. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Co… find
  50. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. ICLR, 2019. find
  51. Aounon Kumar, Alexander Levine, and Soheil Feizi. Policy smoothing for provably robust reinforcement learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=mwdfai8NBrJ. ↗
  52. Fan Wu, Linyi Li, Zijian Huang, Yevgeniy Vorobeychik, Ding Zhao, and Bo Li. CROP: Certifying robust policies for reinforcement learning through functional smoothing. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=HOjLHrlZhmx. ↗
  53. Aounon Kumar, Vinu Sankar Sadasivan, and Soheil Feizi. Provable robustness for streaming models with a sliding window, 2023. find
  54. Marc Fischer, Maximilian Baader, and Martin T. Vechev. Scalable certified segmentation via randomized smoothing. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Mach… find
  55. Aounon Kumar and Tom Goldstein. Center smoothing: Certified robustness for networks with structured outputs. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=sxjpM-kvVv_. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kumar-2023-certifying-llm-safety-against-adversarial/