Elamur Research
☰
home / papers / 2023 / Detecting Language Model Attacks with Perplexity

Detecting Language Model Attacks with Perplexity

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 4.0 (arXiv)" does not allow redistributing the text itself. Original — arxiv.org/pdf/2308.14132

Summary

The paper looks at whether a simple fluency signal can expose jailbreak prompts that use adversarial suffixes. These are odd token strings produced by the GCG algorithm of Zou et al. (2023), and they can push aligned LLMs into producing harmful content. The authors ran the published GCG code on Vicuna-7b-1.5 and collected 1407 attack prompts built from "Harmful Behaviors" objectives. They then scored these prompts and a large pool of ordinary prompts with GPT-2 perplexity. The ordinary prompts came from seven Huggingface datasets, and attacks made up under 1% of the combined data. The attack prompts had very high perplexity. However, a single perplexity cutoff misclassified many benign prompts, such as code, math or multilingual text. To fix this, the authors trained a LightGBM classifier on two features, perplexity and token sequence length, and tuned it for the F2 score. This resolved most false positives and caught most machine-generated attacks in the test split. The method did not detect human-written jailbreaks. The work matters because it offers a cheap, model-agnostic first-line filter that needs no access to the protected model.

Key points

  • Attack data: 1407 adversarial prompts were generated with "individual" GCG on Vicuna-7b-1.5, using 67 randomly chosen harmful objectives. Generation took about 50 hours on one A100 (40GiB).
  • Benign data:
    • DocRED (998)
    • SuperGLUE boolq (3270)
    • SQuAD-v2 (11873)
    • Open-Platypus (24926)
    • Puffin (6994 human turns, plus 3000 opening prompts)
    • Tapir (116862)
    • the first 10,000 items of instructional code-search-net-python
  • Method: GPT-2 perplexity and token length were fed to a LightGBM model. Data was split 50:25:25 for attacks and 70:15:15 for benign prompts. Decision thresholds were tuned on validation data with β = 2, which favours recall.
  • Main finding: Plain perplexity thresholding gives too many false positives, while the two-feature classifier separates the classes far better.
  • Limitation: Human-crafted jailbreaks have natural-looking perplexity and slip through.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

126 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 119 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2308.14132 · arXiv 2308.14132

@misc{alon2023detecting,
  title = {{Detecting Language Model Attacks with Perplexity}},
  author = {Alon, Gabriel and Kamfonas, Michael},
  year = {2023},
  doi = {10.48550/arxiv.2308.14132},
  eprint = {2308.14132},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.14132}
}

References

39 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Sh… find
  2. Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. Theoremqa: A theorem-driven question answering dataset. preprint arXiv:2305.12524, 2023. find our card
  3. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL, 2019. find our card
  4. Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 1310–1320.… find
  5. Maria Cvach. Monitor alarm fatigue: an integrative review. Biomedical instrumentation & technology, 2012. find
  6. Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Campbell-Gillingham, Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug Fr… find
  7. Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples, 2015. find our card
  8. Wenjuan Han, Liwen Zhang, Yong Jiang, and Kewei Tu. Adversarial attack and defense of structured prediction models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2327–2338, Online, November 2020. Association for Computational Linguistics. doi:… find
  9. Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ml safety, 2022. find
  10. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9. ↗
  11. Huggingface. perplexity, howpublished = https://huggingface.co/docs/transformers/perplexity, note = Accessed: 2023-08-26, 2023. ↗
  12. Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. Baseline defenses for adversarial attacks against aligned language models, 2023. find our card
  13. Rubén Darı́o Jaramillo. Jaramilo:gpt4jailbreak, howpublished = https://huggingface.co/datasets/rubend18/chatgpt-jailbreak-prompts, 2023. Accessed: 2023-09-20. ↗
  14. Erik Jones, Anca Dragan, Aditi Raghunathan, and Jacob Steinhardt. Automatically auditing large language models via discrete optimization, 2023. find
  15. Raz Lapid, Ron Langberg, and Moshe Sipper. Open sesame! universal black box jailbreaking of large language models, 2023. find
  16. Ariel N. Lee, Cole J. Hunter, and Nataniel Ruiz. Platypus: Quick, cheap, and powerful refinement of llms, 2023. find
  17. Klas Leino, Zifan Wang, and Matt Fredrikson. Globally-robust neural networks, 2021. find
  18. Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. Rain: Your language models can align themselves without finetuning, 2023. find
  19. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=rJzIBfZAb. ↗ our card
  20. Annunziata Elefante Mattia Limone, Gaetano Cimino. Tapir: Trigger action platform for information retrieval. https://github.com/MattiaLimone/ifttt\_recommendation\_system, 2023. ↗
  21. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018. find
  22. OpenAI. Gpt-4 technical report, 2023. find our card
  23. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training la… find
  24. Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples, 2016. find
  25. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. In NA. OpenAI, 2019. URL https: //api.semanticscholar.org/CorpusID:160025533. find
  26. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv e-prints, art. arXiv:1606.05250, 2016. find our card
  27. Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, John J. Nay, Kshitij Gupta, and Aran Komatsuzaki. Arb: Advanced reasoning benchmark for large language models, 2023. find
  28. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV au2, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts, 2020. find our card
  29. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, and Nikolay Bashlykov. Llama 2: Open foundation and fine-tuned chat models, 2023. find
  30. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537, 2019. find our card
  31. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions, 2023. find
  32. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail?, 2023. find our card
  33. Yotam Wolf, Noam Wies, Oshri Avnery, Yoav Levine, and Amnon Shashua. Fundamental limitations of alignment in large language models, 2023. find
  34. Jingjing Xu, Xuancheng Ren, Junyang Lin, and Xu Sun. Dp-gan: Diversity-promoting generative adversarial network for generating informative and diversified text, 2018. find
  35. Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. DocRED: A large-scale document-level relation extraction dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 764–777, Florence, It… find
  36. Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. Reclor: A reading comprehension dataset requiring logical reasoning. In International Conference on Learning Representations (ICLR), April 2020. find our card
  37. Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher, 2023. find our card
  38. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. find
  39. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023. find our card