Elamur Research
☰
home / papers / 2023 / Tree of Attacks: Jailbreaking Black-Box LLMs Automatically

Tree of Attacks: Jailbreaking Black-Box LLMs Automatically

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2312.02119

Summary

Aligned LLMs can still be pushed into producing harmful output. Most existing jailbreak techniques, however, need human effort or white-box access to weights, or they produce gibberish strings that perplexity filters catch. The paper introduces TAP (Tree of Attacks with Pruning), a fully automatic method that requires only black-box access and produces attack prompts written in natural language. TAP extends PAIR. An attacker LLM, guided by a detailed system prompt with chain-of-thought, repeatedly rewrites a harmful request into several candidate variants (branching). An evaluator LLM then filters out off-topic candidates before they reach the target. After the target answers, the evaluator keeps only the top-scoring candidates for the next round (pruning). Experiments use the AdvBench Subset and a newly built dataset, with GPT4 as an automatic judge plus manual review. TAP jailbreaks GPT4, GPT4-Turbo, GPT4o and Gemini-Pro on over 80% of requests, averaging under 30 queries. It beats PAIR while querying less and also succeeds against models shielded by Llama-Guard. The results show that keeping model internals hidden does not prevent cheap, scalable attacks, which matters for red-teaming and alignment work.

Key points

  • On GPT4o, TAP raises the success rate from 78% (the previous best black-box method) to 94% and uses about 60% fewer queries.
  • On the AdvBench Subset with GPT4 as target, TAP reaches 90% success at 28.8 queries, compared with 60% at 37.7 queries for the strongest prior method (parallel PAIR).
  • In ablations with GPT4-Turbo as target, removing branching drops success to 48% (vs 84%) even with more queries (33.1 vs 22.5). Removing pruning roughly doubles query cost (55.4) without any gain in success.
  • Each iteration keeps w = 10 best-scoring prompts. The GPT4 judge rates prompts on a 1–10 scale and shows about 13% false positives and 0% false negatives when flagging jailbreaks.
  • TAP's prompts transfer to other models about as well as those of the baselines. Against Llama-Guard-protected targets, it stays effective with fewer than 50 queries.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

161 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 157 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2312.02119 · arXiv 2312.02119

@misc{mehrotra2023tree,
  title = {{Tree of Attacks: Jailbreaking Black-Box LLMs Automatically}},
  author = {Mehrotra, Anay and Zampetakis, Manolis and Kassianik, Paul and Nelson, Blaine and Anderson, Hyrum and Singer, Yaron and Karbasi, Amin},
  year = {2023},
  doi = {10.48550/arxiv.2312.02119},
  eprint = {2312.02119},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2312.02119}
}

References

49 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. “Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks”. In: (2024). arXiv: 2404.02151 [cs.CR]. URL : https://doi.org/10.48550/arXiv.2404.02151 (cit. on p. 6). Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmit… ↗
  2. Abubakar Abid, Maheen Farooqi, and James Zou. “Persistent Anti-Muslim Bias in Large Language Models”. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’21. Virtual Event, USA: Association for Computing Machinery, 2021, pp. 298–306. ISBN : 9781450384735. DOI : 10.1145/… find our card
  3. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete Problems in AI Safety. 2016. arXiv: 1606.06565 [cs.AI] (cit. on p. 3). find our card
  4. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?” In: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. FAccT ’21. Virtual Event, Canada: Association for Compu… — [doi:10.1145/3442188.3445922](https://doi.org/10.1145/3442188.3445922) find
  5. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora De… find our card
  6. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova Das-Sarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, S… find
  7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… find
  8. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. “Deep Reinforcement Learning from Human Preferences”. In: Advances in Neural Information Processing Systems. Ed. by I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett. Vol.… find
  9. Stephen Casper, Jason Lin, Joe Kwon, Gatlen Culp, and Dylan Hadfield-Menell. Explore, Establish, Exploit: Red Teaming Language Models from Scratch. 2023. arXiv: 2306.09442 [cs.CL] (cit. on p. 6). find
  10. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. Jailbreaking Black Box Large Language Models in Twenty Queries. 2023. arXiv: 2310.08419 [cs.LG] (cit. on pp. 4–7, 9, 11, 12, 14, 25, 26, 36, 37, 39, 40, 42). find our card
  11. Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. “Extracting Training Data from Large Language Models”. In: 30th USENIX Security Symposium (USENIX Security 21). U… find our card
  12. Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu. “MASTERKEY: Automated Jailbreaking of Large Language Model Chatbots”. In: Proceedings 2024 Network and Distributed System Security Symposium. NDSS 2024. Internet Society, 2024. DOI : 10.1472… — [doi:10.14722/ndss.2024.24188](https://doi.org/10.14722/ndss.2024.24188) find
  13. Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. “Toxicity in ChatGPT: Analyzing Persona-Assigned Language Models”. In: Findings of the Association for Computational Linguistics: EMNLP 2023. Ed. by Houda Bouamor, Juan Pino, and Kalika Bali. Singapore: Asso… find our card
  14. Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. “HotFlip: White-Box Adversarial Examples for Text Classification”. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Ed. by Iryna Gurevych and Yusuke Miyao. Melbourne, Australia… find our card
  15. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan… find our card
  16. Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. “Gradient-based Adversarial Attacks against Text Transformers”. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Ed. by Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yi… find
  17. Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. “Explaining and Harnessing Adversarial Examples”. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. Ed. by Yoshua Bengio and Yann LeCun. 2015. URL : http… find our card
  18. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. “Aligning AI With Shared Human Values”. In: International Conference on Learning Representations. 2021. URL : https://openreview.net/forum?id=dNy\_RKzJacY (cit. on p. 3). Babak Hemmatian and Lav R. V… ↗ our card
  19. Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations. 2023. arXiv: 2312.06674 [cs.CL] (cit. on pp. 4, 11, 38, 40). find our card
  20. Erik Jones, Anca Dragan, Aditi Raghunathan, and Jacob Steinhardt. “Automatically Auditing Large Language Models via Discrete Optimization”. In: Proceedings of the 40th International Conference on Machine Learning. Ed. by Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato… find
  21. Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. “BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset”. In: Thirty-seventh Conference on Neural Information Processing Systems Datasets and Ben… find our card
  22. Hadas Kotek, Rikker Dockum, and David Sun. “Gender Bias and Stereotypes in Large Language Models”. In: Proceedings of The ACM Collective Intelligence Conference. CI ’23. Delft, Netherlands: Association for Computing Machinery, 2023, pp. 12– 24. DOI : 10.1145/3582269.3615599. URL : https://doi.org/10… — [doi:10.1145/3582269.3615599](https://doi.org/10.1145/3582269.3615599) find
  23. Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto. “Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks”. In: 2024 IEEE Security and Privacy Workshops (SPW). 2024, pp. 132–143. DOI : 10.1109/SPW63631.2024.00018 (cit. on p. 6… — [doi:10.1109/spw63631.2024.00018](https://doi.org/10.1109/spw63631.2024.00018) find
  24. Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study. 2023. arXiv: 2305.13860 [cs.SE] (cit. on p. 3). find our card
  25. Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, Jie Huang, Fanpu Meng, and Yangqiu Song. “Multi-step Jailbreaking Privacy Attacks on ChatGPT”. In: The 2023 Conference on Empirical Methods in Natural Language Processing. 2023. URL : https://openreview. net/forum?id=ls4Pfsl2jZ (cit. on pp. 3, 7, 14). find our card
  26. Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. “TextBugger: Generating Adversarial Text Against Real-world Applications”. In: NDSS. The Internet Society, 2019 (cit. on p. 42). Raz Lapid, Ron Langberg, and Moshe Sipper. Open Sesame! Universal Black Box Jailbreaking of Large Language Models… — [arXiv:2309.01446](https://arxiv.org/abs/2309.01446) find
  27. Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. “BERT-ATTACK: Adversarial Attack Against BERT Using BERT”. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Ed. by Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu. Online: Associa… find our card
  28. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”. In: Advances in Neural Information Process… find
  29. Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. “AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models”. In: The Twelfth International Conference on Learning Representations. 2024. URL : https://openreview. net/forum?id=7Jwpw4qKkb (cit. on pp. 6, 42). find our card
  30. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. “HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal”. In: Forty-first International Conference on Mach… find our card
  31. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Ja… find our card
  32. Danish Pruthi, Bhuwan Dhingra, and Zachary C. Lipton. “Combating Adversarial Misspellings with Robust Word Recognition”. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Ed. by Anna Korhonen, David Traum, and Lluís Màrquez. Florence, Italy: Association for… find our card
  33. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models. 2024. arXiv: 2308.03825 [cs.CR] (cit. on p. 6). find our card
  34. Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. “The Woman Worked as a Babysitter: On Biases in Language Generation”. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing… find our card
  35. Rusheb Shah, Quentin Feuillade–Montixi, Soroush Pour, Arush Tagade, Stephen Casper, and Javier Rando. Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation. 2023. arXiv: 2311.03348 [cs.CL] (cit. on p. 6). — [arXiv:2311.03348](https://arxiv.org/abs/2311.03348) find
  36. Muhammad Ahmed Shah, Roshan Sharma, Hira Dhamyal, Raphael Olivier, Ankit Shah, Joseph Konan, Dareen Alharthi, Hazim T Bukhari, Massa Baali, Soham Deshmukh, Michael Kuhlmann, Bhiksha Raj, and Rita Singh. LoFT: Local Proxy Fine-tuning For Improving Transferability Of Adversarial Attacks Against Large… — [arXiv:2310.04445](https://arxiv.org/abs/2310.04445) find
  37. Liwei Song, Xinwei Yu, Hsuan-Tung Peng, and Karthik Narasimhan. “Universal Adversarial Attacks with Natural Triggers for Text Classification”. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Ed. by Kr… find
  38. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. “Intriguing Properties of Neural Networks”. In: ICLR (Poster). 2014 (cit. on p. 42). find our card
  39. Samson Tan, Shafiq Joty, Min-Yen Kan, and Richard Socher. “It’s Morphin’ Time! Combating Linguistic Discrimination with Inflectional Perturbations”. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Ed. by Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joe… find
  40. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… find
  41. Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. “Universal Adversarial Triggers for Attacking and Analyzing NLP”. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processi… find
  42. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. “Jailbroken: How Does LLM Safety Training Fail?” In: Thirty-seventh Conference on Neural Information Processing Systems. 2023. URL : https://openreview.net/forum?id=jA235JGM09 (cit. on pp. 3, 6, 7, 10, 14, 38). WitchBOT. You can use GPT-4 to creat… ↗ our card
  43. Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. “Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery”. In: Advances in Neural Information Processing Systems. Ed. by A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hard… find
  44. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter brian, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models”. In: Advances in Neural Information Processing Systems. Ed. by S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrav… find our card
  45. Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning Large Language Models with Human: A Survey. 2023. arXiv: 2307.12966 [cs.CL] (cit. on p. 3). Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. Recipes f… find our card
  46. Zihao Xu, Yi Liu, Gelei Deng, Yuekang Li, and Stjepan Picek. “A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Models”. In: Findings of the Association for Computational Linguistics ACL 2024. Ed. by Lun-Wei Ku, Andre Martins, and Vivek Srikumar. Bangkok, Thailand and virtu… find our card
  47. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. “Tree of Thoughts: Deliberate Problem Solving with Large Language Models”. In: Advances in Neural Information Processing Systems. Ed. by A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Lev… find
  48. Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. “Word-level Textual Adversarial Attacking as Combinatorial Optimization”. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Ed. by Dan Jurafsky, Joyce Chai, Natalie Sch… find our card
  49. Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-Tuning Language Models from Human Preferences. 2020. arXiv: 1909.08593 [cs.CL] (cit. on p. 3). Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and… find