"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

279 papers of our library cite it (of the 140673 whose bibliographies we read); 16 of them have a page here, 263 not yet.

• Refusal in Language Models Is Mediated by a Single Direction
• JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models
• Multilingual Jailbreak Challenges in Large Language Models
• Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation
• TrustLLM: Trustworthiness in Large Language Models
• Baseline Defenses for Adversarial Attacks Against Aligned Language Models
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training
• DeepInception: Hypnotize Large Language Model to Be Jailbreaker
• LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
• HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
• Tree of Attacks: Jailbreaking Black-Box LLMs Automatically
• OpenAI o1 System Card
• Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations
• A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly
• Low-Resource Languages Jailbreak GPT-4
• LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2308.03825 · arXiv 2308.03825

@misc{shen2023do,
  title = {{"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models}},
  author = {Shen, Xinyue and Chen, Zeyuan and Backes, Michael and Shen, Yun and Zhang, Yang},
  year = {2023},
  doi = {10.48550/arxiv.2308.03825},
  eprint = {2308.03825},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.03825}
}

References

59 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. [1] A pro-innovation approach to AI regulation. https: //assets.publishing.service.gov.uk/government/ uploads/system/uploads/attachment\_data/file/ 1146542/a\_pro-innovation\_approach\_to\_AI\_ regulation.pdf. 1, 3 [18] Nicholas Boucher, Ilia Shumailov, Ross Anderson, and Nicolas Papernot. Bad Characters:… find
2. [2] AIPRM. https://www.aiprm.com/. 4 [19] Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting Training Data from Large Language Models. In USENIX Secur… ↗ our card
3. [3] Awesome ChatGPT Prompts. https://huggingface.co/ datasets/fka/awesome-chatgpt-prompts. 4 [4] ChatGPT. https://chat.openai.com/chat. 1, 3, 8, 17 [5] Disboard. https://disboard.org/. 4 [6] Discord. https://en.wikipedia.org/wiki/Discord. 4 [20] Checkpoint. OPWNAI : Cybercriminals Starting To Use Ch… ↗
4. [7] FlowGPT. https://flowgpt.com/. 4 [8] General Data Protection Regulation (GDPR). gdpr-info.eu/. 3 https://[9] JailbreakChat. https://www.jailbreakchat.com. 1, 4 [21] Xiaoyi Chen, Ahmed Salem, Michael Backes, Shiqing Ma, Qingni Shen, Zhonghai Wu, and Yang Zhang. BadNL: Backdoor Attacks Against NLP… ↗
5. [10] Measures for the Management of Generative Artificial Intelligence Services. http://www.cac.gov.cn/2023-07/13/c\_ 1690898327029107.htm. 1, 3 [11] The Artificial Intelligence Act. artificialintelligenceact.eu/. 1, 3, 13 https://[22] Jianfeng Chi, Wasi Uddin Ahmad, Yuan Tian, and Kai-Wei Chang. PLU… ↗
6. [12] Meysam Alizadeh, Maël Kubli, Zeynab Samei, Shirin Dehghani, Juan Diego Bermeo, Maria Korobeynikova, and Fabrizio Gilardi. Open-Source Large Language Models Outperform Crowd Workers and Approach ChatGPT in Text-Annotation Tasks. CoRR abs/2307.02179, 2023. 17 — [arXiv:2307.02179](https://arxiv.org/abs/2307.02179) find
7. [23] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. 3, 8, 17 find
8. [13] Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinso… find our card
9. [24] Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep Reinforcement Learning from Human Preferences. In Annual Conference on Neural Information Processing Systems (NIPS), pages 4299– 4307. NIPS, 2017. 17 find
10. [25] Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023. 3, 8, 17 find
11. [26] Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu. Jailbreaker: Automated Jailbreak Across Multiple Large Language Model Chatbots. CoRR abs/2307.08715, 2023. 12 — [arXiv:2307.08715](https://arxiv.org/abs/2307.08715) find
12. [14] Eugene Bagdasaryan and Vitaly Shmatikov. Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasures. In IEEE Symposium on Security and Privacy (S&P), pages 769–786. IEEE, 2022. 13 find
13. [27] Discord. BreakGPT. https://disboard.org/server/ 1090300946568986810. 1 [28] Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. GLM: General Language Model Pretraining with Autoregressive Blank Infilling. In Annual Meeting of the Association for Computational… ↗ our card
14. [15] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. A Multitask, Multilingual, Multimodal Evaluation of Chat-GPT on Reasoning, Hallucination, and Interactivity. CoRR abs/2302.0402… find our card
15. [29] Rosa Falotico and Piero Quatto. Fleiss’ kappa statistic without paradoxes. Quality & Quantity, 2015. 5 find
16. Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. ACM Computing Surveys, 2023. 3 [46] Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study. CoRR ab… find our card
17. IEEE, 2023. 13 [48] Todor Markov, Chong Zhang, Sandhini Agarwal, Tyna Eloundou, Teddy Lee, Steven Adler, Angela Jiang, and Lilian Weng. A Holistic Approach to Undesired Content Detection in the Real World. CoRR abs/208.03274, 2022. 1, 2, 12 [49] Leland McInnes, John Healy, Nathaniel Saul, and Lukas… find our card
18. IEEE, 2011. 6, 17 [51] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 11048–11064. find
19. ACL, 2022. 18 [52] Jaron Mink, Licheng Luo, Natã M. Barbosa, Olivia Figueira, Yang Wang, and Gang Wang. DeepPhish: Understanding User Trust Towards Artificially Generated Profiles in Online Social Networks. In USENIX Security Symposium (USENIX Security), pages 1669–1686. find
20. USENIX, 2022. 13 [53] Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg-Kirkpatrick, and Reza Shokri. Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8332–8347. find
21. ACL, 2022. 13 [54] Alexandros Mittos, Savvas Zannettou, Jeremy Blackburn, and Emiliano De Cristofaro. “And We Will Fight For Our Race!” A Measurement Study of Genetic Testing Conversations on Reddit and 4chan. In International Conference on Web and Social Media (ICWSM), pages 452–463. find
22. AAAI, 2020. 3 [55] Mark E. J. Newman. Modularity and community structure in networks. Proceedings of the National Academy of Sciences, 2006. 17 [56] NIST. AI Risk Management Framework. https://www. nist.gov/itl/ai-risk-management-framework. 3, 13 [57] NVIDIA. NeMo-Guardrails. https://github.com/ NVI… ↗
23. [30] Yunhe Feng, Pradhyumna Poralla, Swagatika Dash, Kaicheng Li, Vrushabh Desai, and Meikang Qiu. The Impact of Chat-GPT on Streaming Media: A Crowdsourced and Data-Driven Analysis using Twitter and Reddit. In IEEE International Conference on Big Data Security on Cloud, High Performance and Smart C… find
24. [31] FlowGPT. Paraphrase a text. https://flowgpt.com/p/ paraphrase-a-text. 11 [32] Google. AI ACROSS GOOGLE: PaLM 2. https://ai. google/discover/palm2/. 1, 8 [33] Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. More than you’ve asked for: A Comprehen… ↗ our card
25. [34] Julian Hazell. Large Language Models Can Be Used To Effectively Scale Spear Phishing Campaigns. CoRR abs/2305.06972, 2023. 1, 3, 13 — [arXiv:2305.06972](https://arxiv.org/abs/2305.06972) find
26. [35] Xinlei He, Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. MGTBench: Benchmarking Machine-Generated Text Detection. CoRR abs/2303.14822, 2023. 13 — [arXiv:2303.14822](https://arxiv.org/abs/2303.14822) find
27. [36] Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. Adversarial Example Generation with Syntactically Controlled Paraphrase Networks. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 1875–1885… find
28. [37] Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. Is ChatGPT A Good Translator? A Preliminary Study. CoRR abs/2301.08745, 2023. 3 find our card
29. [38] Jigsaw. Perspective API. https://www.perspectiveapi. com. 9 [39] Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment. In AAAI Conference on Artificial Intelligence (AAAI), pages 80… ↗ our card
30. [40] Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto. Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks. CoRR abs/2302.05733, 2023. 1, 2, 3, 13 — [arXiv:2302.05733](https://arxiv.org/abs/2302.05733) find
31. [41] John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum, and Tom Goldstein. On the Reliability of Watermarks for Large Language Models. CoRR abs/2306.04634, 2023. 11 find
32. [42] Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, and Yangqiu Song. Multi-step Jailbreaking Privacy Attacks on ChatGPT. CoRR abs/2304.05197, 2023. 12, 13 find our card
33. [43] Xuezixiang Li, Yu Qu, and Heng Yin. PalmTree: Learning an Assembly Language Model for Instruction Embedding. In ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 3236–3251. ACM, 2021. 3 find
34. [44] Zilong Lin, Jian Cui, Xiaojing Liao, and XiaoFeng Wang. Malla: Demystifying Real-world Large Language Model Integrated Malicious Services. CoRR abs/2401.03315, 2024. 2, 13 — [arXiv:2401.03315](https://arxiv.org/abs/2401.03315) find
35. [45] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, Prompt, and [62] OpenAI. Pricing. https://openai.com/pricing. 5 [77] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Chri… ↗
36. [63] OpenAI. Usage policies. https://openai.com/policies/ usage-policies. 2, 8, 20 [78] The White House. Blueprint for an AI Bill of Rights. https: //www.whitehouse.gov/ostp/ai-bill-of-rights/. 1, 3 [64] OpenAI. GPT-4 Technical Report. CoRR abs/2303.08774, 2023. 1, 3, 8, 17 ↗ our card
37. [79] Jörg Tiedemann and Santhosh Thottingal. OPUS-MT - Building open translation services for the World. In Conference of the European Association for Machine Translation (EAMT), pages 479–480. European Association for Machine Translation, 2020. 11 find
38. [61] OpenAI. New models and developer products announced at DevDay. https://openai.com/blog/new-models-and-developer-products-announced-at-devday. 5 [65] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray,… ↗
39. [80] Together. OpenChatKit. https://github.com/ togethercomputer/OpenChatKit. 1, 2, 12 [81] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave… ↗
40. [66] Alessandro Pegoraro, Kavita Kumari, Hossein Fereidooni, and Ahmad-Reza Sadeghi. To ChatGPT, or not to ChatGPT: That is the question! CoRR abs/2304.01487, 2023. 3 — [arXiv:2304.01487](https://arxiv.org/abs/2304.01487) find
41. [82] Florian Tramèr, Reza Shokri, Ayrton San Joaquin, Hoang Le, Matthew Jagielski, Sanghyun Hong, and Nicholas Carlini. Truth Serum: Poisoning Machine Learning Models to Reveal Their Secrets. In ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2022. 13 find
42. [67] Kexin Pei, David Bieber, Kensen Shi, Charles Sutton, and Pengcheng Yin. Can Large Language Models Reason about Program Invariants? In International Conference on Machine Learning (ICML). JMLR, 2023. 3 find
43. [83] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Annual Conference on Neural Information Processing Systems (NIPS), pages 5998–6008. NIPS, 2017. 3 find
44. [68] Fábio Perez and Ian Ribeiro. Ignore Previous Prompt: Attack Techniques For Language Models. CoRR abs/2211.09527, 2022. 13 find our card
45. [69] Yiting Qu, Xinyue Shen, Xinlei He, Michael Backes, Savvas Zannettou, and Yang Zhang. Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Models. In ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2023. 1, 3, 13 find
46. [84] Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang T. Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, and Bo Li. DecodingTrust: A Comprehensive Assessment of… find our card
47. [70] Reddit. r/ChatGPTJailbreak. https://www.reddit.com/r/ ChatGPTJailbreak/. 1, 3 [71] Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Conference on Empirical Methods in Natural Language Processing and International Joint Conference on Natural Lan… ↗
48. [85] Zijie J. Wang, Fred Hohman, and Duen Horng Chau. Wiz-Map: Scalable Interactive Visualization for Exploring Large Machine Learning Embeddings. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 516– 523. ACL, 2023. 5 find
49. [72] Marco Túlio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 4902–4912. ACL, 2020. 11 find
50. [86] Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How Does LLM Safety Training Fail? CoRR abs/2307.02483, 2023. 8, 12 find our card
51. [87] Wikipedia. Spearman’s rank correlation coefficient. https://en.wikipedia.org/wiki/Spearman’s\_rank\_ correlation\_coefficient. 10 [73] Caitlin M. Rivers and Bryan L. Lewis. Ethical research standards in a world of big data. F1000Research, 2014. 3 ↗
52. [74] Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang. On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 4454–4470. ACL, 2023. 8 find our card
53. [88] Xiaojun Xu, Qi Wang, Huichen Li, Nikita Borisov, Carl A. Gunter, and Bo Li. Detecting AI Trojans Using Meta Neural Analysis. In IEEE Symposium on Security and Privacy (S&P). IEEE, 2021. 13 find
54. [89] Zhiyuan Yu, Yuhao Wu, Ning Zhang, Chenguang Wang, Yevgeniy Vorobeychik, and Chaowei Xiao. CodeIPPrompt: Intellectual Property Infringement Assessment of Code Language Models. In International Conference on Machine Learning (ICML). JMLR, 2023. 13 find
55. [75] Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. In ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPT. CoRR abs/2304.08979, 2023. 13 — [arXiv:2304.08979](https://arxiv.org/abs/2304.08979) find
56. [76] Wai Man Si, Michael Backes, Jeremy Blackburn, Emiliano De Cristofaro, Gianluca Stringhini, Savvas Zannettou, and Yang Zhang. Why So Toxic? Measuring and Triggering Toxic Behavior in Open-Domain Chatbots. In ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 2659–2673. AC… find
57. [90] Savvas Zannettou, Joel Finkelstein, Barry Bradlyn, and Jeremy Blackburn. A Quantitative Approach to Understanding Online Antisemitism. In International Conference on Web and Social Media (ICWSM), pages 786–797. AAAI, 2020. 6 *Table 9: Comparison of community detection methods.* [91] Aohan Zeng,… find our card
58. Louvain Greedy Modularity LPC Girvan Newman [92] Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs. CoRR abs/2401.06373, 2024. 2 find
59. A.1 \\\# Communities 0.851 0.843 0.845 0.803 131 130 140 127 LLMs find