Elamur Research⌕
☰
home / papers / 2024 / How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs

How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs

Summary

Most jailbreak research treats LLMs as algorithmic systems, attacking them with optimized gibberish suffixes, ciphers or role-play templates. This paper asks a different question: can ordinary persuasion, the kind non-experts use daily, talk a model into unsafe behavior? The authors draw on decades of social science work to build a taxonomy of persuasion techniques. They then train Persuasive Paraphrasers that rewrite plain harmful queries into fluent, human-readable persuasive adversarial prompts (PAPs).

A broad scan across 14 risk categories measures how well each technique works. A second, iteratively applied paraphraser is fine-tuned on the PAPs that succeeded. In 10 trials, this approach jailbreaks Llama-2-7b-Chat, GPT-3.5 and GPT-4 in over 92% of cases, beating algorithm-focused baselines without any specialized optimization. Stronger models turned out to be more vulnerable to persuasion and harder to protect with standard defenses. Existing post-hoc defenses leave a clear gap.

The work matters because it identifies a realistic, low-cost risk from everyday users. It also argues that AI safety needs more fundamental fixes.

Key points

  • Taxonomy: 15 high-level strategies (ethical and unethical) cover 40 fine-grained techniques, such as Evidence-based Persuasion, Expert Endorsement, Emotional Appeal and Foot-in-the-door.
  • Results: PAP reaches an attack success rate above 92% on Llama-2-7b-Chat, GPT-3.5 and GPT-4. Claude was reported to resist PAPs.
  • Comparison: The authors frame prior attacks such as GCG, PAIR, GPTFuzzer and low-resource-language attacks as lower levels of humanizing LLMs, with PAP at the highest level.
  • Defenses: Current post-hoc defenses fall short against PAPs. The authors propose adaptive defenses, including adaptive safety system prompts and a summarization-based method, which also help against other attacks.
  • Release: Findings were shared with Meta and OpenAI. Only the taxonomy is public, at github.com/CHATS-lab/persuasive_jailbreaker. Jailbreak data is available to researchers on request, and the trained paraphraser is withheld.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

100 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 97 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.acl-long.773

@inproceedings{zeng2024how,
  title = {{How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs}},
  author = {Zeng, Yi and Lin, Hongpeng and Zhang, Jingwen and Yang, Diyi and Jia, Ruoxi and Shi, Weiyan},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.773},
  url = {https://doi.org/10.18653/v1/2024.acl-long.773}
}

References

70 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Praveen Aggarwal, Sung Youl Jun, and Jong Ho Huh. 2011. Scarcity messages. Journal of Advertising, 40(3):19–30. find
  2. Gabriel Alon and Michael Kamfonas. 2023. Detecting language model attacks with perplexity. arXiv preprint arXiv:2308.14132. find our card
  3. Anthropic. 2023. Model card and evaluations for claude models. find
  4. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073. find our card
  5. Karen Bartsch, Jennifer Cole Wright, and David Estes. 2010. Young children’s persuasion in everyday conversation: Tactics and attunement to others’ mental states. Social Development, 19(2):394–416. find
  6. Helena Bilandzic and Rick Busselle. 2013. Narrative persuasion. The SAGE handbook of persuasion: Developments in theory and practice, pages 200–219. find
  7. Ted Brader. 2005. Striking a responsive chord: How political ads motivate and persuade voters by appealing to emotions. American Journal of Political Science, 49(2):388–405. find
  8. Judee K Burgoon, Leesa Dillman, and Lesa A Stem. 1993. Adaptation in dyadic interaction: Defining and operationalizing patterns of reciprocity and compensation. Communication Theory, 3(4):295–316. find
  9. Bochuan Cao, Yuanpu Cao, Lu Lin, and Jinghui Chen. 2023. Defending against alignment-breaking attacks via robustly aligned llm. arXiv preprint arXiv:2309.14348. find
  10. Robert B Cialdini and Noah J Goldstein. 2004. Social influence: Compliance and conformity. Annu. Rev. Psychol., 55:591–621. find
  11. Gary Lynn Cronkhite. 1964. Logic, emotion, and the paradigm of persuasion. Quarterly Journal of Speech, 50(1):13–18. find
  12. Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu. 2023a. Jailbreaker: Automated jailbreak across multiple large language model chatbots. arXiv preprint arXiv:2307.08715. — [arXiv:2307.08715](https://arxiv.org/abs/2307.08715) find
  13. Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. 2023b. Multilingual jailbreak challenges in large language models. arXiv preprint arXiv:2310.06474. find our card
  14. Nicholas DiFonzo and Prashant Bordia. 2011. Rumors influence: Toward a dynamic social impact theory of rumor. In The science of social influence, pages 271–295. Psychology Press. find
  15. James Price Dillard and Leanne K Knobloch. 2011. Interpersonal influence. The Sage handbook of interpersonal communication, pages 389–422. find
  16. Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, et al. 2023. What’s in my big data? arXiv preprint arXiv:2310.20707. find
  17. Robert H Gass and John S Seiter. 2022. Persuasion: Social influence and compliance gaining. Routledge. find
  18. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462. find our card
  19. Erving Goffman. 1974. Frame analysis: An essay on the organization of experience. Harvard University Press. find
  20. Nicholas Carlini, Milad Nasr, Christopher A Choquette-Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramer, et al. 2023. Are aligned neural networks adversarially aligned? arXiv preprint arXiv:2306.15447. find
  21. Lewis Griffin, Bennett Kleinberg, Maximilian Mozes, Kimberly Mai, Maria Do Mar Vau, Matthew Caldwell, and Augustine Mavor-Parker. 2023a. Large language models respond to influence like humans. In Proceedings of the First Workshop on Social Influence in Conversations (SICon 2023), pages 15–24. find
  22. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. 2023. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419. find our card
  23. Lewis D Griffin, Bennett Kleinberg, Maximilian Mozes, Kimberly T Mai, Maria Vau, Matthew Caldwell, and Augustine Marvor-Parker. 2023b. Susceptibility to influence of large language models. arXiv preprint arXiv:2303.06074. find
  24. Jiaao Chen and Diyi Yang. 2021. Weakly-supervised hierarchical models for predicting persuasive strategies in good-faith textual requests. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 12648–12656. find
  25. Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. 2021. Gradient-based adversarial attacks against text transformers. arXiv preprint arXiv:2104.13733. find our card
  26. Robert B Cialdini. 2001. The science of persuasion. Scientific American, 284(2):76–81. find
  27. Keise Izuma. 2013. The neural basis of social influence and attitude change. Current opinion in neurobiology, 23(3):456–462. 14333 12 find
  28. Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. 2023. Baseline defenses for adversarial attacks against aligned language models. arXiv preprint arXiv:2309.00614. find our card
  29. Richard L Johannesen and C Larson. 1989. Perspectives on ethics in persuasion. Persuasion: Reception and responsibility, pages 39–70. find
  30. Erik Jones, Anca Dragan, Aditi Raghunathan, and Jacob Steinhardt. 2023. Automatically auditing large language models via discrete optimization. arXiv preprint arXiv:2303.04381. find
  31. Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto. 2023. Exploiting programmatic behavior of llms: Dual-use through standard security attacks. arXiv preprint arXiv:2302.05733. — [arXiv:2302.05733](https://arxiv.org/abs/2302.05733) find
  32. James M Olson and Mark P Zanna. 1990. Self-inference processes: The ontario symposium, vol. 6. In This volume consists of expanded versions of papers originally presented at the Sixth Ontario Symposium on Personality and Social Psychology held at the University of Western Ontario, Jun 4-5, 1988. Law… find
  33. OpenAI. 2023. Gpt-4 technical report. find our card
  34. Daniel J O’keefe. 2018. Persuasion. In The Handbook of Communication Skills, pages 319–335. Routledge. find
  35. Richard M.. Perloff. 2017. The Dynamics of Persuasion: Communication and Attitudes in the 21st Century. Routledge. find
  36. Richard E Petty, Leandre R Fabrigar, and Duane T Wegener. 2003. Emotional factors in attitudes and persuasion. Handbook of affective sciences, 752:772. find
  37. Aounon Kumar, Chirag Agarwal, Suraj Srinivas, Soheil Feizi, and Hima Lakkaraju. 2023. Certifying llm safety against adversarial prompting. arXiv preprint arXiv:2309.02705. find our card
  38. Chanthika Pornpitakpan. 2004. The persuasiveness of source credibility: A critical review of five decades’ evidence. Journal of applied social psychology, 34(2):243–281. find
  39. Raz Lapid, Ron Langberg, and Moshe Sipper. 2023. Open sesame! universal black box jailbreaking of large language models. arXiv preprint arXiv:2309.01446. — [arXiv:2309.01446](https://arxiv.org/abs/2309.01446) find
  40. Penny Powers. 2007. Persuasion and coercion: a critical review of philosophical and empirical approaches. HEC F., 19:125. find
  41. Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. 2023. Rain: Your language models can align themselves without finetuning. arXiv preprint arXiv:2309.07124. — [arXiv:2309.07124](https://arxiv.org/abs/2309.07124) find
  42. Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2023. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451. find our card
  43. Jamie Luguri and Lior Jacob Strahilevitz. 2021. Shining a light on dark patterns. Journal of Legal Analysis, 13(1):43–109. find
  44. Arunesh Mathur, Gunes Acar, Michael J Friedman, Eli Lucherini, Jonathan Mayer, Marshini Chetty, and Arvind Narayanan. 2019. Dark patterns at scale: Findings from a crawl of 11k shopping websites. Proceedings of the ACM on Human-Computer Interaction, 3(CSCW):1–32. find
  45. Maximilian Mozes, Xuanli He, Bennett Kleinberg, and Lewis D Griffin. 2023. Use of llms for illicit purposes: Threats, prevention measures, and vulnerabilities. arXiv preprint arXiv:2308.12833. — [arXiv:2308.12833](https://arxiv.org/abs/2308.12833) find
  46. Arvind Narayanan, Arunesh Mathur, Marshini Chetty, and Mihir Kshirsagar. 2020. Dark patterns: Past, present, and future: The evolution of tricky user interfaces. Queue, 18(2):67–92. find
  47. Daniel O’Keefe. 2016. Evidence-based advertising using persuasion principles: Predictive validity and proof of concept. European Journal of Marketing, 50(1/2):294–300. find
  48. Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2023. Fine-tuning aligned language models compromises safety, even when users do not intend to! find our card
  49. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. find
  50. Soo Young Rieh and David R Danielson. 2007. Credibility: A multidisciplinary framework. find
  51. Alexander Robey, Eric Wong, Hamed Hassani, and George J Pappas. 2023. Smoothllm: Defending large language models against jailbreaking attacks. arXiv preprint arXiv:2310.03684. find our card
  52. Sander Schulhoff, Jeremy Pinto, Anaum Khan, Louis-François Bouchard, Chenglei Si, Svetlina Anati, Valen Tagliabue, Anson Kost, Christopher Carnahan, and Jordan Boyd-Graber. 2023. Ignore this title and hackaprompt: Exposing systemic vulnerabilities of llms through a global prompt hacking competition.… find
  53. Rusheb Shah, Quentin Feuillade-Montixi, Soroush Pour, Arush Tagade, Stephen Casper, and Javier Rando. 2023. Scalable and transferable black-box jailbreaks for language models via persona modulation. arXiv preprint arXiv:2311.03348. 14334 13 — [arXiv:2311.03348](https://arxiv.org/abs/2311.03348) find
  54. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/ stanford_alpaca. ↗
  55. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. find
  56. Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. 2023. Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher. arXiv preprint arXiv:2308.06463. find our card
  57. Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043. find our card
  58. Alex Wang. 2005. The effects of expert and consumer endorsements on audience response. Journal of advertising research, 45(4):402–412. find
  59. Jiongxiao Wang, Zichen Liu, Keun Hee Park, Muhao Chen, and Chaowei Xiao. 2023. Adversarial demonstration attacks on large language models. arXiv preprint arXiv:2305.14950. — [arXiv:2305.14950](https://arxiv.org/abs/2305.14950) find
  60. Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu. 2019. Persuasion for good: Towards a personalized persuasive dialogue system for social good. arXiv preprint arXiv:1906.06725. find our card
  61. Zeming Wei, Yifei Wang, and Yisen Wang. 2023. Jailbreak and guard aligned language models with only few in-context demonstrations. arXiv preprint arXiv:2310.06387. find our card
  62. Timothy D Wilson, JC Olson, and MP Zanna. 2013. Self-persuasion via self-reflection. In Self-Inference Processes: The Ontario Symposium, J. Olson, M. Zanna, Eds.(Erlbaum, Hillsdale, NJ, 1990), volume 6, pages 43–67. find
  63. Arch G Woodside, Suresh Sood, and Kenneth E Miller. 2008. When consumers and brands talk: Storytelling theory and research in psychology and marketing. Psychology & Marketing, 25(2):97–145. find
  64. Chloe Xiang. 2023. “he would still be here”: Man dies by suicide after talking with ai chatbot, widow says. find
  65. Rongwu Xu, Brian Lin, Shujian Yang, Tianqi Zhang, Weiyan Shi, Tianwei Zhang, Zhixuan Fang, Wei Xu, and Han Qiu. 2023. The earth is flat because...: Investigating llms’ belief towards misinformation via persuasive conversation. arXiv preprint arXiv:2312.09085. find
  66. Zheng-Xin Yong, Cristina Menghini, and Stephen H Bach. 2023. Low-resource languages jailbreak gpt-4. arXiv preprint arXiv:2310.02446. find our card
  67. Jiahao Yu, Xingwei Lin, and Xinyu Xing. 2023. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts. arXiv preprint arXiv:2309.10253. 14335 14 find our card
  68. Persuasion Technique Mapping Persuasion Technique Mapping Idx. Reference 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. A B, C C, D, F E, F F G G G G G, H I I I I C, J C, G, I C, G, I C, I C, G, I I, K 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35. 36. 37. 38. 39… find
  69. O’Keefe (2016) Cronkhite (1964) Perloff (2017) Pornpitakpan (2004) Wang (2005) Rieh and Danielson (2007) Cialdini and Goldstein (2004) Cialdini (2001) Dillard and Knobloch (2011) Brader (2005) Petty et al. (2003) Woodside et al. (2008) Bilandzic and Busselle (2013) Burgoon et al. (1993) Aggarwal et… find
  70. Evidence-based Persuasion Logical Appeal Expert Endorsement Non-expert Testimonial Authority Endorsement Social Proof Injunctive Norm Foot-in-the-door Commitment Door-in-the-face Commitment Public Commitment Alliance Building Complimenting Shared Values Relationship Leverage Loyalty Appeals Favor Ne… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zeng-2024-how-johnny-can-persuade-llms/