Elamur Research
☰
home / papers / 2023 / Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection

Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2302.12173

Summary

Earlier work on prompt injection assumed that the attacker is the user, typing malicious instructions straight into the model. This paper shows that this assumption no longer holds once LLMs are built into applications that pull in outside content such as web pages, emails, documents or code. Any retrieved text can be read by the model as instructions, so a third party can steer it without direct access. The authors call this Indirect Prompt Injection and frame it in security terms: ingesting untrusted retrieved data works much like running arbitrary code.

They build a taxonomy of delivery routes, victims and attack outcomes. They then write concrete attack prompts and test them on real products, including Bing's GPT-4-based Chat and code-completion engines, and on GPT-4 applications they built themselves. The injected prompts can hijack application behaviour, decide whether and how external APIs are called, leak data and propagate further. Effective mitigations are still missing, so the work is an early warning for teams deploying retrieval-augmented or tool-using LLMs and autonomous agents.

Key points

  • Threat model: adversaries plant prompts in sources likely to be fetched at inference time. They need no ML expertise, no model access and little cost, which makes the attack cheaper than backdoor or model-hijacking attacks.
  • Taxonomy:
    • Injection methods: passive (via retrieval), active (e.g., emails), user-driven and hidden.
    • Affected parties: end-users, developers, automated systems and the LLM's own availability.
    • Threat classes: information gathering, fraud, intrusion, malware, manipulated content and availability.
  • Concrete risks named:
    • theft of personal data, credentials and chat history;
    • phishing, scams and masquerading;
    • persistent compromise and remote control of the model;
    • self-spreading "prompt worms";
    • disinformation, biased or hidden content and covert ads;
    • denial of service and wasted computation.
  • Targets tested: real systems (Bing Chat on GPT-4, code-completion engines) and synthetic GPT-4 applications. Jailbreaking GPT-4 remains feasible.
  • Released resources: demos are on GitHub (greshake/llm-security), and every attack prompt is listed in the paper's appendix as a basis for security testing of LLM-integrated apps.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

175 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 164 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2302.12173 · arXiv 2302.12173

@misc{greshake2023not,
  title = {{Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection}},
  author = {Greshake, Kai and Abdelnabi, Sahar and Mishra, Shailesh and Endres, Christoph and Holz, Thorsten and Fritz, Mario},
  year = {2023},
  doi = {10.48550/arxiv.2302.12173},
  eprint = {2302.12173},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2302.12173}
}

References

80 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 2023. Australian mayor readies world’s first defamation lawsuit over ChatGPT content. [Link]. find
  2. 2023. AUTO-GPT VS CHATGPT: HOW DO THEY DIFFER AND EVERYTHING YOU NEED TO KNOW. [Link]. find
  3. 2023. Building the New Bing. [Link]. find
  4. 2023. ChatGPT banned in Italy over privacy concerns. [Link]. find
  5. 2023. ChatGPT bug leaked users’ conversation histories. [Link]. find
  6. 2023. ChatGPT invented a sexual harassment scandal and named a real law prof as the accused. [Link]. find
  7. 2023. ChatGPT Plugins. [Link]. find
  8. 2023. ChatGPT sets record for fastest-growing user base - analyst note. [Link]. find
  9. 2023. Confirmed: the new Bing runs on OpenAI’s GPT-4. [Link]. find
  10. 2023. A Conversation With Bing’s Chatbot Left Me Deeply Unsettled. [Link]. find
  11. 2023. Copilot Internals. [Link]. find
  12. 2023. Driving more traffic and value to publishers from the new Bing. [Link]. find
  13. 2023. GitHub Copilot - Your AI pair programmer. [Link]. find
  14. 2023. Google and Microsoft’s chatbots are already citing one another in a misinformation shitshow. [Link]. find
  15. 2023. Google’s AI chatbot Bard makes factual error in first demo. [Link]. find
  16. 2023. How to Jailbreak ChatGPT. [Link]. find
  17. 2023. Introducing Microsoft 365 Copilot – your copilot for work. [Link]. find
  18. 2023. Introducing Microsoft Security Copilot. [Link]. find
  19. 2023. Jailbreak Chat. [Link]. find
  20. 2023. LangChain library for composing and integrating LLMs into applications. [Link]. find
  21. 2023. The LLaMA is out of the bag. Should we expect a tidal wave of disinformation? [Link]. find
  22. 2023. Microsoft limits Bing chat to five replies. [Link]. find
  23. 2023. Microsoft’s AI chatbot is going off the rails. [Link]. find
  24. 2023. Microsoft’s Bing A.I. made several factual errors in last week’s launch demo. [Link]. find
  25. 2023. Microsoft’s Bing chatbot gets smarter with restaurant bookings, image results, and more. [Link]. find
  26. 2023. The New Bing and Edge – Progress from Our First Month. [Link]. find
  27. 2023. New prompt injection attack on ChatGPT web version. Reckless copy-pasting may lead to serious privacy issues in your chat. [Link]. find
  28. 2023. OpenAI Codex. [Link]. find
  29. 2023. Reinventing search with a new AI-powered Microsoft Bing and Edge, your copilot for the web. [Link]. find
  30. 2023. That was fast! Microsoft slips ads into AI-powered Bing Chat. [Link]. find
  31. 2023. These are Microsoft’s Bing AI secret rules and why it says it’s named Sydney. [Link]. find
  32. Jacob Andreas. 2022. Language models as agent models. In Findings of EMNLP. find
  33. Giovanni Apruzzese, Hyrum Anderson, Savino Dambra, David Freeman, Fabio Pierazzi, and Kevin Roundy. 2022. Position:“Real Attackers Don’t Compute Gradients”: Bridging the Gap Between Adversarial ML Research and Practice. In SaTML. find
  34. Eugene Bagdasaryan and Vitaly Shmatikov. 2022. Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasures. In S&P. find
  35. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv (2022). find
  36. Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. 2023. Eliciting Latent Predictions from Transformers with the Tuned Lens. arXiv (2023). find our card
  37. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. In the ACM conference on Fairness, Accountability, and Transparency. find
  38. Daniil A Boiko, Robert MacKnight, and Gabe Gomes. 2023. Emergent autonomous scientific research capabilities of large language models. arXiv (2023). find
  39. Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv (2021). find our card
  40. Nicholas Boucher, Ilia Shumailov, Ross Anderson, and Nicolas Papernot. 2022. Bad characters: Imperceptible nlp attacks. In S&P. find
  41. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In NeurIPS. find
  42. Matthew Burtell and Thomas Woodside. 2023. Artificial Influence: An Analysis Of AI-Driven Persuasion. arXiv (2023). find
  43. Clarence Chio and David Freeman. 2018. Machine learning and security. "O’Reilly Media, Inc.". find
  44. Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. 2023. Toxicity in ChatGPT: Analyzing Persona-assigned Language Models. arXiv (2023). find our card
  45. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. In Findings of EMNLP. find our card
  46. Josh A Goldstein, Girish Sastry, Micah Musser, Renee DiResta, Matthew Gentzel, and Katerina Sedova. 2023. Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigations. arXiv (2023). find
  47. Jochen Hartmann, Jasper Schwenzow, and Maximilian Witte. 2023. The political ideology of conversational AI: Converging evidence on ChatGPT’s pro-environmental, left-libertarian orientation. arXiv (2023). find
  48. Noah Goodman Jesse Mu, Xiang Lisa Li. 2023. Learning to Compress Prompts with Gist Tokens. arXiv (2023). find our card
  49. Ana Jojic, Zhen Wang, and Nebojsa Jojic. 2023. GPT is becoming a Turing machine: Here are some ways to program it. arXiv (2023). find
  50. Keith S Jones, Miriam E Armstrong, McKenna K Tornblad, and Akbar Siami Namin. 2021. How social engineers use persuasion principles during vishing attacks. Information & Computer Security 29, 2 (2021), 314–331. find
  51. Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto. 2023. Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks. arXiv (2023). find
  52. Sebastian Krügel, Andreas Ostermaier, and Matthias Uhl. 2023. ChatGPT’s inconsistent moral advice influences users’ judgment. Scientific Reports 13, 1 (2023), 4569. find
  53. Mina Lee, Megha Srivastava, Amelia Hardy, John Thickstun, Esin Durmus, Ashwin Paranjape, Ines Gerard-Ursin, Xiang Lisa Li, Faisal Ladhak, Frieda Rong, et al. 2022. Evaluating Human-Language Model Interaction. arXiv (2022). find
  54. Yaobo Liang, Chenfei Wu, Ting Song, Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuai Lu, Lei Ji, Shaoguang Mao, et al. 2023. TaskMatrix. AI: Completing Tasks by Connecting Foundation Models with Millions of APIs. arXiv (2023). find
  55. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring How Models Mimic Human Falsehoods. In ACL. find our card
  56. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruction Tuning. arXiv (2023). find our card
  57. Nelson F Liu, Tianyi Zhang, and Percy Liang. 2023. Evaluating Verifiability in Generative Search Engines. arXiv (2023). find our card
  58. Microsoft. 2023. Bing Preview Release Notes: Bing in the Edge Sidebar. [Link]. find
  59. Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereotypical bias in pretrained language models. In ACL \| IJCNLP. find our card
  60. OpenAI. 2023. GPT-4 Technical Report. arXiv (2023). find our card
  61. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, et al. 2022. Training language models to follow instructions with human feedback. In NeurIPS. find
  62. Joon Sung Park, Joseph C O’Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative Agents: Interactive Simulacra of Human Behavior. arXiv (2023). find
  63. Roma Patel and Ellie Pavlick. 2021. “Was it “stated” or was it “claimed”?: How linguistic bias affects generative language models. In EMNLP. find
  64. Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. 2022. Discovering Language Model Behaviors with Model-Written Evaluations. arXiv (2022). find our card
  65. Fábio Perez and Ian Ribeiro. 2022. Ignore Previous Prompt: Attack Techniques For Language Models. In NeurIPS ML Safety Workshop. find our card
  66. Tianyi Zhang Yann Dubois Xuechen Li Carlos Guestrin Percy Liang Tatsunori B. Hashimoto Rohan Taori, Ishaan Gulrajani. 2023. Alpaca: A Strong, Replicable Instruction-Following Model. [Link]. find
  67. Ahmed Salem, Michael Backes, and Yang Zhang. 2022. Get a Model! Model Hijacking Attack Against Machine Learning Models. In NDSS. find
  68. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose Opinions Do Language Models Reflect? arXiv (2023). find our card
  69. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. arXiv (2023). find our card
  70. Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace. arXiv (2023). find
  71. Arabella Sinclair, Jaap Jumelet, Willem Zuidema, and Raquel Fernández. 2022. Structural Persistence in Language Models: Priming as a Window into Abstract Language Representations. Transactions of the Association for Computational Linguistics 10 (09 2022), 1031–1050. find
  72. Jacob Steinhardt. 2023. Emergent Deception and Emergent Optimization. [Link]. find
  73. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. In NeurIPS. find
  74. Jonas Thiergart, Stefan Huber, and Thomas Übellacker. 2021. Understanding emails and drafting responses–An approach using GPT-3. arXiv (2021). find
  75. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In NeurIPS. find our card
  76. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021. Ethical and social risks of harm from language models. arXiv (2021). find our card
  77. Shunyu Yao, Jeffrey Zhao, Dian Yu, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In ICLR. find our card
  78. Yoav Levine Amnon Shashua Yotam Wolf, Noam Wies. 2023. Fundamental Limitations of Alignment in Large Language Models. arXiv (2023). find
  79. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023. Large Language Models are Human-Level Prompt Engineers. In ICLR. find our card
  80. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. arXiv (2023). find our card