Elamur Research
☰
home / papers / 2023 / Prompt Injection attack against LLM-integrated Applications

Prompt Injection attack against LLM-integrated Applications

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2306.05499

Summary

The paper examines prompt injection, where a user's crafted input overrides an LLM-integrated application's built-in instructions. A pilot study on ten commercial black-box services showed that published injection tricks rarely work there. The authors found three causes: applications treat user text as data rather than as a question, enforce input/output formats, or run multi-step pipelines with response timeouts.

Drawing on SQL injection and XSS, they built HOUYI, a black-box attack. HOUYI uses an LLM to infer what the target application does, then assembles a payload from three parts. A Framework Component blends in with the app's normal use, a Separator Component breaks the context away from the preset prompt, and a Disruptor Component carries the attacker's real request. A feedback loop refines failed payloads.

Applied to 36 real applications, the resulting toolkit compromised 31 of them. Successful attacks either stole the app's hidden prompt or used its LLM freely at the provider's expense. This matters because such services, some with millions of users, could be cloned or abused cheaply, and common defenses did not stop the attack.

Key points

  • Pilot study: existing techniques achieved only partial success on 2 of 10 commercial apps.
  • Attack workflow: HOUYI runs in three phases, Context Inference, Payload Generation and Feedback. Separators are built in three ways: syntax-based, language switching and semantic-based.
  • Results: 31 of 36 applications were vulnerable, an 86.1% success rate. 10 vendors, including Notion, confirmed the findings.
  • Two exploit scenarios:
    • prompt leak, i.e. theft of the application's prompt, allowing zero-cost imitation;
    • prompt abuse, i.e. unrestricted use of the backend LLM.
  • Defenses: standard countermeasures applied to open-source projects were bypassed. These included XML+escape wrapping of user input and Separate LLM Evaluation. The authors note that some vulnerabilities may stop being reproducible as backend models evolve.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

113 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 108 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2306.05499 · arXiv 2306.05499

@misc{zhang2023prompt,
  title = {{Prompt Injection attack against LLM-integrated Applications}},
  author = {Zhang;, Yi Liu; Gelei Deng; Yuekang Li; Kailong Wang; Zihao Wang; Xiaofeng Wang; Tianwei Zhang; Yepang Liu; Haoyu Wang; Yan Zheng; Leo Yu and Liu, Yang},
  year = {2023},
  doi = {10.48550/arxiv.2306.05499},
  eprint = {2306.05499},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.05499}
}

References

46 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. [16] Exploring Prompt Injection Attacks - NCC Group Research Blog. https://research.nccgroup. com/2022/12/05/exploring-prompt-injection-attacks/, Apr 2023. ↗
  2. [1] Notion. https://www.notion.so/. [2] Parea AI. https://www.parea.ai/. [3] Supertools \| Best AI Tools Guide. supertools.therundown.ai/. [17] Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. I… ↗ our card
  3. [4] Prompt Injection Attacks against GPT-3. https: //simonwillison.net/2022/Sep/12/prompt-injection/. find
  4. [18] Google AI. PaLM 2. https://ai.google/discover/ palm2/. [5] Rate Limits OpenAI API. https://platform. openai.com/docs/guides/rate-limits. [19] Significant Gravitas. Auto-GPT. https://github. com/Significant-Gravitas/Auto-GPT. [6] Giovanni Apruzzese, Hyrum S. Anderson, Savino Dambra, David Freema… ↗
  5. [20] Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. In arXiv preprint, 2023. find our card
  6. [7] Eugene Bagdasaryan and Vitaly Shmatikov. Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasures. In S&P, pages 769–786. IEEE, 2022. find
  7. [21] Haifeng Gu, Jianning Zhang, Tian Liu, Ming Hu, Junlong Zhou, Tongquan Wei, and Mingsong Chen. Diava: A traffic-based framework for detection of sql injection attacks and vulnerability analysis of leaked data. IEEE Transactions on Reliability, 69(1):188–202, 2020. find
  8. [8] Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? In FAccT, pages 610–623. [22] Prompt Engineering Guide. Defense Tactics. https: //www.promptingguide.ai/risks/adversarial. [9] Daniil A Boiko,… find
  9. [23] Shashank Gupta and Brij Bhooshan Gupta. Cross-Site Scripting (XSS) attacks and defense mechanisms: classification and state-of-the-art. Int. J. Syst. Assur. Eng. Manag., 8(1s):512–530, 2017. find
  10. [10] Stephen W Boyd and Angelos D Keromytis. SQLrand: Preventing SQL injection attacks. In ACNS, pages 292– 302, 2004. find
  11. [24] Emet GURL. Swot analysis: a theoretical review. 2017. find
  12. [25] William G Halfond, Jeremy Viegas, Alessandro Orso, et al. A classification of SQL-injection attacks and countermeasures. In ISSSR, volume 1, pages 13–15. IEEE, 2006. find
  13. [11] Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. Large Language Models as Tool Makers. arXiv preprint, 2023. find
  14. [12] Yuzhe Cai, Shaoguang Mao, Wenshan Wu, Zehua Wang, Yaobo Liang, Tao Ge, Chenfei Wu, Wang You, Ting Song, Yan Xia, et al. Low-code LLM: Visual Programming over LLMs. arXiv preprint, 2023. find
  15. [26] Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings. arXiv preprint, 2023. find our card
  16. [13] ChatAIWriter. Writesonic. https://app. writesonic.com/botsonic/780dc6b4-fbe9-4d5e-911c-014c9367ba32. [14] Justin Clarke. SQL injection attacks and defense. Elsevier, 2009. find
  17. [27] Isatou Hydara, Abu Bakar Md Sultan, Hazura Zulzalil, and Novia Admodisastro. Current state of research on cross-site scripting (XSS)–A systematic literature review. Information and Software Technology, 58:170– 186, 2015. find
  18. [15] Lavina Daryanani. How to Jailbreak ChatGPT. https://watcher.guru/news/how-to-jailbreak-chatgpt. [28] Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. arXiv preprint, 2023. ↗ our card
  19. [29] Minghao Li, Feifan Song, Bowen Yu, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A benchmark for tool-augmented llms. arXiv preprint, 2023. find
  20. [42] Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. ART: Automatic multi-step reasoning and tool-use for large language models. arXiv preprint, 2023. find
  21. [30] Yaobo Liang, Chenfei Wu, Ting Song, Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuai Lu, Lei Ji, Shaoguang Mao, et al. Taskmatrix. ai: Completing tasks by connecting foundation models with millions of apis. arXiv preprint, 2023. find
  22. [43] Joon Sung Park, Joseph C O’Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint, 2023. find
  23. [31] Shengchao Liu, Jiongxiao Wang, Yijin Yang, Chengpeng Wang, Ling Liu, Hongyu Guo, and Chaowei Xiao. ChatGPT-powered Conversational Drug Editing Using Retrieval and Domain Feedback. arXiv preprint, 2023. find
  24. [44] Fábio Perez and Ian Ribeiro. Ignore Previous Prompt: Attack Techniques For Language Models. In NeurIPS ML Safety Workshop, 2022. find our card
  25. [32] Xiaodong Liu, Hao Cheng, Pengcheng He, Weizhu Chen, Yu Wang, Hoifung Poon, and Jianfeng Gao. Adversarial Training for Large Neural Language Models. CoRR, abs/2004.08994, 2020. — [arXiv:2004.08994](https://arxiv.org/abs/2004.08994) find
  26. [45] Pricing. https://openai.com/pricing. Instruction Defense. [46] Learn Prompting. https://learnprompting.org/docs/prompt\_ hacking/defensive\_measures/instruction. [33] Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. Jailbreaking ChatGPT… ↗ our card
  27. [47] Learn Prompting. Instruction Defense. https://learnprompting.org/docs/prompt\_ hacking/defensive\_measures/post\_prompting. [48] Learn Prompting. Prompt Leaking. https: //learnprompting.org/docs/prompt\_hacking/ leaking. [34] Potsawee Manakul, Adian Liusie, and Mark JF Gales. Selfcheckgpt: Zero-res… ↗
  28. [49] Learn Prompting. Random Sequence Enclosure. https://learnprompting.org/docs/prompt\_ hacking/defensive\_measures/random\_sequence. [35] Nick McKenna, Tianyi Li, Liang Cheng, Mohammad Javad Hosseini, Mark Johnson, and Mark Steedman. Sources of Hallucination by Large Language Models on Inference Tas… ↗
  29. [50] Learn Prompting. Sandwich Defense. https: //learnprompting.org/docs/prompt\_hacking/ defensive\_measures/sandwich\_defense. [36] Kai Mei, Zheng Li, Zhenting Wang, Yang Zhang, and Shiqing Ma. NOTABLE: Transferable Backdoor Attacks Against Prompt-based NLP Models. In ACL, 2023. find
  30. Separate LLM Evaluation. [51] Learn Prompting. https://learnprompting.org/docs/prompt\_ hacking/defensive\_measures/llm\_eval. [37] Meta. Introducing LLaMA: A foundational, 65-billion-parameter large language model. https://ai.facebook.com/blog/large-language-model-llama-meta-ai. [52] Learn Prompting.… ↗
  31. [53] Cheng Qian, Chi Han, Yi R Fung, Yujia Qin, Zhiyuan Liu, and Heng Ji. CREATOR: Disentangling Abstract and Concrete Reasonings of Large Language Models through Tool Creation. arXiv preprint, 2023. find
  32. [39] OpenAI. GPT-4. https://openai.com/research/ gpt-4. [54] Marco Ramponi. The Full Story of Large Language Models and RLHF. https://www.assemblyai. com/blog/the-full-story-of-large-language-models-and-rlhf. [40] OWASP. OWASP Top 10 List for Large Language Models version 0.1. https://owasp.org/www-… ↗
  33. [41] Kaushik Pal. What is Jailbreaking in AI models like ChatGPT? https://www.techopedia.com/what-is-jailbreaking-in-ai-models-like-chatgpt. [56] Ahmed Salem, Michael Backes, and Yang Zhang. Get a Model! Model Hijacking Attack Against Machine Learning Models. In NDSS, 2022. ↗
  34. [68] Zhiyuan Zhang, Lingjuan Lyu, Xingjun Ma, Chenguang Wang, and Xu Sun. Fine-mixing: Mitigating Backdoors in Fine-tuned Language Models. In EMNLP, pages 355–372, 2022. find
  35. [57] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint, 2023. find our card
  36. A [58] Murray Shanahan, Kyle McDonell, and Laria Reynolds. Role-play with large language models. arXiv preprint, 2023. find our card
  37. [59] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint, 2023. find
  38. [60] Wai Man Si, Michael Backes, Jeremy Blackburn, Emiliano De Cristofaro, Gianluca Stringhini, Savvas Zannettou, and Yang Zhang. Why So Toxic?: Measuring and Triggering Toxic Behavior in Open-Domain Chatbots. In CCS, pages 2659–2673, 2022. find
  39. [61] Wai Man Si, Michael Backes, Yang Zhang, and Ahmed Salem. Two-in-One: A Model Hijacking Attack Against Text Generation Models. arXiv preprint, 2023. find
  40. [62] Weiwei Sun, Zhengliang Shi, Shen Gao, Pengjie Ren, Maarten de Rijke, and Zhaochun Ren. Contrastive Learning Reduces Hallucination in Conversations. arXiv preprint, 2022. find
  41. [63] Joel Weinberger, Prateek Saxena, Devdatta Akhawe, Matthew Finifter, Richard Shin, and Dawn Song. A systematic analysis of XSS sanitization in web application frameworks. In ESORICS, pages 150–171, 2011. find
  42. [64] Yotam Wolf, Noam Wies, Yoav Levine, and Amnon Shashua. Fundamental limitations of alignment in large language models. arXiv preprint, 2023. find
  43. [65] Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. On the Tool Manipulation Capability of Open-source Large Language Models. arXiv preprint, 2023. find
  44. [66] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. ICLR, 2023. find our card
  45. [67] Yunxiang Zhang, Liangming Pan, Samson Tan, and Min-Yen Kan. Interpreting the Robustness of Neural NLP Models to Textual Perturbations. In ACL, pages 3993– 4007, 2022. find
  46. List of Anonymized LLM-integrated Applications *Table 5: Overview of LLM-Integrated Applications Used in Our Evaluation. We include the full list of LLM-integrated applications tested and evaluated in our work in this table. Note that we refer to them using the anonymized alias, together with a shor… find