Elamur Research⌕
☰
home / papers / 2024 / InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents

InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents

Paper · 2024 · Qiusi Zhan, Zhixiang Liang, Zifan Ying et al.

Summary

LLM agents that call tools often read outside content such as emails, reviews, notes or web pages. An attacker can plant instructions in that content to steer the agent against its own user. This is called indirect prompt injection (IPI). InjecAgent is presented as the first benchmark that measures how vulnerable tool-integrated agents are to IPI.

The authors started from the tool definitions of Ruan et al. (2023). They used GPT-4, followed by manual correction, to build 1,054 test cases. Each case pairs a user request with an attacker instruction hidden inside a tool response. Attacks either cause direct harm (for example, moving money or unlocking doors) or steal private data and email it to the attacker.

Across 30 agents, ReAct-prompted models proved easy to manipulate. Adding a fixed hacking prompt to the injected text pushed success rates higher. Agents fine-tuned for tool calling resisted noticeably better. The results suggest that agents with access to personal data and real-world actions need dedicated defenses before wide deployment.

Key points

  • The benchmark covers 1,054 cases, built from 17 user tools and 62 attacker tools. These were drawn from 330 tools across 36 toolkits (Ruan et al., 2023), spanning domains such as finance, smart home, email and health.
  • GPT-4 generated the test cases in stages: user case templates with an attacker-instruction placeholder, then attacker cases. About 30% of the generated attacker instructions lacked required parameters and were manually fixed.
  • ReAct-prompted GPT-4 followed injected instructions in 24% of cases. With the hacking prompt, this rose to 47%.
  • Fine-tuned GPT-4 had an attack success rate of only 7.1%. Fine-tuned agents also produced valid outputs more often.
  • Stated limitations include:
    • only one fixed hacking prompt was tested;
    • only single-turn scenarios with attacks of at most two steps;
    • injected text was never mixed with benign content;
    • only two fine-tuned agents were studied.
  • The findings were disclosed to OpenAI and Anthropic.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Illinois Urbana-ChampaignUrbana · United States108 works · 67,432 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

Cite

Cite as Qiusi Zhan, Zhixiang Liang, Zifan Ying and Daniel Kang. 2024. InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents. Findings of the Association for Computational Linguistics: ACL 2024.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.findings-acl.624

@inproceedings{zhan2024injecagent,
  title = {{InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents}},
  author = {Qiusi Zhan, Zhixiang Liang, Zifan Ying and Daniel Kang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2024},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.624},
  url = {https://doi.org/10.18653/v1/2024.findings-acl.624}
}

References

58 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sahar Abdelnabi, Kai Greshake, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security,…
  2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  3. Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. 2022. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691. https://www.anthropic.com/ ↗
  4. Anthropic. 2023. news/claude-2. find
  5. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609.
  6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  7. Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. 2024. Struq: Defending against prompt injection with structured queries. arXiv preprint arXiv:2402.06363. — [arXiv:2402.06363](https://arxiv.org/abs/2402.06363) find
  8. Harald Cramér. 1999. Mathematical methods of statistics, volume 26. Princeton university press. find
  9. Luigi Daniele and Suphavadeeprasit. 2023. Amplify-instruct: Synthetically generated diverse multi-turn conversations for effecient llm training. arXiv preprint arXiv:(comming soon). find
  10. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2web: Towards a generalist agent for the web. arXiv preprint arXiv:2306.06070.
  11. Olive Jean Dunn. 1961. Multiple comparisons among means. Journal of the American statistical association, 56(293):52–64. find
  12. Aysan Esmradi, Daniel Wankit Yip, and Chun Fai Chan. 2023. A comprehensive survey of attack techniques, implementation, and mitigation strategies in large language models. arXiv preprint arXiv:2312.10982. — [arXiv:2312.10982](https://arxiv.org/abs/2312.10982) find
  13. Kai Greshake. 2023. Inject my pdf: Prompt injection for your resume. find
  14. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023a. Mistral 7b. arXiv preprint arXiv:2310.06825. find
  15. Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023b. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion.
  16. Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto. 2023. Exploiting programmatic behavior of llms: Dual-use through standard security attacks. arXiv preprint arXiv:2302.05733. — [arXiv:2302.05733](https://arxiv.org/abs/2302.05733) find
  17. Ariel N. Lee, Cole J. Hunter, and Nataniel Ruiz. 2023. Platypus: Quick, cheap, and powerful refinement of llms. find
  18. Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, and Yang Liu. 2023. Prompt injection attack against llm-integrated applications. arXiv preprint arXiv:2306.05499.
  19. Pattie Maes. 1995. Agents that reduce work and information overload. In Readings in human–computer interaction, pages 811–821. Elsevier. find
  20. Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. Orca: Progressive learning from complex explanation traces of gpt-4.
  21. NVIDIA. 2023. Securing LLM Systems Against Prompt Injection \| NVIDIA Technical Blog — developer.nvidia.com. https://developer.nvidia.com/blog/securing-llm-systems-against-prompt-injection/. ↗
  22. NVIDIA. 2024. NVIDIA Chat With RTX — nvidia.com. https://www.nvidia.com/en-us/ai-on-rtx/chat-with-rtx-generative-ai/. https://openai.com/blog/ ↗
  23. OpenAI. 2023a. chatgpt-plugins. find
  24. OpenAI. 2023b. https://platform.openai.com/docs/guides/function-calling/supported-models. ↗
  25. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155.
  26. Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. 2023. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334.
  27. Fábio Perez and Ian Ribeiro. 2022. Ignore previous prompt: Attack techniques for language models. arXiv preprint arXiv:2211.09527.
  28. Julien Piet, Maha Alrashed, Chawin Sitawarin, Sizhe Chen, Zeming Wei, Elizabeth Sun, Basel Alomair, and David Wagner. 2023. Jatmo: Prompt injection defense by task-specific finetuning. arXiv preprint arXiv:2312.17673. — [arXiv:2312.17673](https://arxiv.org/abs/2312.17673) find
  29. Avram Piltch. 2023. Chatgpt plugins open security holes from pdfs, websites and more. find
  30. Michael Poli, Jue Wang, Stefano Massaroli, Jeffrey Quesnelle, Ryan Carlow, Eric Nguyen, and Armin Thomas. 2023. StripedHyena: Moving Beyond Transformers with Hybrid Signal Processing Models. find
  31. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2023. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789.
  32. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model.
  33. Johann Rehberger. 2023a. Indirect prompt injection via youtube transcripts. find
  34. Johann Rehberger. 2023b. Openai removes the "chat with code" plugin from store. find
  35. Johann Rehberger. 2023c. Plugin vulnerabilities: Visit a website and have your source code stolen. find
  36. Alexander Robey, Eric Wong, Hamed Hassani, and George J Pappas. 2023. Smoothllm: Defending large language models against jailbreaking attacks. arXiv preprint arXiv:2310.03684.
  37. Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J Maddison, and Tatsunori Hashimoto. 2023. Identifying the risks of lm agents with an lm-emulated sandbox. arXiv preprint arXiv:2309.15817. find
  38. Stuart J Russell and Peter Norvig. 2010. Artificial intelligence a modern approach. London. find
  39. Roman Samoilenko. 2023. New prompt injection attack on chatgpt web version. markdown images can steal your chat data. find
  40. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761.
  41. Jose Selvi. 2022. Exploring prompt injection attacks. https://research.nccgroup.com/2022/12/05/exploring-prompt-injection-attacks/. ↗
  42. Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. 2023. Llm-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2998–3009.
  43. Austin Stubbs. 2023. LLM Hacking: Prompt Injection Techniques — austin-stubbs. https://medium.com/@austin-stubbs/llm-security-types-of-prompt-injection-d7ad8d7d75a3. ↗
  44. Xuchen Suo. 2024. Signed-prompt: A new approach to prevent prompt injection attacks against llm-integrated applications. arXiv preprint arXiv:2401.07612. — [arXiv:2401.07612](https://arxiv.org/abs/2401.07612) find
  45. Yu Tian, Xiao Yang, Jingyuan Zhang, Yinpeng Dong, and Hang Su. 2023. Evil geniuses: Delving into the safety of llm-based agents. arXiv preprint arXiv:2311.11855. — [arXiv:2311.11855](https://arxiv.org/abs/2311.11855) find
  46. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. find
  47. Sam Toyer, Olivia Watkins, Ethan Adrian Mendes, Justin Svegliato, Luke Bailey, Tiffany Wang, Isaac Ong, Karim Elmaaroufi, Pieter Abbeel, Trevor Darrell, et al. 2023. Tensor trust: Interpretable prompt injection attacks from an online game. arXiv preprint arXiv:2311.01011. — [arXiv:2311.01011](https://arxiv.org/abs/2311.01011) find
  48. Lilian Weng. 2023. Llm powered autonomous agents. find
  49. Frank Wilcoxon. 1992. Individual comparisons by ranking methods. In Breakthroughs in Statistics: Methodology and Distribution, pages 196–202. Springer. find
  50. Michael Wooldridge and Nicholas R Jennings. 1995. Intelligent agents: Theory and practice. The knowledge engineering review, 10(2):115–152. find
  51. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244.
  52. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629.
  53. Jingwei Yi, Yueqi Xie, Bin Zhu, Keegan Hines, Emre Kiciman, Guangzhong Sun, Xing Xie, and Fangzhao Wu. 2023. Benchmarking and defending against indirect prompt injection attacks on large language models. arXiv preprint arXiv:2312.14197. — [arXiv:2312.14197](https://arxiv.org/abs/2312.14197) find
  54. Jiahao Yu, Yuhang Wu, Dong Shu, Mingyu Jin, and Xinyu Xing. 2023. Assessing prompt injection risks in 200+ custom gpts. arXiv preprint arXiv:2311.11538. — [arXiv:2311.11538](https://arxiv.org/abs/2311.11538) find
  55. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2023. Mmmu: A massive multi-disc…
  56. Ge Zhang, Xinrun Du, Bei Chen, Yiming Liang, Tongxu Luo, Tianyu Zheng, Kang Zhu, Yuyang Cheng, Chunpu Xu, Shuyue Guo, Haoran Zhang, Xingwei Qu, Junjie Wang, Ruibin Yuan, Yizhi Li, Zekun Wang, Yudong Liu, Yu-Hsuan Tsai, Fengji Zhang, Chenghua Lin, Wenhao Huang, Wenhu Chen, and Jie Fu. 2024. Cmmmu: A… find
  57. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685. find
  58. Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhan-2024-injecagent-benchmarking-indirect-prompt-injections/