Elamur Research⌕
☰
home / papers / 2025 / Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplification

Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplification

Paper · 2025 · Boyang Zhang, Yicong Tan, Yun Shen et al.

Summary

Most safety studies of LLM agents look for overtly harmful behaviour, such as stealing data or moving money. Such attacks are easy to spot and depend on the specific agent. This paper takes a denial-of-service view instead. The attacker, with only black-box access, uses harmless-looking inputs to make tool-using agents malfunction. The two basic failure types are endless repetition of actions until the iteration limit runs out, and execution of a benign but wrong function. The authors adapt prompt injection and three adversarial perturbation methods (SCPN, VIPER, GCG) to this goal. They test them on many emulated agents, on three core LLMs, and across different attack surfaces. They also scale the attacks to multi-agent settings with two implemented, deployable agents, where one compromised agent spreads loops or spam-like actions downstream. Failure rates exceed 80% in several scenarios. Self-examination defenses detect these attacks far less often than earlier harmful-intent attacks. This shows that agent robustness needs protection beyond policy-violation filters.

Key points

  • Attack types. There are two basic attacks: infinite loop and incorrect function execution. Each extends to an advanced "viral" multi-agent form that ties up downstream agents' resources or triggers mass actions such as spamming.
  • Most effective setup. Prompt injection that pushes the agent into repetitive action loops works best. Direct manipulation of user input is the strongest attack surface, while tool outputs sometimes boost certain attacks.
  • Policy-violation check. A self-examination prompt asking whether an action violates policy flagged 0% of the malfunction attacks. The same check caught 83.7–100% of the harmful injections from Zhan et al. (2024).
  • Malfunction-focused check. A detection prompt aimed at malfunctions performed only slightly better, catching at most 20% of loop injections and 30% of GCG inputs. It caught 0% of incorrect-function, VIPER and SCPN inputs.
  • Limitations. Only two real agents were implemented, plus an agent emulator, and only three proprietary core LLM variants were tested.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States378 works · 161,630 citations🔬Helmholtz Center for Information SecuritySaarbrücken · Germany20 works · 1,172 citations🎓Delft University of TechnologyDelft · Netherlands6 works · 290 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Bias, fairness & safety · 587 works

Cite

Cite as Boyang Zhang, Yicong Tan, Yun Shen, Ahmed Salem, Michael Backes, Savvas Zannettou and Yang Zhang. 2025. Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplification. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.1771

@inproceedings{zhang2025breaking,
  title = {{Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplification}},
  author = {Boyang Zhang, Yicong Tan, Yun Shen, Ahmed Salem, Michael Backes, Savvas Zannettou and Yang Zhang},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.1771},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.1771}
}

References

53 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sahar Abdelnabi, Kai Greshake, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not What You’ve Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. In Workshop on Security and Artificial Intelligence (AISec), pages 79–90. ACM.
  2. Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, J Zico Kolter, Matt Fredrikson, Yarin Gal, and Xander Davies. 2025. Agentharm: A benchmark for measuring harmfulness of LLM agents. In International Conference on Learning Repre… find
  3. AutoGPT. https://news.agpt.co/. ↗
  4. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity. CoRR abs/2302.0402…
  5. Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Srndic, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. 2013. Evasion Attacks against Machine Learning at Test Time. In European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML/PK… find
  6. Nicholas Boucher, Ilia Shumailov, Ross Anderson, and Nicolas Papernot. 2022. Bad Characters: Imperceptible NLP Attacks. In IEEE Symposium on Security and Privacy (S&P), pages 1987–2004. IEEE. find
  7. Ting-Yun Chang and Robin Jia. 2023. Data Curation Alone Can Stabilize In-context Learning. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 8123–8144. ACL. find
  8. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. 2023. Jailbreaking Black Box Large Language Models in Twenty Queries. CoRR abs/2310.08419.
  9. Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. 2024. Agentdojo: A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents. In Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS. find
  10. Gelei Deng, Yi Liu, Yuekang Li, Kailong Wang, Ying Zhang, Zefeng Li, Haoyu Wang, Tianwei Zhang, and Yang Liu. 2023. Jailbreaker: Automated Jailbreak Across Multiple Large Language Model Chatbots. CoRR abs/2307.08715. — [arXiv:2307.08715](https://arxiv.org/abs/2307.08715) find
  11. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. 2023. A Survey on In-context Learning. CoRR abs/2301.00234.
  12. Haonan Duan, Adam Dziedzic, Mohammad Yaghini, Nicolas Papernot, and Franziska Boenisch. 2023. On the Privacy Risk of In-context Learning. In Workshop on Trustworthy Natural Language Processing (TrustNLP). find
  13. Steffen Eger, Gözde Gül Sahin, Andreas Rücklé, Ji-Ung Lee, Claudia Schulz, Mohsen Mesgar, Krishnkant Swarnkar, Edwin Simpson, and Iryna Gurevych. 2019. Text Processing Like Humans Do: Visually Attacking and Shielding NLP Systems. In Conference of the North American Chapter of the Association for Com…
  14. Xuanjie Fang, Sijie Cheng, Yang Liu, and Wei Wang. 2023. Modeling Adversarial Attack on Pre-trained Language Models as Sequential Decision Making. In Annual Meeting of the Association for Computational Linguistics (ACL), pages 7322–7336. ACL. find
  15. Piotr Gainski and Klaudia Balazy. 2023. Step by Step Loss Goes Very Far: Multi-Step Quantization for Adversarial Text Attacks. In Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 2030–2040. ACL. find
  16. Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and Harnessing Adversarial Examples. In International Conference on Learning Representations (ICLR).
  17. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. More than you’ve asked for: A Comprehensive Analysis of Novel Prompt Injection Threats to Application-Integrated Large Language Models. CoRR abs/2302.12173.
  18. Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. 2021. Gradient-based Adversarial Attacks against Text Transformers. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5747–5757. ACL.
  19. Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. 2023. Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation. CoRR abs/2310.06987.
  20. Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial Example Generation with Syntactically Controlled Paraphrase Networks. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 1875– 18…
  21. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of Hallucination in Natural Language Generation. ACM Computing Surveys.
  22. LangChain. https://www.langchain.com/. ↗
  23. Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, and Yangqiu Song. 2023a. Multi-step Jailbreaking Privacy Attacks on ChatGPT. CoRR abs/2304.05197.
  24. Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. 2019. TextBugger: Generating Adversarial Text Against Real-world Applications. In Network and Distributed System Security Symposium (NDSS). Internet Society.
  25. Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023b. HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6449–6464. ACL.
  26. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. 2023a. AgentBench: Evaluating LLMs…
  27. Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2023b. AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models. CoRR abs/2310.04451.
  28. Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. 2023c. Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study. CoRR abs/2305.13860.
  29. Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. 2023d. InstrPrompt Injection Attacks and Defenses in LLM-Integrated Applications. CoRR abs/2310.12815.
  30. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 11048–11064. ACL.
  31. Lingbo Mo, Zeyi Liao, Boyuan Zheng, Yu Su, Chaowei Xiao, and Huan Sun. 2024. A Trembling House of Cards? Mapping Adversarial Attacks against Language Agents. CoRR abs/2402.10196. — [arXiv:2402.10196](https://arxiv.org/abs/2402.10196) find
  32. OpenAI. GPT-4o. hello-gpt-4o/. https://openai.com/index/ ↗
  33. OpenAI. 2023. GPT-4 Blog. https://openai.com/ research/gpt-4/. ↗
  34. Jane Pan, Tianyu Gao, Howard Chen, and Danqi Chen. 2023. What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learning. CoRR abs/2305.09731.
  35. Ashwinee Panda, Tong Wu, Jiachen T. Wang, and Prateek Mittal. 2023. Differentially Private In-Context Learning. CoRR abs/2305.01639. — [arXiv:2305.01639](https://arxiv.org/abs/2305.01639) find
  36. Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative Agents: Interactive Simulacra of Human Behavior. CoRR abs/2304.03442. find
  37. Yao Qiang, Xiangyu Zhou, and Dongxiao Zhu. 2023. Hijacking Large Language Models via Adversarial In-Context Learning. CoRR abs/2311.09948. — [arXiv:2311.09948](https://arxiv.org/abs/2311.09948) find
  38. Vipula Rawte, Amit P. Sheth, and Amitava Das. 2023. A Survey of Hallucination in Large Foundation Models. CoRR abs/2309.05922.
  39. Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J. Maddison, and Tatsunori Hashimoto. 2024. Identifying the Risks of LM Agents with an LM-Emulated Sandbox. In International Conference on Learning Representations (ICLR). ICLR.
  40. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2024. Do Anything Now: Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models. In ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM.
  41. Octavian Suciu, Radu Mărginean, Yiğitcan Kaya, Hal Daumé III, and Tudor Dumitraş. 2018. When Does Machine Learning FAIL? Generalized Transferability for Evasion and Poisoning Attacks. In USENIX Security Symposium (USENIX Security), pages 1299–1316. USENIX. find
  42. Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal Adversarial Triggers for Attacking and Analyzing NLP. In Conference on Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2153… find
  43. Jiongxiao Wang, Zichen Liu, Keun Hee Park, Muhao Chen, and Chaowei Xiao. 2023. Adversarial Demonstration Attacks on Large Language Models. CoRR abs/2305.14950. — [arXiv:2305.14950](https://arxiv.org/abs/2305.14950) find
  44. WolframAlpha. https://products.wolframalpha.com/llm-api/documentation. ↗
  45. Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. 2023. Defending ChatGPT against jailbreak attack via self-reminders. Nature Machine Intelligence. find
  46. Jun Yan, Vikas Yadav, Shiyang Li, Lichang Chen, Zheng Tang, Hai Wang, Vijay Srinivasan, Xiang Ren, and Hongxia Jin. 2023. Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection. CoRR abs/2307.16888. find
  47. Wenkai Yang, Xiaohan Bi, Yankai Lin, Sishuo Chen, Jie Zhou, and Xu Sun. 2024. Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents. CoRR abs/2402.11208. — [arXiv:2402.11208](https://arxiv.org/abs/2402.11208) find
  48. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In International Conference on Learning Representations (ICLR). ICLR.
  49. Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. 2023. GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts. CoRR abs/2309.10253.
  50. Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. 2024. InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents. CoRR abs/2403.02691.
  51. Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Neil Zhenqiang Gong, Yue Zhang, and Xing Xie. 2023. PromptBench: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts. CoRR abs/2306.04528.
  52. Terry Yue Zhuo, Yujin Huang, Chunyang Chen, and Zhenchang Xing. 2023. Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicity. CoRR abs/2301.12867.
  53. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. 2023. Universal and Transferable Adversarial Attacks on Aligned Language Models. CoRR abs/2307.15043.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2025-breaking-agents-compromising-autonomous-llm/