Elamur Research⌕
☰
home / papers / 2025 / The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agents

The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agents

Paper · 2025 · Feiran Jia, Tong Wu, Xin Qin et al.

Summary

LLM agents that call tools can be hijacked by indirect prompt injection, where an attacker hides instructions in web pages, documents or tool outputs that the agent reads. Existing defenses rely on rules, delimiters or authentication. They tend either to miss attacks phrased in a harmless tone or to hurt the agent's usefulness. The authors change the question from whether an action is harmful to whether it serves the user's stated goals. They formalize this as task alignment: every assistant-level instruction or tool call must contribute to at least one user-level task instruction, given the conversation history. Their test-time defense, Task Shield, uses an LLM to:

  • extract actionable instructions from assistant messages and tool outputs,
  • check each one against the user's objectives,
  • send corrective feedback to the agent when it finds a mismatch.

On the AgentDojo benchmark with GPT-4o, attack success fell to about 2% while utility stayed near 70%, which beats prior defenses. The work matters because it gives a model-agnostic way to protect agents that needs no retraining.

Key points

  • On AgentDojo with GPT-4o, Task Shield reaches an attack success rate (ASR) of 2.07% and utility of 69.79%.
  • The design has three parts: an Input Shield, an Action Shield (a final check on tool calls) and a step that restates the user's goal. On the Slack suite with GPT-4o-mini, the full Task Shield lowers ASR from 57.14% to 0.95%.
  • In the ablation, removing goal restatement drops utility from 63.8% to 43.8%, while security stays largely intact.
  • Defender strength matters but is not critical. When protecting a GPT-4o assistant on the Slack suite, ASR falls from 92.38% to 0.95% with GPT-4o as defender, 0.00% with GPT-4o-mini and 4.76% with GPT-3.5-turbo.
  • Limitations:
    • Every message requires extra LLM calls.
    • A weak defender can cause false positives or missed attacks.
    • Tests cover only GPT models on a single benchmark, with no open-source models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Princeton UniversityPrinceton · United States90 works · 39,109 citations🎓Pennsylvania State UniversityState College · United States56 works · 5,360 citations🎓California State University, Long BeachLong Beach · United States2 works · 34 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Feiran Jia, Tong Wu, Xin Qin and Anna Squicciarini. 2025. The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agents. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.1435

@inproceedings{jia2025task,
  title = {{The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agents}},
  author = {Feiran Jia, Tong Wu, Xin Qin and Anna Squicciarini},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1435},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1435}
}

References

35 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tom B Brown, Benjamin Mann, Nick Ryder, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems.
  2. Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. 2024. Struq: Defending against prompt injection with structured queries. arXiv preprint arXiv:2402.06363. — [arXiv:2402.06363](https://arxiv.org/abs/2402.06363) find
  3. Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. 2024. Agentdojo: A dynamic environment to evaluate attacks and defenses for llm agents. arXiv preprint arXiv:2406.13352. — [arXiv:2406.13352](https://arxiv.org/abs/2406.13352) find
  4. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2024. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36.
  5. Erik Derner, Kristina Batistič, Jan Zahálka, and Robert Babuška. 2024. A security risk taxonomy for prompt-based interaction with large language models. IEEE Access. find
  6. Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A survey on rag meeting llms: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’24, page 64916501…
  7. Xiaohan Fu, Zihan Wang, Shuheng Li, Rajesh K. Gupta, Niloofar Mireshghallah, Taylor Berg-Kirkpatrick, and Earlence Fernandes. 2024. Misusing tools in large language models with visual adversarial examples. find
  8. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security,…
  9. Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Zihao Wang, Xiaofeng Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, and Yang Liu. 2023. Prompt injection attack against llm-integrated applications. arXiv preprint arXiv:2306.05499.
  10. Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. 2024. Formalizing and benchmarking prompt injection attacks and defenses. In USENIX Security Symposium. find
  11. Jiageng Mao, Junjie Ye, Yuxi Qian, Marco Pavone, and Yue Wang. 2024. A language agent for autonomous driving. In First Conference on Language Modeling. find
  12. Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ramakanth Pasunuru, Roberta Raileanu, Baptiste Roziere, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. 2023. Augmented language models: a survey. Transactions on Machine Learning Re…
  13. OpenAI. 2024. Introducing openai o1-preview. find
  14. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
  15. Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2023. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334.
  16. Fábio Perez and Ian Ribeiro. 2022. Ignore previous prompt: Attack techniques for language models. arXiv preprint arXiv:2211.09527.
  17. Julien Piet, Maha Alrashed, Chawin Sitawarin, Sizhe Chen, Zeming Wei, Elizabeth Sun, Basel Alomair, and David Wagner. 2023. Jatmo: Prompt injection defense by task-specific finetuning. ArXiv, abs/2312.17673. — [arXiv:2312.17673](https://arxiv.org/abs/2312.17673) find
  18. Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. 2023. Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings. Advances in neural information processing systems, 36:45870–45894.
  19. Learn Prompting. 2024. Sandwich defense. https://learnprompting.org/docs/prompt_hacking/ defensive_measures/sandwich_defense. Accessed: 2024-11-07. ↗
  20. Keegan Hines, Gary Lopez, Matthew Hall, Federico Zarfati, Yonatan Zunger, and Emre Kiciman. 2024. Defending against indirect prompt injection attacks with spotlighting. arXiv preprint arXiv:2403.14720. — [arXiv:2403.14720](https://arxiv.org/abs/2403.14720) find
  21. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2023. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789.
  22. Sahasra Kokkula, G Divya, et al. 2024. Palisade– prompt injection detection framework. arXiv preprint arXiv:2410.21146. — [arXiv:2410.21146](https://arxiv.org/abs/2410.21146) find
  23. Mei Li et al. 2024. Securing tool use in llm agents: Challenges and strategies. arXiv preprint arXiv:2402.03014. — [arXiv:2402.03014](https://arxiv.org/abs/2402.03014) find
  24. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2024. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36.
  25. Ashish Sharma, Sudha Rao, Chris Brockett, Akanksha Malhotra, Nebojsa Jojic, and Bill Dolan. 2024. Investigating agency of LLMs in human-AI collaboration tasks. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), page… find
  26. Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. 2023. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. arXiv preprint arXiv:2306.05301.
  27. Hugo Touvron, Thibaut Lavril, Gautier Izacard, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  28. Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel. 2024. The instruction hierarchy: Training llms to prioritize privileged instructions. arXiv preprint arXiv:2404.13208. find
  29. Jiongxiao Wang, Fangzhou Wu, Wendi Li, Jinsheng Pan, Edward Suh, Z. Morley Mao, Muhao Chen, and Chaowei Xiao. 2024. Fath: Authentication-based test-time defense against indirect prompt injection attacks. arXiv preprint arXiv:2410.21492. — [arXiv:2410.21492](https://arxiv.org/abs/2410.21492) find
  30. Jason Wei et al. 2022. Inverse scaling: When bigger isn’t better. arXiv preprint arXiv:2206.04615.
  31. Tong Wu, Shujian Zhang, Kaiqiang Song, Silei Xu, Sanqiang Zhao, Ravi Agrawal, Sathish Reddy Indurthi, Chong Xiang, Prateek Mittal, and Wenxuan Zhou. 2024. Instructional segment embedding: Improving llm safety with instruction hierarchy. Preprint, arXiv:2410.09102. — [arXiv:2410.09102](https://arxiv.org/abs/2410.09102) find
  32. Zhen Xiang, Linzhi Zheng, Yanjie Li, Junyuan Hong, Qinbin Li, Han Xie, Jiawei Zhang, Zidi Xiong, Chulin Xie, Carl Yang, et al. 2024. Guardagent: Safeguard llm agents by a guard agent via knowledge-enabled reasoning. arXiv preprint arXiv:2406.09187. — [arXiv:2406.09187](https://arxiv.org/abs/2406.09187) find
  33. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations.
  34. Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. 2024. Injecagent: Benchmarking indirect prompt injections in tool-integrated large language model agents. arXiv preprint arXiv:2403.02691.
  35. Xiangzhe Zou et al. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.09283.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jia-2025-task-shield-enforcing-task-alignment/