Elamur Research⌕
☰
home / papers / 2025 / Defense Against Prompt Injection Attack by Leveraging Attack Techniques

Defense Against Prompt Injection Attack by Leveraging Attack Techniques

Paper · 2025 · Yulin Chen, Haoran Li, Zihao Zheng et al.

Summary

Prompt injection hides attacker instructions inside data that an LLM processes, such as retrieved web results. The model then abandons the user's task and follows the attacker instead. Existing defenses have drawbacks. Fine-tuning needs labelled data and heavy compute, and prompt-based defenses are cheap but usually weak. The authors observe that attacks and defenses pursue mirror-image goals: both try to make the model drop one instruction and obey another. They therefore turn training-free attack techniques into defenses. After the data content, they append a "shield prompt" that copies an attack's structure (ignore, escape-character, or fake-completion style), followed by the original user instruction rather than the injected one. The strongest variant, Fakecom-t, uses the conversation template. It plays the assistant to flag the attack, then plays the user to restate the real request. Tests on Llama3, Llama3.1 and Qwen2 covered direct and indirect injections, including gradient-based attacks. The new defenses beat training-free baselines and rival the fine-tuned StruQ, with only minor utility loss. A further finding is that stronger attacks yield stronger defenses.

Key points

  • Training-free baselines in the comparison were Sandwich, Instructional, Reminder, Isolation and Spotlight. The proposed variants are Ours-Ignore, Ours-Escape, Ours-Fakecom and Ours-Fakecom-t.
  • A fake-completion attack that exploits the conversation template reached roughly 98–100% ASR with no defense. Ours-Fakecom-t lowered it to 1.92% (Llama3), 16.82% (Llama3.1) and 8.17% (Qwen2). Most baselines barely reduced it.
  • Against the fine-tuning defense StruQ, Ours-Ignore scored comparably on most attacks. On the Fakecom attack it reached about 0.10% ASR, while both StruQ variants stayed near 35%.
  • Removing the original instruction from the input barely affected the new defenses but noticeably weakened Sandwich. Removing the retrieved data entirely sharply hurt QA quality.
  • Code is available at github.com/LukeChen-go/pia-defense-by-attack. Gradient-based attacks were not turned into defenses, and the paper offers no formal proof of why the method works.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National University of SingaporeSingapore · Singapore180 works · 34,304 citations🎓Harbin Institute of TechnologyHarbin · China158 works · 28,841 citations🎓Hong Kong University of Science and TechnologyHong Kong · China142 works · 25,136 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Yulin Chen, Haoran Li, Zihao Zheng, Dekai Wu, Yangqiu Song and Bryan Hooi. 2025. Defense Against Prompt Injection Attack by Leveraging Attack Techniques. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.897

@inproceedings{chen2025defense,
  title = {{Defense Against Prompt Injection Attack by Leveraging Attack Techniques}},
  author = {Yulin Chen, Haoran Li, Zihao Zheng, Dekai Wu, Yangqiu Song and Bryan Hooi},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.897},
  url = {https://doi.org/10.18653/v1/2025.acl-long.897}
}

References

39 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 2023. Instruction defense. https://learnprompting.org/docs/prompt_hacking/defensive_ measures/instruction. 2023. Sandwich defense. https://learnprompting. org/docs/prompt_hacking/defensive_ measures/sandwich_defense. ↗
  2. AI@Meta. 2024. Llama 3 model card. find
  3. Mark Breitenbach, Adrian Wood, Win Suen, and Po-Ning Tseng. 2023. Don’t you (forget nlp): Prompt injection with control characters in chatgpt. https://dropbox.tech/machine-learning/ prompt-injection-with-control-characters_ openai-chatgpt-llm. ↗
  4. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Aleth… — Evaluating Large Language Models Trained on Code
  5. Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. 2024. Struq: Defending against prompt injection with structured queries. arXiv preprint arXiv:2402.06363. — [arXiv:2402.06363](https://arxiv.org/abs/2402.06363) find
  6. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, et al. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  7. Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. 2024. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36.
  8. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security,…
  9. Yufei He, Yuan Sui, Xiaoxin He, and Bryan Hooi. 2024. Unigraph: Learning a unified cross-domain foundation model for text-attributed graphs. arXiv preprint arXiv:2402.13630. — [arXiv:2402.13630](https://arxiv.org/abs/2402.13630) find
  10. Keegan Hines, Gary Lopez, Matthew Hall, Federico Zarfati, Yonatan Zunger, and Emre Kiciman. 2024. Defending against indirect prompt injection attacks with spotlighting. arXiv preprint arXiv:2403.14720. — [arXiv:2403.14720](https://arxiv.org/abs/2403.14720) find
  11. Yihao Huang, Chong Wang, Xiaojun Jia, Qing Guo, Felix Juefei-Xu, Jian Zhang, Geguang Pu, and Yang Liu. 2024. Semantic-guided prompt organization for universal goal hijacking against llms. arXiv preprint arXiv:2405.14189. — [arXiv:2405.14189](https://arxiv.org/abs/2405.14189) find
  12. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  13. Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Structgpt: A general framework for large language model to reason over structured data. arXiv:2305.09645. arXiv preprint Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. La… find
  14. Haoran Li, Yulin Chen, Jinglong Luo, Jiecong Wang, Hao Peng, Yan Kang, Xiaojin Zhang, Qi Hu, Chunkit Chan, Zenglin Xu, et al. 2023a. Privacy in large language models: Attacks, defenses and future directions. arXiv preprint arXiv:2310.10383. — [arXiv:2310.10383](https://arxiv.org/abs/2310.10383) find
  15. Yunxin Li, Zhenyu Liu, Zitao Li, Xuanyu Zhang, Zhenran Xu, Xinyu Chen, Haoyuan Shi, Shenyuan Jiang, Xintong Wang, Jifang Wang, Shouzheng Huang, Xinping Zhao, Borui Jiang, Lanqing Hong, Longyue Wang, Zhuotao Tian, Baoxing Huai, Wenhan Luo, Weihua Luo, Zheng Zhang, Baotian Hu, and Min Zhang. 2025. Per… — [arXiv:2505.04921](https://arxiv.org/abs/2505.04921) find
  16. Zekun Li, Baolin Peng, Pengcheng He, and Xifeng Yan. 2023b. Evaluating the instruction-following robustness of large language models to prompt injection. find
  17. Xiaogeng Liu, Zhiyuan Yu, Yizhe Zhang, Ning Zhang, and Chaowei Xiao. 2024a. Automatic and universal prompt injection attacks against large language models. arXiv preprint arXiv:2403.04957. — [arXiv:2403.04957](https://arxiv.org/abs/2403.04957) find
  18. Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Zihao Wang, Xiaofeng Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, et al. 2023. Prompt injection attack against llm-integrated applications. arXiv preprint arXiv:2306.05499.
  19. Yue Liu, Hongcheng Gao, Shengfang Zhai, Xia Jun, Tianyi Wu, Zhiwei Xue, Yulin Chen, Kenji Kawaguchi, Jiaheng Zhang, and Bryan Hooi. 2025a. Guardreasoner: Towards reasoning-based llm safeguards. arXiv preprint arXiv:2501.18492. — [arXiv:2501.18492](https://arxiv.org/abs/2501.18492) find
  20. Yue Liu, Jiaying Wu, Yufei He, Hongcheng Gao, Hongyu Chen, Baolong Bi, Jiaheng Zhang, Zhiqi Huang, and Bryan Hooi. 2025b. Efficient inference for large reasoning models: A survey. arXiv preprint arXiv:2503.23077. — [arXiv:2503.23077](https://arxiv.org/abs/2503.23077) find
  21. Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. 2024b. Formalizing and benchmarking prompt injection attacks and defenses. In USENIX Security Symposium. find
  22. OWASP. 2023. OWASP Top 10 for LLM Applications, 2023. https://llmtop10.com. ↗
  23. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.
  24. Fábio Perez and Ian Ribeiro. 2022. Ignore previous prompt: Attack techniques for language models. arXiv preprint arXiv:2211.09527.
  25. Julien Piet, Maha Alrashed, Chawin Sitawarin, Sizhe Chen, Zeming Wei, Elizabeth Sun, Basel Alomair, and David Wagner. 2023. Jatmo: Prompt injection defense by task-specific finetuning. arXiv preprint arXiv:2312.17673. — [arXiv:2312.17673](https://arxiv.org/abs/2312.17673) find
  26. Avital Shafran, Roei Schuster, and Vitaly Shmatikov. 2024. Machine against the rag: Jamming retrieval-augmented generation with blocker documents. arXiv preprint arXiv:2406.05870. — [arXiv:2406.05870](https://arxiv.org/abs/2406.05870) find
  27. Jiawen Shi, Zenghui Yuan, Yinuo Liu, Yue Huang, Pan Zhou, Lichao Sun, and Neil Zhenqiang Gong. 2024. Optimization-based prompt injection attack to llm-as-a-judge. arXiv preprint arXiv:2403.17710. — [arXiv:2403.17710](https://arxiv.org/abs/2403.17710) find
  28. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631…
  29. Yuan Sui, Yufei He, Zifeng Ding, and Bryan Hooi. 2024. Can knowledge graphs make large language models more trustworthy? an empirical study over open-ended question answering. arXiv preprint arXiv:2410.08085.
  30. Jingwei Yi, Yueqi Xie, Bin Zhu, Keegan Hines, Emre Kiciman, Guangzhong Sun, Xing Xie, and Fangzhao Wu. 2023. Benchmarking and defending against indirect prompt injection attacks on large language models. arXiv preprint arXiv:2312.14197. — [arXiv:2312.14197](https://arxiv.org/abs/2312.14197) find
  31. Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. 2024. Injecagent: Benchmarking indirect prompt injections in tool-integrated large language model agents. arXiv preprint arXiv:2403.02691.
  32. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. 2023. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representation…
  33. Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. 2023. Autodan: Automatic and interpretable adversarial attacks on large language models. arXiv preprint arXiv:2310.15140. find
  34. Qing Zong, Zhaowei Wang, Tianshi Zheng, Xiyu Ren, and Yangqiu Song. 2024. Comparisonqa: Evaluating factuality robustness of llms through knowledge frequency control and uncertainty. find
  35. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.
  36. Xuchen Suo. 2024. Signed-prompt: A new approach to prevent prompt injection attacks against llm-integrated applications. arXiv preprint arXiv:2401.07612. — [arXiv:2401.07612](https://arxiv.org/abs/2401.07612) find
  37. Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel. 2024. The instruction hierarchy: Training llms to prioritize privileged instructions. arXiv preprint arXiv:2404.13208. find
  38. Simon Willison. 2023. Delimiters won’t save you from prompt injection. https://simonwillison.net/ 2023/May/11/delimiters-wont-save-you. ↗
  39. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Juny… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2025-defense-against-prompt-injection-attack/