Elamur Research⌕
☰
home / papers / 2025 / PIGuard: Prompt Injection Guardrail via Mitigating Overdefense for Free

PIGuard: Prompt Injection Guardrail via Mitigating Overdefense for Free

Paper · 2025 · Hao Li, Xiaogeng Liu, Ning Zhang et al.

Summary

Lightweight prompt guard classifiers screen LLM inputs for prompt injection, but they often learn a shortcut. They flag harmless text as an attack simply because it contains words typical of injections, such as "ignore". The authors call this over-defense. To measure it, they built NotInject, a benchmark of benign prompts deliberately seeded with injection-style trigger words found through statistical analysis of benign and attack corpora. Popular guards, including ProtectAIv2 and PromptGuard, score below 60% on it, which is near chance.

They then trained PIGuard, a DeBERTa-based detector. It uses a fully open training collection with data-centric augmentation and a new strategy, Mitigating Over-defense for Free (MOF). MOF lowers trigger-word bias without training on any dedicated over-defense data. PIGuard reaches about 83.5% average accuracy across benign, malicious and over-defense tests. That nearly matches GPT-4o while running hundreds of times faster. The work matters because it is presented as the first fully open-source injection guard, with data, code and model all released.

Key points

  • NotInject: contains 339 benign samples, split into three difficulty levels by how many trigger words each prompt contains.
  • PIGuard accuracy: 83.48% average (87.32% over-defense, 85.74% benign, 77.39% malicious). This is 30.4% better than the strongest existing open-source guard.
  • Efficiency: PIGuard needs 15.34 ms per input and 60.45 GFLOPS. GPT-4o reaches 85.53% but takes about 7907 ms. Their efficiency scores are 5.44 and 0.01 respectively.
  • Comparison with shortcut mitigation: a standard method (Wang et al., 2021) raised over-defense accuracy but cut malicious-detection accuracy by 10.42%. MOF improved both.
  • Attention analysis: attention visualization shows that ProtectAIv2 focuses heavily on "ignore". PIGuard spreads its attention across the whole sentence.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Wisconsin–MadisonMadison · United States42 works · 48,486 citations🎓Washington University in St. LouisSt Louis · United States12 works · 1,028 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Hao Li, Xiaogeng Liu, Ning Zhang and Chaowei Xiao. 2025. PIGuard: Prompt Injection Guardrail via Mitigating Overdefense for Free. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.1468

@inproceedings{li2025piguard,
  title = {{PIGuard: Prompt Injection Guardrail via Mitigating Overdefense for Free}},
  author = {Hao Li, Xiaogeng Liu, Ning Zhang and Chaowei Xiao},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1468},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1468}
}

References

63 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sahar Abdelnabi, Aideen Fay, Giovanni Cherubin, Ahmed Salem, Mario Fritz, and Andrew Paverd. 2024. Are you still on track!? catching LLM task drift with activations. Preprint, arXiv:2406.00799. to evaluate attacks and defenses for llm agents. arXiv preprint arXiv:2406.13352. — [arXiv:2406.00799](https://arxiv.org/abs/2406.00799) find
  2. Deepset. 2024a. Deepset Prompt Injection Dataset. https://huggingface.co/datasets/deepset/ prompt-injections. ↗
  3. Deepset. 2024b. Deepset Prompt Injection Guardrail. https://huggingface.co/deepset/ deberta-v3-base-injection. ↗
  4. P Kingma Diederik. 2015. Adam: A method for stochastic optimization. In the Proceedings of ICLR.
  5. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing chat language models by scaling high-quality instructional conversations. Preprint, arXiv:2305.14233.
  6. Basemah Alshemali and Jugal Kalita. 2020. Improving the reliability of deep neural networks in nlp: A review. Knowledge-Based Systems, 191:105210. find
  7. Xin Luna Dong, Seungwhan Moon, Yifan Ethan Xu, Kshitiz Malik, and Zhou Yu. 2023. Towards next-generation intelligent assistants leveraging llm techniques. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’23, page 5792–5793, New York, NY, USA. Association… find
  8. Bang An, Sicheng Zhu, Ruiyi Zhang, Michael-Andrei Panaitescu-Liess, Yuancheng Xu, and Furong Huang. 2024. Automatic pseudo-harmful prompt generation for evaluating false refusals in large language models. CoRR, abs/2409.00598. — [arXiv:2409.00598](https://arxiv.org/abs/2409.00598) find
  9. Mengnan Du, Varun Manjunatha, Rajiv Jain, Ruchi Deshpande, Franck Dernoncourt, Jiuxiang Gu, Tong Sun, and Xia Hu. 2021. Towards interpreting and mitigating shortcut learning behavior of nlu models. arXiv preprint arXiv:2103.06922. find
  10. Hezekiah J. Branch, Jonathan Rodriguez Cefalu, Jeremy McHugh, Leyla Hujer, Aditya Bahl, Daniel del Castillo Iglesias, Ron Heichman, and Ramesh Darwishi. 2022. Evaluating the susceptibility of pre-trained language models via handcrafted adversarial examples. Preprint, arXiv:2209.02128. — [arXiv:2209.02128](https://arxiv.org/abs/2209.02128) find
  11. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez,… find
  12. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  13. Sizhe Chen, Julien Piet, Chawin Sitawarin, and David A. Wagner. 2024. Struq: Defending against prompt injection with structured queries. find
  14. Justin Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui Hsieh. 2024. Or-bench: An over-refusal benchmark for large language models. CoRR, abs/2405.20947.
  15. Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. 2024. Agentdojo: A dynamic environment Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenk…
  16. Lutfi Eren Erdogan, Chuyi Shang, Aryan Goyal, and Siddarth Ijju. 2024. Safe-Guard Prompt Injection Dataset. https://huggingface.co/datasets/ xTRam1/safe-guard-prompt-injection. fmops. 2024. Fmops Prompt Injection Guardrail. https://huggingface.co/fmops/ distilbert-prompt-injection. ↗
  17. Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. 2020. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665–673. find
  18. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. arXiv preprint. ArXiv:2302.12173 [cs].
  19. Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. 2024. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks. Preprint, arXiv:2406.18495.
  20. Jack Hao. 2023. Jailbreak Classification Dataset. https://huggingface.co/datasets/jackhhao/ jailbreak-classification. ↗
  21. Rich Harang. 2023a. Securing llm systems against prompt injection. find
  22. Rich Harang. 2023b. Securing LLM Systems Against Prompt Injection. https://developer.nvidia.com/blog/securing-llm-systems-against-prompt-injection. ↗
  23. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. In the Proceedings of ICLR.
  24. HuggingfaceH4. 2023. Grok Conversation Harmless Dataset. https://huggingface.co/datasets/ HuggingFaceH4/grok-conversation-harmless. ↗
  25. Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. 2023. Llama guard: Llm-based input-output safeguard for human-ai conversations. Preprint, arXiv:2312.06674.
  26. Fereshte Khani and Percy Liang. 2021. Removing spurious features can hurt accuracy and affect groups disproportionately. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 196–205. find
  27. LakeraAI. 2024a. LakeraGuard. lakera.ai/lakera-guard. https://www. find
  28. LakeraAI. 2024b. Prompt Injection Test Dataset. https://www.lakera.ai/product-updates/ lakera-pint-benchmark. ↗
  29. Manxi Lin, Nina Weng, Kamil Mikolaj, Zahra Bashir, Morten BS Svendsen, Martin G Tolsgaard, Anders N Christensen, and Aasa Feragen. 2024. Shortcut learning in medical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 623–633. Springer… find
  30. Frederick Liu and Besim Avci. 2019. Incorporating priors with feature attribution on text classification. arXiv preprint arXiv:1906.08286. find
  31. Xiaogeng Liu, Zhiyuan Yu, Yizhe Zhang, Ning Zhang, and Chaowei Xiao. 2024. Automatic and universal prompt injection attacks against large language models. arXiv preprint arXiv:2403.04957. — [arXiv:2403.04957](https://arxiv.org/abs/2403.04957) find
  32. Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. 2023. Prompt Injection Attacks and Defenses in LLM-Integrated Applications. arXiv preprint. ArXiv:2310.12815 [cs]. find
  33. Varun Manjunatha, Nirat Saini, and Larry S Davis. 2019. Explicit bias discovery in visual question answering models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9562–9571. find
  34. Michael Mendelson and Yonatan Belinkov. 2021. Debiasing methods in natural language understanding make bias more accessible. arXiv preprint arXiv:2109.04095. find
  35. Meta. 2024. PromptGuard Prompt Injection Guardrail. https://www.llama.com/ docs/model-cards-and-prompt-formats/ prompt-guard/. ↗
  36. Timothy Niven and Hung-Yu Kao. 2019. Probing neural network comprehension of natural language arguments. arXiv preprint arXiv:1907.07355. find
  37. OpenAI. 2024a. GPT-4o. index/hello-gpt-4o/. https://openai.com/ ↗
  38. OpenAI. 2024b. Gpt-4o mini: Advancing cost-efficient intelligence. find
  39. Alessandro Palla. 2024. Chatbot Instruction Prompt Dataset. https://huggingface.co/datasets/ alespalla/chatbot_instruction_prompts. ↗
  40. Dario Pasquini, Martin Strohmeier, and Carmela Tron-coso. 2024. Neural exec: Learning (and learning from) execution triggers for prompt injection attacks. Preprint, arXiv:2403.03792. — [arXiv:2403.03792](https://arxiv.org/abs/2403.03792) find
  41. Jiawen Shi, Zenghui Yuan, Yinuo Liu, Yue Huang, Pan Zhou, Lichao Sun, and Neil Zhenqiang Gong. 2024b. Optimization-based prompt injection attack to llm-as-a-judge. Preprint, arXiv:2403.17710. — [arXiv:2403.17710](https://arxiv.org/abs/2403.17710) find
  42. Rodrigo Pedro, Daniel Castro, Paulo Carreira, and Nuno Santos. 2023. From Prompt Injections to SQL Injection Attacks: How Protected is Your LLM-Integrated Web Application? arXiv preprint. ArXiv:2308.01990 [cs]. — [arXiv:2308.01990](https://arxiv.org/abs/2308.01990) find
  43. Adam Swanda. 2023. Vigil-Jailbreak-ada Dataset. https://huggingface.co/datasets/deadbits/ vigil-jailbreak-ada-002. ↗
  44. Fábio Perez and Ian Ribeiro. 2022. Ignore Previous Prompt: Attack Techniques For Language Models. arXiv preprint. ArXiv:2211.09527 [cs].
  45. ProtectAI.com. 2024. Fine-tuned deberta-v3-base for prompt injection detection. find
  46. Nazneen Rajani, Lewis Tunstall, Edward Beeching, Nathan Lambert, Alexander M. Rush, and Thomas Wolf. 2023. No robots. https://huggingface.co/ datasets/HuggingFaceH4/no_robots. ↗
  47. Rubén Darío Jaramillo Romero. 2023. Chat-GPT Jailbreak Promtps Dataset. https://huggingface.co/datasets/rubend18/ ChatGPT-Jailbreak-Prompts. ↗
  48. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca. ↗
  49. Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. 2023. Large language models in medicine. Nature medicine, 29(8):1930– 1940. find
  50. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  51. Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. 2023. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. Preprint, arXiv:2308.01263.
  52. Sam Toyer, Olivia Watkins, Ethan Adrian Mendes, Justin Svegliato, Luke Bailey, Tiffany Wang, Isaac Ong, Karim Elmaaroufi, Pieter Abbeel, Trevor Darrell, Alan Ritter, and Stuart Russell. 2023. Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game. arXiv preprint. ArXiv:2311.01011 [… find
  53. Ahmed Salem, Andrew Paverd, and Boris Köpf. 2023. Maatphor: Automated Variant Analysis for Prompt Injection Attacks. arXiv preprint. ArXiv:2312.11513 [cs]. — [arXiv:2312.11513](https://arxiv.org/abs/2312.11513) find
  54. VMware. 2023. Open-Instruct Dataset. https://huggingface.co/datasets/VMware/ open-instruct. ↗
  55. Sander V Schulhoff, Jeremy Pinto, Anaum Khan, Louis-François Bouchard, Chenglei Si, Jordan Lee Boyd-Graber, Svetlina Anati, Valen Tagliabue, Anson Liu Kost, and Christopher R Carnahan. 2023. Ignore this title and hackaprompt: Exposing systemic vulnerabilities of llms through a global prompt hacking… find
  56. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2023. "do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. Preprint, arXiv:2308.03825.
  57. Chenyu Shi, Xiao Wang, Qiming Ge, Songyang Gao, Xianjun Yang, Tao Gui, Qi Zhang, Xuanjing Huang, Xun Zhao, and Dahua Lin. 2024a. Navigating the overkill in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2… find
  58. Chaofan Wang, Samuel Kernan Freire, Mo Zhang, Jing Wei, Jorge Goncalves, Vassilis Kostakos, Zhanna Sarsenbayeva, Christina Schneegass, Alessandro Bozzon, and Evangelos Niforatos. 2023. Safeguarding Crowdsourcing Surveys from ChatGPT with Prompt Injection. arXiv preprint. ArXiv:2306.08833 [cs]. — [arXiv:2306.08833](https://arxiv.org/abs/2306.08833) find
  59. Tianlu Wang, Rohit Sridhar, Diyi Yang, and Xuezhi Wang. 2021. Identifying and mitigating spurious correlations for improving robustness in nlp models. arXiv preprint arXiv:2110.07736. find
  60. Simon Willison. 2022. Prompt injection attacks against GPT-3. https://simonwillison.net/2022/Sep/ 12/prompt-injection/. ↗
  61. Simon Willison. 2023. Delimiters won’t save you from prompt injection. https://simonwillison.net/ 2023/May/11/delimiters-wont-save-you. ↗
  62. Yuxiang Wu, Matt Gardner, Pontus Stenetorp, and Pradeep Dasigi. 2022. Generating data to mitigate spurious correlations in natural language inference datasets. arXiv preprint arXiv:2203.12942.
  63. Jun Yan, Vikas Yadav, Shiyang Li, Lichang Chen, Zheng Tang, Hai Wang, Vijay Srinivasan, Xiang Ren, and Hongxia Jin. 2023. Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection. arXiv preprint. ArXiv:2307.16888 [cs]. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2025-piguard-prompt-injection-guardrail-mitigating/