Elamur Research⌕
☰
home / papers / 2025 / AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detection

AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detection

Paper · 2025 · Weidi Luo, Shenghong Dai, Xiaogeng Liu et al.

Summary

LLM-based agents that run shell commands, browse websites or query databases can cause harm in two ways. Task-specific risks break rules that an administrator sets for a given job, such as access restrictions on medical records. Systemic risks threaten confidentiality, integrity or availability, for example through prompt injection or accidental file overwrites. Earlier guardrails such as GuardAgent and Conseca either depend on hand-written trusted contexts or produce policies that block too much or too little.

AGrail is a lifelong guardrail that addresses both problems. It derives safety checks from universal safety criteria and any task-specific requests. Two cooperating LLMs then refine these checks during test-time adaptation (TTA), and the checks that prove effective are kept in memory. External tools can be called when a check needs them. The authors also release Safe-OS, a benchmark of real online OS-agent interactions that mixes attacks with benign tasks.

With Claude-3.5-Sonnet as the base model, AGrail detects risks accurately while rarely blocking legitimate actions. It also carries its learned checks across domains, which makes agents both safer and still usable.

Key points

  • Evaluation covers five benchmarks:
    • Task-specific risks: Mind2Web-SC and EICU-AC.
    • Systemic risks: AdvWeb, EIA and the new Safe-OS benchmark.
    • Safe-OS design: three attack scenarios (prompt injection, system sabotage and environment-dependent attacks) plus benign samples to measure overdefensiveness.
  • With Claude-3.5-Sonnet, AGrail reaches:
    • 96.3% average accuracy on task-specific risks;
    • 96% of benign actions preserved;
    • attack success rate on Safe-OS of 3.8% (environment), 5% (sabotage) and 0% (prompt injection);
    • 0% attack success rate on AdvWeb and about 17% on EIA.
  • In agreement with manually annotated ground-truth risk reasons, AGrail beats baselines by 46.2% on average and reaches 100% on OS prompt injection.
  • Memory learned on Mind2Web-SC transferred to EICU-AC without any loss for Claude-3.5-Sonnet and with a 5.6% drop for GPT-4o.
  • Stronger base models are largely insensitive to data order (F1 99.1 ±1.3 for Claude vs. 69.7 ±9.5 for GPT-4o-mini). Learned memories across random seeds converge to over 98% similarity after about four TTA iterations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The Ohio State UniversityColumbus · United States90 works · 19,112 citations🎓University of Wisconsin–MadisonMadison · United States54 works · 49,089 citations🎓University of California, DavisDavis · United States34 works · 42,940 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSafety Research · 626 works

🧭 Topics 1Research subjects and languages the work is about.Bias, fairness & safety · 587 works

Cite

Cite as Weidi Luo, Shenghong Dai, Xiaogeng Liu, Suman Banerjee, Huan Sun, Muhao Chen and Chaowei Xiao. 2025. AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detection. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.399

@inproceedings{luo2025agrail,
  title = {{AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detection}},
  author = {Weidi Luo, Shenghong Dai, Xiaogeng Liu, Suman Banerjee, Huan Sun, Muhao Chen and Chaowei Xiao},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.399},
  url = {https://doi.org/10.18653/v1/2025.acl-long.399}
}

References

59 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mahyar Abbasian, Iman Azimi, Amir M. Rahmani, and Ramesh Jain. 2024. Conversational health agents: A personalized llm-powered agent framework. Preprint, arXiv:2310.02374. — [arXiv:2310.02374](https://arxiv.org/abs/2310.02374) find
  2. Meta AI. 2025. Llama guard 3: Model cards and prompt formats. Accessed: 2025-02-02. find
  3. Daniil A. Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes. 2023. Autonomous chemical research with large language models. Nature, 624(7992):570– 578.
  4. Andres M Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. 2023. Chemcrow: Augmenting large-language models with chemistry tools. Nature Machine Intelligence.
  5. Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. 2024. Agentdojo: A dynamic environment to evaluate prompt injection attacks and defenses for llm agents. Preprint, arXiv:2406.13352. find
  6. A. Ghafarollahi and M. J. Buehler. 2024. Protagents: Protein discovery via large language model multi-agent collaborations combining physics and machine learning. Preprint, arXiv:2402.04268. — [arXiv:2402.04268](https://arxiv.org/abs/2402.04268) find
  7. Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. 2024. Navigating the digital world as humans do: Universal visual grounding for gui agents. International Conference on Learning Representations (ICLR).
  8. Yu Gu, Yiheng Shu, Hao Yu, Xiao Liu, Yuxiao Dong, Jie Tang, Jayanth Srinivasa, Hugo Latapie, and Yu Su. 2024a. Middleware for llms: Tools are instrumental for language agents in complex environments. Preprint, arXiv:2402.14672. find
  9. Yu Gu, Boyuan Zheng, Boyu Gou, Kai Zhang, Cheng Chang, Sanjari Srivastava, Yanan Xie, Peng Qi, Huan Sun, and Yu Su. 2024b. Is your llm secretly a world model of the internet? model-based planning for web agents. Preprint, arXiv:2411.06559. — [arXiv:2411.06559](https://arxiv.org/abs/2411.06559) find
  10. Yifeng He, Ethan Wang, Yuyang Rong, Zifei Cheng, and Hao Chen. 2024. Security of ai agents. Preprint, arXiv:2406.08689. — [arXiv:2406.08689](https://arxiv.org/abs/2406.08689) find
  11. Wenyue Hua, Xianjun Yang, Mingyu Jin, Zelong Li, Wei Cheng, Ruixiang Tang, and Yongfeng Zhang. 2024a. Trustagent: Towards safe and trustworthy llm-based agents. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10000–10016. find
  12. Wenyue Hua, Xianjun Yang, Mingyu Jin, Zelong Li, Wei Cheng, Ruixiang Tang, and Yongfeng Zhang. 2024b. Trustagent: Towards safe and trustworthy llm-based agents. Preprint, arXiv:2402.01586. find
  13. Mingyu Jin, Qinkai Yu, Jingyuan Huang, Qingcheng Zeng, Zhenting Wang, Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng, Kaize Ding, Fan Yang, Mengnan Du, and Yongfeng Zhang. 2025. Exploring concept depth: How large language models acquire knowledge and concept at different layers? Proceedings of the 31s…
  14. Mingyu Jin, Qinkai Yu, Dong Shu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, and Mengnan Du. 2024. The impact of reasoning step length on large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 1830–1842, Bangkok, Thailand. Association for Comput…
  15. Hao Li and Xiaogeng Liu. 2024. Injecguard: Benchmarking and mitigating over-defense in prompt injection guardrail models. Preprint, arXiv:2410.22770. — [arXiv:2410.22770](https://arxiv.org/abs/2410.22770) find
  16. Junkai Li, Siyu Wang, Meng Zhang, Weitao Li, Yunghwei Lai, Xinhui Kang, Weizhi Ma, and Yang Liu. 2024. Agent hospital: A simulacrum of hospital with evolvable medical agents. Preprint, arXiv:2405.02957. — [arXiv:2405.02957](https://arxiv.org/abs/2405.02957) find
  17. Zeyi Liao, Lingbo Mo, Chejian Xu, Mintong Kang, Jiawei Zhang, Chaowei Xiao, Yuan Tian, Bo Li, and Huan Sun. 2025. Eia: Environmental injection attack on generalist web agents for privacy leakage. International Conference on Learning Representations (ICLR). find
  18. Shuhang Lin, Wenyue Hua, Lingyao Li, Che-Jui Chang, Lizhou Fan, Jianchao Ji, Hang Hua, Mingyu Jin, Jiebo Luo, and Yongfeng Zhang. 2024. BattleAgent: Multi-modal dynamic emulation on historical battles to complement historical analysis. In Proceedings of the 2024 Conference on Empirical Methods in Na… find
  19. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. 2024a. Agentbench: Evaluating llms…
  20. Xiaogeng Liu, Zhiyuan Yu, Yizhe Zhang, Ning Zhang, and Chaowei Xiao. 2024b. Automatic and universal prompt injection attacks against large language models. Preprint, arXiv:2403.04957. — [arXiv:2403.04957](https://arxiv.org/abs/2403.04957) find
  21. Weidi Luo, He Cao, Zijing Liu, Yu Wang, Aidan Wong, Bing Feng, Yuan Yao, and Yu Li. 2025a. Dynamic guided and domain applicable safeguards for enhanced security in large language models. Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL). find
  22. Weidi Luo, Siyuan Ma, Xiaogeng Liu, Xiaoyu Guo, and Chaowei Xiao. 2024. Jailbreakv: A benchmark for assessing the robustness of multimodal large language models against jailbreak attacks. Conference On Language Modeling (COLM).
  23. Weidi Luo, Qiming Zhang, Tianyu Lu, Xiaogeng Liu, Yue Zhao, Zhen Xiang, and Chaowei Xiao. 2025b. Doxing via the lens: Revealing privacy leakage in image geolocation for agentic multi-modal large reasoning model. Preprint, arXiv:2504.19373. — [arXiv:2504.19373](https://arxiv.org/abs/2504.19373) find
  24. Kai Mei, Zelong Li, Shuyuan Xu, Ruosong Ye, Yingqiang Ge, and Yongfeng Zhang. 2024a. Llm agent operating system. arXiv preprint arXiv:2403.16971. — [arXiv:2403.16971](https://arxiv.org/abs/2403.16971) find
  25. Kai Mei, Xi Zhu, Wujiang Xu, Wenyue Hua, Mingyu Jin, Zelong Li, Shuyuan Xu, Ruosong Ye, Yingqiang Ge, and Yongfeng Zhang. 2024b. Aios: Llm agent operating system. Preprint, arXiv:2403.16971. — [arXiv:2403.16971](https://arxiv.org/abs/2403.16971) find
  26. MetaAI. 2024a. Llama guard 3 8b. Accessed: 2024-08-16. find
  27. MetaAI. 2024b. Meta llama 3 8b instruct. Accessed: 2024-08-16. find
  28. Silen Naihin, David Atkinson, Marc Green, Mer-wane Hamadi, Craig Swift, Douglas Schonholtz, Adam Tauman Kalai, and David Bau. 2023. Testing language model agents safely in the wild. Preprint, arXiv:2311.10538. — [arXiv:2311.10538](https://arxiv.org/abs/2311.10538) find
  29. Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In In the 36th Annual ACM Symposium on User Interface Software and Technology (UIST ’23), UIST ’23, New York, NY, USA. As… find
  30. Traian Rebedea, Razvan Dinu, Makesh Narsimhan Sreedhar, Christopher Parisien, and Jonathan Cohen. 2023. NeMo guardrails: A toolkit for controllable and safe LLM applications with programmable rails. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Dem… find
  31. Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J Maddison, and Tatsunori Hashimoto. 2024. find
  32. Identifying the risks of lm agents with an lm-emulated sandbox. International Conference on Learning Representations (ICLR).
  33. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2024. “Do Anything Now”: Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models. In ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM.
  34. Wenqi Shi, Ran Xu, Yuchen Zhuang, Yue Yu, Jieyu Zhang, Hang Wu, Yuanda Zhu, Joyce C. Ho, Carl Yang, and May Dongmei Wang. 2024. EHRAgent: Code empowers large language models for few-shot complex tabular reasoning on electronic health records. In Proceedings of the 2024 Conference on Empirical Method… find
  35. Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. Conference on Neural Information Processing Systems (Neurips).
  36. Yu Su. 2023. Language agents: a critical evolutionary step of artificial intelligence. yusu.substack.com. find
  37. Lillian Tsai and Eugene Bagdasarian. 2025. Context is key for agent security. Preprint, arXiv:2501.17070. — [arXiv:2501.17070](https://arxiv.org/abs/2501.17070) find
  38. Tao Tu, Anil Palepu, Mike Schaekermann, Khaled Saab, Jan Freyberg, Ryutaro Tanno, Amy Wang, Brenna Li, Mohamed Amin, Nenad Tomasev, Shekoofeh Azizi, Karan Singhal, Yong Cheng, Le Hou, Albert Webson, Kavita Kulkarni, S Sara Mahdavi, Christopher Semturs, Juraj Gottweis, Joelle Barral, Katherine Chou,… — [arXiv:2401.05654](https://arxiv.org/abs/2401.05654) find
  39. Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Jirong Wen. 2024. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6).
  40. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models. Conference on Neural Information Processing Systems (Neurips).
  41. Zhen Xiang, Linzhi Zheng, Yanjie Li, Junyuan Hong, Qinbin Li, Han Xie, Jiawei Zhang, Zidi Xiong, Chulin Xie, Carl Yang, Dawn Song, and Bo Li. 2024. Guardagent: Safeguard llm agents by a guard agent via knowledge-enabled reasoning. Preprint, arXiv:2406.09187. — [arXiv:2406.09187](https://arxiv.org/abs/2406.09187) find
  42. Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. 2024. Travelplanner: A benchmark for real-world planning with language agents. In International Conference on Machine Learning (ICML).
  43. Chejian Xu, Mintong Kang, Jiawei Zhang, Zeyi Liao, Lingbo Mo, Mengqi Yuan, Huan Sun, and Bo Li. 2024. Advweb: Controllable black-box attacks on vlm-powered web agents. Preprint, arXiv:2410.17401. — [arXiv:2410.17401](https://arxiv.org/abs/2410.17401) find
  44. Qisen Yang, Zekun Wang, Honghui Chen, Shenzhi Wang, Yifan Pu, Xin Gao, Wenhao Huang, Shiji Song, and Gao Huang. 2024. PsychoGAT: A novel psychological measurement paradigm through interactive fiction games with LLM agents. In Proceedings of the 62nd Annual Meeting of the Association for Computationa… find
  45. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. International Conference on Learning Representations (ICLR).
  46. Botao Yu, Frazier N. Baker, Ziru Chen, Garrett Herb, Boyu Gou, Daniel Adu-Ampratwum, Xia Ning, and Huan Sun. 2024. Tooling or not tooling? the impact of tools on language agents for chemistry problem solving. Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL). find
  47. Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang, Rui Wang, and Gongshen Liu. 2024a. R-judge: Benchmarking safety risk awareness for llm agents. Empirical Methods in Natural Language Processing (EMNLP).
  48. Zhuowen Yuan, Zidi Xiong, Yi Zeng, Ning Yu, Ruoxi Jia, Dawn Song, and Bo Li. 2024b. Rigorllm: Resilient guardrails for large language models against undesired content. Preprint, arXiv:2403.13031. find
  49. Yi Zeng, Yu Yang, Andy Zhou, Jeffrey Ziwei Tan, Yuheng Tu, Yifan Mai, Kevin Klyman, Minzhou Pan, Ruoxi Jia, Dawn Song, Percy Liang, and Bo Li. 2024. find
  50. Air-bench 2024: A safety benchmark based on risk categories from regulations and policies. Preprint, arXiv:2407.17436. find
  51. Chong Zhang, Xinyi Liu, Zhongmou Zhang, Mingyu Jin, Lingyao Li, Zhenting Wang, Wenyue Hua, Dong Shu, Suiyuan Zhu, Xiaobo Jin, Sujian Li, Mengnan Du, and Yongfeng Zhang. 2024a. When ai meets finance (stockagent): Large language model-based stock trading in simulated real-world environments. Preprint,… — [arXiv:2407.18957](https://arxiv.org/abs/2407.18957) find
  52. Hanrong Zhang, Jingyuan Huang, Kai Mei, Yifei Yao, Zhenting Wang, Chenlu Zhan, Hongwei Wang, and Yongfeng Zhang. 2025. Agent security bench (asb): Formalizing and benchmarking attacks and defenses in llm-based agents. International Conference on Learning Representations (ICLR). find
  53. Zhexin Zhang, Shiyao Cui, Yida Lu, Jingzhuo Zhou, Junxiao Yang, Hongning Wang, and Minlie Huang. 2024b. Agent-safetybench: Evaluating the safety of llm agents. Preprint, arXiv:2412.14470. — [arXiv:2412.14470](https://arxiv.org/abs/2412.14470) find
  54. Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. 2024a. Gpt-4v(ision) is a generalist web agent, if grounded. International Conference on Machine Learning (ICML).
  55. SUMMARY OF THE APPENDIX This appendix contains additional details for the “AGrail: A Lifelong AI Agent Guardrail with Effective and Adaptive Safety Detection”. The appendix is organized as follows: • §A Data Construction – A.1 Implement Details – A.2 Dataset Details – A.3 More Examples Huaixiu Steve… — Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models
  56. §B Methodology – B.1 Algorithm Details – B.2 Application Details – B.3 Prompt Configuration Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2024. Webarena: A realistic web environment for bu…
  57. §C Preliminary Study – C.1 Experiment Setting Details – C.2 Evaluation Metric Details Andrew Zhu, Lara Martin, Andrew Head, and Chris Callison-Burch. 2023a. Calypso: Llms as dungeon master’s assistants. Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainme… find
  58. §D Ablation Study – D.1 OOD and ID Analysis Details – D.2 Sequence Analysis Details – D.3 Domain Transferability Analysis – D.4 Universal Safety Criteria Analysis Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, Yu Qiao, Zhaoxiang Zh… — Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory
  59. §E Case Study – E.1 Error Analysis – E.2 Computing Cost – E.3 Experiment with Observation – E.4 Learning Analysis • §F Tool Development – F.1 OS Environment Detector – F.2 EHR Permission Detector – F.3 Web HTML Detector • §G More Examples Demo – G.1 Mind2Web-SC – G.2 EICU-AC – G.3 Safe-OS – G.4 AdvW… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/luo-2025-agrail-lifelong-agent-guardrail-effective/