Elamur Research⌕
☰
home / papers / 2026 / SafeMCP: Proactive Power Regulation for LLM Agent Defense via Environment-Grounded Look-Ahead Reasoning

SafeMCP: Proactive Power Regulation for LLM Agent Defense via Environment-Grounded Look-Ahead Reasoning

Paper · 2026 · Lichao Wang, ZhaoXing Ren, Tianzhuo Yang et al.

Summary

When LLM agents use the Model Context Protocol (MCP), they can acquire new tools automatically. This widens their action space and encourages power-seeking: the agent moves toward privileged states, such as admin access or a disabled firewall, where one error or hallucination can become catastrophic. Current guardrails judge each action's surface meaning after the agent picks it. They ignore how the environment will evolve and tend to stop the workflow outright.

The authors model the defender and the agent as leader and follower in a Cooperative Stackelberg Power Game, where "power" is the set of tools usable in a given state. Their plugin, SafeMCP, runs on the MCP server and uses a learned world model to forecast the next state and judge its safety. It then removes tools that could lead to unsafe states and, as a fallback, blocks dangerous calls at the current step.

Training has three stages: environment dynamic grounding, safe policy initialization, and RL with two verifiable rewards combined via Smooth Tchebycheff (STCH). On PowerSeeking Bench, ToolEmu and AgentHarm, SafeMCP reduces risk with little over-blocking. It is an agent-agnostic, plug-and-play safeguard.

Key points

  • Formal framing: the paper proves that a Safe Stackelberg Equilibrium exists, given a finite action set and a safety-viability assumption.
  • Runtime-risk evaluation: even on the weaker GPT-4o-mini backbone, SafeMCP's safety rate hits 0.98, and its Libra score is 0.40 versus RL-Guard's 0.35.
  • AgentHarm: SafeMCP gets the top Libra Score of 0.83 on GPT-4o, while wrongly blocking only 0.01 of benign requests. Most baselines block every harmful case but over-refuse benign ones.
  • Ablations on AgentHarm:
    • Removing Stage 3 (the RL stage) raises the harmful score from 0.19 to 0.36.
    • Replacing STCH with sparse rewards drops the benign score from 0.69 to 0.59.
  • Released resources: the environment dynamic grounding dataset, PowerSeeking Bench, and the training code. All evaluations ran in simulated sandboxes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China244 works · 39,359 citations🔬Beijing Academy of Artificial IntelligenceBeijing · China32 works · 7,511 citations🎓Beijing Institute of TechnologyBeijing · China34 works · 3,652 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksHardware and Architecture · 81 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Lichao Wang, ZhaoXing Ren, Tianzhuo Yang, Jiaming Ji, Chi Harold Liu, Yaodong Yang and Juntao Dai. 2026. SafeMCP: Proactive Power Regulation for LLM Agent Defense via Environment-Grounded Look-Ahead Reasoning. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.522

@inproceedings{wang2026safemcp,
  title = {{SafeMCP: Proactive Power Regulation for LLM Agent Defense via Environment-Grounded Look-Ahead Reasoning}},
  author = {Lichao Wang, ZhaoXing Ren, Tianzhuo Yang, Jiaming Ji, Chi Harold Liu, Yaodong Yang and Juntao Dai},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.522},
  url = {https://doi.org/10.18653/v1/2026.acl-long.522}
}

References

38 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2024a. Claude 3.5 Sonnet. Accessed: 2026-01-01. find
  2. Anthropic. 2024b. Model Context Protocol (MCP). https://github.com/modelcontextprotocol. Accessed: 2025-12-29. ↗
  3. Joseph Carlsmith. 2024. Is Power-Seeking AI an Existential Risk? Preprint, arXiv:2206.13353. — [arXiv:2206.13353](https://arxiv.org/abs/2206.13353) find
  4. Zhaorun Chen, Mintong Kang, and Bo Li. 2025. Shield-agent: Shielding agents via verifiable safety policy reasoning. arXiv preprint arXiv:2503.22738. — [arXiv:2503.22738](https://arxiv.org/abs/2503.22738) find
  5. Zhi-Yuan Chen, Shiqi Shen, Guangyao Shen, Gong Zhi, Xu Chen, and Yankai Lin. 2024. Towards tool use alignment of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1382–1400, Miami, Florida, USA. Association for Computational Lingu… find
  6. Sahana Chennabasappa, Cyrus Nikolaidis, Daniel Song, David Molnar, Stephanie Ding, Shengye Wan, Spencer Whitman, Lauren Deason, Nicholas Doucette, Abraham Montilla, Alekhya Gampa, Beto de Paola, Dominik Gabi, James Crnkovich, Jean-Christophe Testud, Kat He, Rashnil Chaturvedi, Wu Zhou, and Joshua Sa… — [arXiv:2505.03574](https://arxiv.org/abs/2505.03574) find
  7. Sadia Sultana Chowa, Riasad Alvi, Subhey Sadi Rahman, Md Abdur Rahman, Mohaimenul Azam Khan Raiaan, Md Rafiqul Islam, Mukhtar Hussain, and Sami Azam. 2026. From language to action: a review of large language models as autonomous agents and tool users. Artificial Intelligence Review, 59(2). find
  8. Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. 2024. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Bench… find
  9. Junfeng Fang, Zijun Yao, Ruipeng Wang, Haokai Ma, Xiang Wang, and Tat-Seng Chua. 2025. We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systems. Preprint, arXiv:2506.13666. — [arXiv:2506.13666](https://arxiv.org/abs/2506.13666) find
  10. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, and 175 others. 2025. DeepSeek-R1 incentivizes reasoning in LLMs through rein… find
  11. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming – The Rise of Code Intelligence. Preprint, arXiv:2401.14196. find
  12. Yue Huang, Hang Hua, Yujun Zhou, Pengcheng Jing, Manish Nagireddy, Inkit Padhi, Greta Dolcetti, Zhangchen Xu, Subhajit Chaudhury, Ambrish Rawat, Liubov Nedoshivina, Pin-Yu Chen, Prasanna Sattigeri, and Xiangliang Zhang. 2025. Building a Foundational Guardrail for General Agentic Systems via Syntheti… — [arXiv:2510.09781](https://arxiv.org/abs/2510.09781) find
  13. Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. 2023. Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations. Preprint, arXiv:2312.06674.
  14. Xi Lin, Xiaoyuan Zhang, Zhiyuan Yang, Fei Liu, Zhenkun Wang, and Qingfu Zhang. 2024. Smooth Tchebycheff scalarization for multi-objective optimization. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org. find
  15. Weidi Luo, Shenghong Dai, Xiaogeng Liu, Suman Banerjee, Huan Sun, Muhao Chen, and Chaowei Xiao. 2025. AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detection. Preprint, arXiv:2502.11448.
  16. Margaret Mitchell, Avijit Ghosh, Alexandra Sasha Luccioni, and Giada Pistilli. 2025. Fully Autonomous AI Agents Should Not be Developed. Preprint, arXiv:2502.02649. — [arXiv:2502.02649](https://arxiv.org/abs/2502.02649) find
  17. Silen Naihin, David Atkinson, Marc Green, Mer-wane Hamadi, Craig Swift, Douglas Schonholtz, Adam Tauman Kalai, and David Bau. 2023. Testing Language Model Agents Safely in the Wild. Preprint, arXiv:2311.10538. — [arXiv:2311.10538](https://arxiv.org/abs/2311.10538) find
  18. OpenRouter. 2024. OpenRouter: A unified interface for LLMs. https://openrouter.ai/. Accessed: 2026-01-03. ↗
  19. Traian Rebedea, Razvan Dinu, Makesh Narsimhan Sreedhar, Christopher Parisien, and Jonathan Cohen. 2023. NeMo guardrails: A toolkit for controllable and safe LLM applications with programmable rails. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Dem… find
  20. Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J Maddison, and Tatsunori Hashimoto. 2024. Identifying the Risks of LM Agents with an LM-Emulated Sandbox. In The Twelfth International Conference on Learning Representations. find
  21. Gemini Team. 2025. Gemini: A Family of Highly Capable Multimodal Models. Preprint, arXiv:2312.11805.
  22. Lakera Team. 2024a. ChainGuard: Guard Your LangChain Apps with Lakera. Blog post. Retrieved from Lakera website. find
  23. LLaMA3 Team. 2024b. The Llama 3 Herd of Models. Preprint, arXiv:2407.21783.
  24. OpenAI Team. 2024c. GPT-4o System Card. ArXiv, abs/2410.21276.
  25. Alex Turner, Logan Smith, Rohin Shah, Andrew Critch, and Prasad Tadepalli. 2021. Optimal Policies Tend To Seek Power. In Advances in Neural Information Processing Systems, volume 34, pages 23063–23074. Curran Associates, Inc. find
  26. Heinrich von Stackelberg. 2010. Market Structure and Equilibrium. Springer Science & Business Media. find
  27. Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Jirong Wen. 2024. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345.
  28. Zhen Xiang, Linzhi Zheng, Yanjie Li, Junyuan Hong, Qinbin Li, Han Xie, Jiawei Zhang, Zidi Xiong, Chulin Xie, Carl Yang, Dawn Song, and Bo Li. 2024. GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning. Preprint, arXiv:2406.09187. — [arXiv:2406.09187](https://arxiv.org/abs/2406.09187) find
  29. Yijun Yang, Lichao Wang, Chi Harold Liu, Qiang Xu, and Xiao Yang. 2024. RL-Guard: Rescuing LLM Agents from Pitfalls. CoRR. find
  30. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In The Eleventh International Conference on Learning Representations (ICLR).
  31. Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang, Rui Wang, and Gongshen Liu. 2024. R-Judge: Benchmarking Safety Risk Awareness for LLM Agents. In Conference on Empirical Methods in Natural Language Processing.
  32. Hanrong Zhang, Jingyuan Huang, Kai Mei, Yifei Yao, Zhenting Wang, Chenlu Zhan, Hongwei Wang, and Yongfeng Zhang. 2024. Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents. ArXiv, abs/2410.02644. — [arXiv:2410.02644](https://arxiv.org/abs/2410.02644) find
  33. Jinchuan Zhang, Lu Yin, Yan Zhou, and Songlin Hu. 2025a. AgentAlign: Navigating Safety Alignment in the Shift from Informative to Agentic Large Language Models. Preprint, arXiv:2505.23020. — [arXiv:2505.23020](https://arxiv.org/abs/2505.23020) find
  34. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. OPT: Open Pre-t…
  35. Zhexin Zhang, Shiyao Cui, Yida Lu, Jingzhuo Zhou, Junxiao Yang, Hongning Wang, and Minlie Huang. 2025b. Agent-SafetyBench: Evaluating the Safety of LLM Agents. Preprint, arXiv:2412.14470. — [arXiv:2412.14470](https://arxiv.org/abs/2412.14470) find
  36. Geng Zhao, Banghua Zhu, and Jiantao Jiao. 2023. Online learning in stackelberg games with an omniscient follower. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org. find
  37. Haiquan Zhao, Chenhan Yuan, Fei Huang, Xiaomeng Hu, Yichang Zhang, An Yang, Bowen Yu, Dayiheng Liu, Jingren Zhou, Junyang Lin, Baosong Yang, Chen Cheng, Jialong Tang, Jiandong Jiang, Jianwei Zhang, Jijie Xu, Ming Yan, Minmin Sun, Pei Zhang, and 24 others. 2025. Qwen3Guard Technical Report. Preprint,…
  38. Peter Yong Zhong, Siyuan Chen, Ruiqi Wang, McKenna McCall, Ben L. Titzer, Heather Miller, and Phillip B. Gibbons. 2025. Rtbas: Defending llm agents against prompt injection and privacy leakage. Preprint, arXiv:2502.08966. — [arXiv:2502.08966](https://arxiv.org/abs/2502.08966) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-safemcp-proactive-power-regulation-llm/