Elamur Research⌕
☰
home / papers / 2026 / AgenticEval: Toward Agentic and Self-Evolving Safety Evaluation of Large Language Models

AgenticEval: Toward Agentic and Self-Evolving Safety Evaluation of Large Language Models

Paper · 2026 · Yixu Wang, Xin Wang, Yang Yao et al.

Summary

Static safety benchmarks for LLMs age quickly. They struggle to reflect complex legal requirements and are hard to adapt to an organization's own policies. AgenticEval addresses this by treating safety evaluation as an ongoing, self-updating process rather than a single audit. In this multi-agent framework:

  • A Specialist agent breaks an unstructured regulation into a hierarchy of atomic rules, either autonomously or following a user-supplied JSON template. It then uses web search to attach examples of compliant and violating content, producing a Testable Knowledge Base.
  • A Generator agent builds Question Groups per rule, which probe each rule from several angles, including multimodal and adversarial variants.
  • An Evaluator agent and an Analyst agent run a Self-evolving Evaluation loop. The Analyst studies the target model's failures and steers the Generator toward harder, more targeted tests.

Across 11 models and three regulatory frameworks, safety rates fall steadily as the loop progresses. The full system also beats ablated variants and generic jailbreak baselines at exposing weaknesses. This suggests that one-off benchmarks overstate how compliant models really are.

Key points

  • Regulations tested: EU AI Act, NIST AI RMF, and MAS FEAT, with 11 target models evaluated.
  • On the EU AI Act, GPT-5's safety rate fell from 72.50% to 36.36% over successive iterations.
  • Ablations: removing the refinement loop raised GPT-5's final EU AI Act score to 48.60%. Removing rule decomposition raised its NIST AI RMF score from 64.29% to 75.40%.
  • Reliability of the Evaluator agent was checked on 100 human-annotated cases per framework. It reached 88.33–91.00% accuracy, 87.87–89.79% F1, and Cohen's κ of 0.77–0.81.
  • Stated limitations:
    • The rule extraction is a technical reading, not a legal judgment.
    • Accuracy depends on the backbone model.
    • The loop costs more compute than static benchmarks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China167 works · 20,987 citations🎓University of Hong KongPok Fu Lam · China89 works · 11,774 citations🔬Shanghai Artificial Intelligence LaboratoryShanghai · China117 works · 11,281 citations🎓East China Normal UniversityShanghai · China50 works · 2,174 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSafety Research · 543 worksEngineering · 133 worksSafety, Risk, Reliability and Quality · 10 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksBias, fairness & safety · 517 works

Cite

Cite as Yixu Wang, Xin Wang, Yang Yao, Xinyuan Li, Xibang Yang, Yan Teng, Xingjun Ma and Yingchun Wang. 2026. AgenticEval: Toward Agentic and Self-Evolving Safety Evaluation of Large Language Models. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.727

@inproceedings{wang2026agenticeval,
  title = {{AgenticEval: Toward Agentic and Self-Evolving Safety Evaluation of Large Language Models}},
  author = {Yixu Wang, Xin Wang, Yang Yao, Xinyuan Li, Xibang Yang, Yan Teng, Xingjun Ma and Yingchun Wang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.727},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.727}
}

References

31 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. EU Artificial Intelligence Act. 2024. The eu artificial intelligence act. European Union. find
  2. Ali Asad, Stephen Obadinma, Radin Shayanfar, and Xiaodan Zhu. 2025. Reddebate: Safer responses through multi-agent red teaming debates. arXiv preprint arXiv:2506.11083. — [arXiv:2506.11083](https://arxiv.org/abs/2506.11083) find
  3. Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, and 1 others. 2023. Meditron-70b: Scaling medical pretraining for large language models. arXiv preprint arXiv:2311.16079.
  4. Zhaorun Chen, Mintong Kang, and Bo Li. 2025. Shieldagent: Shielding agents via verifiable safety policy reasoning. arXiv preprint arXiv:2503.22738. — [arXiv:2503.22738](https://arxiv.org/abs/2503.22738) find
  5. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arX…
  6. DeepSeek-AI. 2024. Deepseek-v3 technical report. Preprint, arXiv:2412.19437. find
  7. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  8. Philipp Guldimann, Alexander Spiridonov, Robin Staab, Nikola Jovanović, Mark Vero, Velko Vechev, Anna-Maria Gueorguieva, Mislav Balunović, Nikola Konstantinov, Pavol Bielik, and 1 others. 2024. Compl-ai framework: A technical interpretation and llm benchmarking suite for the eu artificial intellig… — [arXiv:2410.07959](https://arxiv.org/abs/2410.07959) find
  9. Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. 2024. MetaGPT: Meta programming for a multi-agent collaborative framework. In ICLR.
  10. Geoffrey Irving, Paul Christiano, and Dario Amodei. 2018. Ai safety via debate. arXiv preprint arXiv:1805.00899. find
  11. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, and 1 others. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110. find
  12. Xiaogeng Liu, Peiran Li, Edward Suh, Yevgeniy Vorobeychik, Zhuoqing Mao, Somesh Jha, Patrick Mc-Daniel, Huan Sun, Bo Li, and Chaowei Xiao. 2025. find
  13. Autodan-turbo: A lifelong agent for strategy self-exploration to jailbreak llms. In ICLR. Xingjun Ma, Yifeng Gao, Yixu Wang, Ruofan Wang, Xin Wang, Ye Sun, Yifan Ding, Hengyuan Xu, Yunhao Chen, Yunhan Zhao, and 1 others. 2025. Safety at scale: A comprehensive survey of large model safety. arXiv prep… — [arXiv:2502.05206](https://arxiv.org/abs/2502.05206) find
  14. Meta AI. 2025. The Llama 4 Herd of Models. find
  15. Monetary Authority of Singapore. 2018. Principles to promote fairness, ethics, accountability and transparency (FEAT) in the use of artificial intelligence and data analytics in singapore’s financial sector. Technical report, Monetary Authority of Singapore. find
  16. Qianqian Xie, Qingyu Chen, Aokun Chen, Cheng Peng, Yan Hu, Fongci Lin, Xueqing Peng, Jimin Huang, Jeffrey Zhang, Vipina Keloth, and 1 others. 2024. Me-llama: Foundation large language models for medical applications. Research square.
  17. Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. 2023. Pixiu: A large language model, instruction data and evaluation benchmark for finance. arXiv preprint arXiv:2306.05443. — [arXiv:2306.05443](https://arxiv.org/abs/2306.05443) find
  18. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  19. Tai D Nguyen, Long H Pham, and Jun Sun. 2025. Autolaw: Enhancing legal compliance in large language models via case law generation and jury-inspired deliberation. arXiv preprint arXiv:2505.14015. find
  20. Yi Yang, Yixuan Tang, and Kar Yan Tam. 2023. Investlm: A large language model for investment using financial domain instruction tuning. arXiv preprint arXiv:2309.13064. — [arXiv:2309.13064](https://arxiv.org/abs/2309.13064) find
  21. OpenAI. 2025. Gpt-5 system card. Technical report. find
  22. Fan Zhang, Shulin Tian, Ziqi Huang, Yu Qiao, and Ziwei Liu. 2025. Evaluation agent: Efficient and promptable evaluation framework for visual generative models. In ACL. find
  23. Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, and 1 others. 2024. A strongreject for empty jailbreaks. In NeurIPS. find
  24. Elham Tabassi. 2023. Artificial intelligence risk management framework (ai rmf 1.0). find
  25. V Vendetti, LD Comencini, F Deriu, V Modugno, and 1 others. 2025. Passing the turing test in political discourse: Fine-tuning llms to mimic polarized social media comments. arXiv preprint arXiv:2506.14645. — [arXiv:2506.14645](https://arxiv.org/abs/2506.14645) find
  26. Jingnan Zheng, Han Wang, An Zhang, Nguyen Duy Tai, Jun Sun, and Tat-Seng Chua. 2024. Ali-agent: Assessing llms’ alignment with human values via agent-based evaluation. NeurIPS. find
  27. Andy Zhou, Kevin Wu, Francesco Pinto, Zhaorun Chen, Yi Zeng, Yu Yang, Shuang Yang, Sanmi Koyejo, James Zou, and Bo Li. 2025. Autoredteamer: Autonomous red teaming with lifelong attack integration. arXiv preprint arXiv:2503.15754. — [arXiv:2503.15754](https://arxiv.org/abs/2503.15754) find
  28. Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, and 1 others. 2023. Decodingtrust: A comprehensive assessment of trustworthiness in gpt models. In NeurIPS Datasets and Benchmarks Track.
  29. Kun Wang, Guibin Zhang, Zhenhong Zhou, Jiahao Wu, Miao Yu, Shiqian Zhao, Chenlong Yin, Jinhu Fu, Yibo Yan, Hanjun Luo, and 1 others. 2025a. A comprehensive survey in llm (-agent) full stack safety: Data, training and deployment. arXiv preprint arXiv:2504.15585. — [arXiv:2504.15585](https://arxiv.org/abs/2504.15585) find
  30. Maggie Wang, Ella Colby, Jennifer Okwara, Varun Nagaraj Rao, Yuhan Liu, and Andrés Monroy-Hernández. 2025b. Policypulse: Llm-synthesis tool for policy researchers. In CHI. find
  31. Yixu Wang, Yan Teng, Kexin Huang, Chengqi Lyu, Songyang Zhang, Wenwei Zhang, Xingjun Ma, Yu-Gang Jiang, Yu Qiao, and Yingchun Wang. 2024. Fake alignment: Are llms really aligned well? In NAACL. xAI. 2025. Grok-4 Technical Report. Technical report. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-agenticeval-agentic-self-evolving-safety/