Elamur Research⌕
☰
home / papers / 2026 / AutoMonitor-Bench: Evaluating the Reliability of LLM-Based Misbehavior Monitor

AutoMonitor-Bench: Evaluating the Reliability of LLM-Based Misbehavior Monitor

Paper · 2026 · Shu Yang, Jingyu Hu, Tong Li et al.

Summary

LLMs are increasingly used to check other models' outputs and reasoning traces for misbehavior, but it is unclear how reliable these monitors are. They may raise too many alarms, or they may miss subtle problems. The authors build AutoMonitor-Bench to measure this. It contains 1,505 annotated misbehavior cases, each paired with a benign counterpart, for 3,010 test items. The items cover question answering, code generation and reasoning, with and without explicit thinking traces. Monitors are scored on two metrics: Miss Rate (MR), the share of misbehavior that goes undetected, and False Alarm Rate (FAR), the share of benign outputs wrongly flagged. Across 22 LLMs, performance varies widely, and open-source models tend to miss more misbehavior. Lowering one error type usually raises the other, which points to a built-in conflict between safety and usefulness. The authors also fine-tune Qwen3-4B-Instruct-2507 on a 153,581-sample corpus. The gains mostly stay within the misbehavior types seen in training and do not carry over to subtler ones. This suggests that monitors need training approaches designed around specific tasks.

Key points

  • Three misbehavior categories: the benchmark spans three kinds of misbehavior, namely Specification Gaming, Sycophancy & Bias, and Safety & Permission Violations. Examples include insecure code (XSS), answers steered by cues in metadata, and fabricated citations.
  • Models evaluated: 12 proprietary and 10 open-source monitors were tested. Across models, MR and FAR are negatively correlated (Pearson −0.54).
  • Narrow fine-tuning backfires on unseen types: training only on Safety & Permission Violations code data cut MR on its matching test split from 0.59 to 0.00, with FAR at 0.03. However, MR on Specification Gaming code rose to 0.86.
  • QA-only training trades misses for false alarms: training on Sycophancy & Bias QA data gave low MR almost everywhere, but FAR climbed to between 0.64 and 0.93.
  • Full-corpus training still fails to generalize: using all training data lowered FAR overall, but MR on unseen Specification Gaming was 0.70, worse than the base model's 0.49. The same pattern appears for Qwen3-8B.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Renmin University of ChinaBeijing · China139 works · 17,752 citations🎓King's College LondonLondon · United Kingdom46 works · 1,714 citations🎓King Abdullah University of Science and TechnologyJeddah · Saudi Arabia30 works · 1,270 citations🎓Washington University in St. LouisSt Louis · United States15 works · 1,129 citations🎓University of BristolBristol · United Kingdom18 works · 822 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksSocial Sciences · 1,954 worksSafety Research · 626 works

Cite

Cite as Shu Yang, Jingyu Hu, Tong Li, Hanqi Yan, Wenxuan Wang and Di Wang. 2026. AutoMonitor-Bench: Evaluating the Reliability of LLM-Based Misbehavior Monitor. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.116

@inproceedings{yang2026automonitor,
  title = {{AutoMonitor-Bench: Evaluating the Reliability of LLM-Based Misbehavior Monitor}},
  author = {Shu Yang, Jingyu Hu, Tong Li, Hanqi Yan, Wenxuan Wang and Di Wang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.116},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.116}
}

References

59 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Iván Arcuschin, Jett Janiak, Robert Krzyzanowski, Senthooran Rajamanoharan, Neel Nanda, and Arthur Conmy. 2025. Chain-of-thought reasoning in the wild is not always faithful. In Workshop on Reasoning and Planning for Large Language Models. find
  2. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and 1 others. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.
  3. Xuechunzi Bai, Angelina Wang, Ilia Sucholutsky, and Thomas L Griffiths. 2024. Measuring implicit bias in explicitly unbiased large language models. In NeurIPS 2024 Workshop on Behavioral Machine Learning. find
  4. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, and 1 others. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. find
  5. Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y. Guan, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, and David Farhi. 2025. Monitoring reasoning models for misbehavior and the risks of promoting obfuscation. Preprint, arXiv:2503.11926.
  6. Yejin Bang, Ziwei Ji, Alan Schelten, Anthony Hartshorn, Tara Fowler, Cheng Zhang, Nicola Cancedda, and Pascale Fung. 2025. HalluLens: LLM hallucination benchmark. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 24128– 24156, V…
  7. Jan Betley, Daniel Chee Hian Tan, Niels Warncke, Anna Sztyber-Betley, Xuchan Bao, Martín Soto, Nathan Labenz, and Owain Evans. 2025. Emergent misalignment: Narrow finetuning can produce broadly misaligned llms. In Forty-second International Conference on Machine Learning. find
  8. Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, and 1 others. 2024. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954. find
  9. Gerard Blokdijk. 2008. ITIL IT Service Management-100 Most Asked Questions on IT Service Management and ITIL Foundation Certification, Training and Exams. Emereo Pty Ltd. find
  10. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 202… — Evaluating Large Language Models Trained on Code
  11. Xin Chen, Junchao Wu, Shu Yang, Runzhe Zhan, Zeyu Wu, Min Yang, Shujian Huang, Lidia S Chao, and Derek F Wong. 2026. Neuron-aware data selection in instruction tuning for large language models. arXiv preprint arXiv:2603.13201. — [arXiv:2603.13201](https://arxiv.org/abs/2603.13201) find
  12. Myra Cheng, Sunny Yu, Cinoo Lee, Pranav Khadpe, Lujain Ibrahim, and Dan Jurafsky. 2025. Social sycophancy: A broader understanding of llm sycophancy. arXiv preprint arXiv:2505.13995.
  13. Bilel Cherif, Tamas Bisztray, Richard A Dubniczky, Aaesha Aldahmani, Saeed Alshehhi, and Norbert Tihanyi. 2025. Dfir-metric: A benchmark dataset for evaluating large language models in digital forensics and incident response. In International Conference on Neural Information Processing, pages 17–31.… find
  14. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.
  15. Xiangxiang Cui, Shu Yang, Tianjin Huang, Wanyu Lin, Lijie Hu, and Di Wang. 2025. The compositional architecture of regret in large language models. arXiv preprint arXiv:2506.15617. — [arXiv:2506.15617](https://arxiv.org/abs/2506.15617) find
  16. Carson Denison, Monte MacDiarmid, Fazl Barez, David Duvenaud, Shauna Kravec, Samuel Marks, Nicholas Schiefer, Ryan Soklaski, Alex Tamkin, Jared Kaplan, and 1 others. 2024. Sycophancy to subterfuge: Investigating reward-tampering in large language models. arXiv preprint arXiv:2406.10162. find
  17. Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 320–335…
  18. Aaron Fanous, Jacob Goldberg, Ank Agarwal, Joanna Lin, Anson Zhou, Sonnet Xu, Vasiliki Bikia, Roxana Daneshjou, and Sanmi Koyejo. 2025. Syceval: Evaluating llm sycophancy. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 8, pages 893–900. find
  19. Yufei Guo, Muzhe Guo, Juntao Su, Zhou Yang, Mengqiu Zhu, Hongfei Li, Mengyang Qiu, and Shuo Shuo Liu. 2024. Bias in large language models: Origin, evaluation, and mitigation. arXiv preprint arXiv:2411.10915.
  20. Zikun Guo, Xinyue Xu, Pei Xiang, Shu Yang, Xin Han, Di Wang, and Lijie Hu. 2025. Benchmarking and mitigate psychological sycophancy in medical vision-language models. arXiv e-prints, pages arXiv–2509. find
  21. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In Proceedings of ICLR.
  22. Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  23. Jingyu Hu, Shu Yang, Xilin Gong, Hongming Wang, Weiru Liu, and Di Wang. 2025. Monica: Real-time monitoring and calibration of chain-of-thought sycophancy in large reasoning models. arXiv preprint arXiv:2511.06419. — [arXiv:2511.06419](https://arxiv.org/abs/2511.06419) find
  24. Xinke Jiang, Yue Fang, Rihong Qiu, Haoyu Zhang, Yongxin Xu, Hao Chen, Wentao Zhang, Ruizhe Zhang, Yuchen Fang, Xu Chu, and 1 others. 2025a. TC-RAG: Turing-complete RAG’s case study on medical LLM systems. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics.
  25. Xinke Jiang, Ruizhe Zhang, Yongxin Xu, Rihong Qiu, Yue Fang, Zhiyuan Wang, Jinyi Tang, Hongxin Ding, Xu Chu, Junfeng Zhao, and 1 others. 2025b. HyKGE: A hypothesis knowledge graph enhanced framework for accurate and reliable medical LLM responses. In Proceedings of the 63rd Annual Meeting of the Ass… find
  26. Tomek Korbak, Mikita Balesni, Elizabeth Barnes, Yoshua Bengio, Joe Benton, Joseph Bloom, Mark Chen, Alan Cooney, Allan Dafoe, Anca Dragan, and 1 others. 2025. Chain of thought monitorability: A new and fragile opportunity for ai safety. arXiv preprint arXiv:2507.11473. find
  27. Tong Li, Shu Yang, Junchao Wu, Jiyao Wei, Lijie Hu, Mengdi Li, Derek F Wong, Joshua R Oltmanns, and Di Wang. 2025. Can large language models identify implicit suicidal ideation? an empirical evaluation. arXiv preprint arXiv:2502.17899. find
  28. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics…
  29. Aengus Lynch, Benjamin Wright, Caleb Larson, Kevin K. Troy, Stuart J. Ritchie, Sören Mindermann, Ethan Perez, and Evan Hubinger. 2025. Agentic misalignment: How llms could be an insider threat. Anthropic Research. Https://www.anthropic.com/research/agentic-misalignment. find
  30. Johnston, and 1 others. 2023. Towards understanding sycophancy in language models. arXiv preprint arXiv:2310.13548.
  31. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Esin DURMUS, Zac Hatfield-Dodds, Scott R Johnston, Shauna M Kravec, and 1 others. 2024. Towards understanding sycophancy in language models. In The Twelfth International Conference on Learning Representations.
  32. Miles Turpin, Andy Arditi, Marvin Li, Joe Benton, and Julian Michael. 2025. Teaching models to verbalize reward hacking in chain-of-thought reasoning. arXiv preprint arXiv:2506.22777. — [arXiv:2506.22777](https://arxiv.org/abs/2506.22777) find
  33. Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. 2023. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems, 36:74952–74965.
  34. UK Foundation Programme Office. 2025. F2 stand-alone 2025 – situational judgement test (sjt). https://foundationprogramme.nhs.uk/ programmes/f2-stand-alone/f2sa2025-sjt/. Accessed 2026-01-05. ↗
  35. Emanuele Marconato, Samuele Bortolotti, Emile van Krieken, Antonio Vergari, Andrea Passerini, and Stefano Teso. 2024. Bears make neuro-symbolic models aware of their reasoning shortcuts. Preprint, arXiv:2402.12240. — [arXiv:2402.12240](https://arxiv.org/abs/2402.12240) find
  36. Sydney Von Arx, Lawrence Chan, and Elizabeth Barnes. 2025. Recent frontier models are reward hacking. https://metr.org/blog/ 2025-06-05-recent-reward-hacking/. METR Blog. ↗
  37. Mathematical Association of America. 2025. American invitational mathematics examination (aime). https://www.maa.org/math-competitions. Accessed: 2026-01-05. ↗
  38. Jianwei Wang, Mengqi Wang, Yinsi Zhou, Zhenchang Xing, Qing Liu, Xiwei Xu, Wenjie Zhang, and Liming Zhu. 2025a. Llm-based hse compliance assessment: Benchmark, performance, and advancements. arXiv preprint arXiv:2505.22959. — [arXiv:2505.22959](https://arxiv.org/abs/2505.22959) find
  39. Dagmara Panas, Ali Payani, and Vaishak Belle. 2025. Unreasonable effectiveness of LLM reasoning: a doubly cautionary tale of temporal question-answering. Transactions on Machine Learning Research. find
  40. David Rein and 1 others. 2023. GPQA: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022.
  41. Aswin Rrv, Nemika Tyagi, Md Nayem Uddin, Neeraj Varshney, and Chitta Baral. 2024. Chaos with keywords: Exposing large language models sycophancy to misleading keywords and evaluating defense strategies. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12717–12733, Bangko… find
  42. Yijia Shao, Humishka Zope, Yucheng Jiang, Jiaxin Pei, David Nguyen, Erik Brynjolfsson, and Diyi Yang. 2025. Future of work with ai agents: Auditing automation and augmentation potential across the us workforce. arXiv preprint arXiv:2506.06576. find
  43. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R Keyu Wang, Jin Li, Shu Yang, Zhuoran Zhang, and Di Wang. 2025b. When truth is overridden: Uncovering the internal origins of sycophancy in large language mo… — [arXiv:2508.02087](https://arxiv.org/abs/2508.02087) find
  44. Yixu Wang, Yan Teng, Kexin Huang, Chengqi Lyu, Songyang Zhang, Wenwei Zhang, Xingjun Ma, Yu-Gang Jiang, Yu Qiao, and Yingchun Wang. 2024. Fake alignment: Are llms really aligned well? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics… find
  45. Junchao Wu, Runzhe Zhan, Derek F Wong, Shu Yang, Xuebo Liu, Lidia S Chao, and Min Zhang. 2025. Who wrote this? the key to zero-shot LLM-generated text detection is GECScore. In Proceedings of the 31st International Conference on Computational Linguistics, pages 10275–10292.
  46. Junchao Wu, Runzhe Zhan, Derek F Wong, Shu Yang, Xinyi Yang, Yulin Yuan, and Lidia S Chao. 2024. DetectRL: Benchmarking LLM-generated text detection in real-world scenarios. Advances in Neural Information Processing Systems, 37:100369–100401. find
  47. Haochuan Xu, Yun Sing Koh, Shuhuai Huang, Zirun Zhou, Di Wang, Jun Sakuma, and Jingfeng Zhang. 2025. Model-agnostic adversarial attack and defense for vision-language-action models. arXiv preprint arXiv:2510.13237. — [arXiv:2510.13237](https://arxiv.org/abs/2510.13237) find
  48. Shu Yang, Junchao Wu, Xin Chen, Yunze Xiao, Xinyi Yang, Derek F Wong, and Di Wang. 2025a. Understanding aha moments: from external observations to internal mechanisms. arXiv preprint arXiv:2504.02956. — [arXiv:2504.02956](https://arxiv.org/abs/2504.02956) find
  49. Shu Yang, Junchao Wu, Xilin Gou, Xuansheng Wu, Derek Wong, Ninhao Liu, and Di Wang. 2025b. Investigating cot monitorability in large reasoning models. arXiv preprint arXiv:2511.08525. modalities conflict: How unimodal reasoning uncertainty governs preference dynamics in mllms. arXiv preprint arXiv:2… — [arXiv:2511.08525](https://arxiv.org/abs/2511.08525) find
  50. Wenrui Zhou, Mohamed Hendy, Shu Yang, Qingsong Yang, Zikun Guo, Yuyu Luo, Lijie Hu, and Di Wang. 2025a. Flattery in motion: Benchmarking and analyzing sycophancy in video-llms. arXiv preprint arXiv:2506.07180.
  51. Zirun Zhou, Zhengyang Xiao, Haochuan Xu, Jing Sun, Di Wang, and Jingfeng Zhang. 2025b. Goal-oriented backdoor attack against vision-language-action models via physical objects. arXiv preprint arXiv:2510.09269. — [arXiv:2510.09269](https://arxiv.org/abs/2510.09269) find
  52. Shu Yang, Shenzhe Zhu, Zeyu Wu, Keyu Wang, Junchi Yao, Junchao Wu, Lijie Hu, Mengdi Li, Derek F Wong, and Di Wang. 2025c. Fraud-r1: A multi-round benchmark for assessing the robustness of llm against augmented fraud and phishing inducements. arXiv preprint arXiv:2502.12904. find
  53. Tiancheng Yang, Lin Zhang, Jiaye Lin, Guimin Hu, Di Wang, and Lijie Hu. 2025d. D-leaf: Localizing and correcting hallucinations in multimodal llms via layer-to-head attention diagnostics. arXiv e-prints, pages arXiv–2509. find
  54. Junchi Yao, Jianhua Xu, Tianyu Xin, Ziyi Wang, Shenzhe Zhu, Shu Yang, and Di Wang. 2025. Is your llm-based multi-agent a reliable real-world planner? exploring fraud detection in travel planning. arXiv preprint arXiv:2505.16557. — [arXiv:2505.16557](https://arxiv.org/abs/2505.16557) find
  55. Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. 2024. A survey on large language model (llm) security and privacy: The good, the bad, and the ugly. High-Confidence Computing, 4(2):100211.
  56. Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, and 1 others. 2025. Justice or prejudice? quantifying biases in llm-as-a-judge. In International Conference on Learning Representations.
  57. Yu Yuan, Lili Zhao, Kai Zhang, Guangting Zheng, and Qi Liu. 2024. Do llms overcome shortcut learning? an evaluation of shortcut challenges in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12188–12200. find
  58. Ruizhe Zhang, Xinke Jiang, Zhibang Yang, Zhixin Zhang, Jiaran Gao, Yuzhen Xiao, Hongbin Lai, Xu Chu, Junfeng Zhao, and Yasha Wang. 2026. StackPlanner: A centralized hierarchical multi-agent system with task-experience memory management. In Proceedings of the 64th Annual Meeting of the Association fo… find
  59. Zhuoran Zhang, Tengyue Wang, Xilin Gong, Yang Shi, Haotian Wang, Di Wang, and Lijie Hu. 2025. When find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2026-automonitor-bench-evaluating-reliability-llm/