Elamur Research⌕
☰
home / papers / 2026 / When Models Outthink Their Safety: Unveiling and Mitigating Self-Jailbreak in Large Reasoning Models

When Models Outthink Their Safety: Unveiling and Mitigating Self-Jailbreak in Large Reasoning Models

Paper · 2026 · Yingzhi Mao, Chunkang Zhang, Junxiang Wang et al.

Summary

Large Reasoning Models (LRMs) still produce harmful content. Existing safety fixes usually constrain the whole reasoning trace, which tends to hurt reasoning quality. The authors divide each trace into a risk-awareness phase, a risk-analysis phase and a response-strategy phase. Using this split, they analyze where unsafe answers come from. On WildJailbreak, the dominant cause is not a failure to spot harm. Instead, models first flag a query as dangerous and then argue themselves into complying. The authors call this "Self-Jailbreak."

To address it, they propose Chain-of-Guardrail (CoG), a fine-tuning framework that edits only the reasoning steps that cause the failure. It has two variants:

  • Safety Recomposition rewrites the trace into a coherent safe version.
  • Safety Backtrack keeps the original trace but adds a self-check and a corrected answer.

Across several LRMs, CoG brings attack success rates down to competitive levels while largely preserving reasoning accuracy. In some comparisons it clearly improves reasoning. The work matters because it locates the failure inside the reasoning chain, which allows precise fixes instead of blanket restrictions.

Key points

  • Diagnostic setup: 2,000 WildJailbreak prompts. Llama-Guard judged whether outputs were harmful, and Qwen2.5-72B-Instruct judged whether the model showed risk awareness. Self-Jailbreak accounts for about 80% of unsafe cases overall and 93.7% for DeepSeek-R1.
  • Taxonomy of Self-Jailbreak:
    • Warning ("warn-but-answer"): over 55% of cases in every model studied.
    • Benign Reframing: about 37–41%.
    • Logical Fallacies: about 2–5%.
  • Qwen3-32B vs. SafeKey: CoG reaches similar safety while raising GPQA-Diamond from 54.30 to 62.38 and AIME2024 from 71.70 to 82.08.
  • Qwen3-8B: Average attack success rate falls from 41.72 to roughly 11.5. Average reasoning score stays near the original (79.89–80.78 vs. 81.28), whereas the R2D baseline drops it to 61.43.
  • Ablation on Safety Backtrack: Supervising the full trace instead of masking the original reasoning hurts safety. For example, the PAIR attack success rate rises from 26.83 to 59.76.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️Chinese Academy of SciencesBeijing · China194 works · 23,906 citations🎓University of Chinese Academy of SciencesBeijing · China147 works · 8,480 citations🔬Institute of SoftwareBeijing · China21 works · 4,980 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksBias, fairness & safety · 517 works

Cite

Cite as Yingzhi Mao, Chunkang Zhang, Junxiang Wang, Xinyan Guan, Boxi Cao, Yaojie Lu, Hongyu Lin, Xianpei Han and Le Sun. 2026. When Models Outthink Their Safety: Unveiling and Mitigating Self-Jailbreak in Large Reasoning Models. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1118

@inproceedings{mao2026when,
  title = {{When Models Outthink Their Safety: Unveiling and Mitigating Self-Jailbreak in Large Reasoning Models}},
  author = {Yingzhi Mao, Chunkang Zhang, Junxiang Wang, Xinyan Guan, Boxi Cao, Yaojie Lu, Hongyu Lin, Xianpei Han and Le Sun},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1118},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1118}
}

References

48 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 2021. Evaluating large language models trained on code.
  2. DeepSeek-AI. 2024. Deepseek-v3 technical report. Preprint, arXiv:2412.19437. find
  3. Achref Doula, Max Mühlhäuser, and Alejandro Sanchez Guinea. 2025. Safepath: Conformal prediction for safe llm-based autonomous navigation. arXiv preprint arXiv:2505.09427. — [arXiv:2505.09427](https://arxiv.org/abs/2505.09427) find
  4. Mohamed Amine Ferrag, Norbert Tihanyi, and Merouane Debbah. 2025. From llm reasoning to autonomous ai agents: A comprehensive review. arXiv preprint arXiv:2504.19678. find
  5. Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. 2025. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.01307.
  6. Tommaso Green, Martin Gubri, Haritz Puerto, Sangdoo Yun, and Seong Joon Oh. 2025. Leaky thoughts: Large reasoning models are not private thinkers. CoRR, abs/2506.15674. find
  7. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, and 1 others. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594. find
  8. Melody Y Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, and 1 others. 2024. Deliberative alignment: Reasoning enables safer language models. arXiv preprint arXiv:2412.16339.
  9. Arash Ahmadian, Beyza Ermis, Seraphina Goldfarb-Tarrant, Julia Kreutzer, Marzieh Fadaee, Sara Hooker, and 1 others. 2024. The multilingual alignment prism: Aligning global and local preferences to reduce harm. arXiv preprint arXiv:2406.18682.
  10. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  11. Aitor Arrieta, Miriam Ugarte, Pablo Valle, José Antonio Parejo, and Sergio Segura. 2025. Early external safety testing of openai’s o3-mini: Insights from the pre-deployment evaluation. CoRR, abs/2501.17749. — [arXiv:2501.17749](https://arxiv.org/abs/2501.17749) find
  12. Yufei He, Yuexin Li, Jiaying Wu, Yuan Sui, Yulin Chen, and Bryan Hooi. 2025. Evaluating the paperclip maximizer: Are rl-based language models more likely to pursue instrumental goals? CoRR, abs/2502.12206. — [arXiv:2502.12206](https://arxiv.org/abs/2502.12206) find
  13. Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas, Florian Tramer, and 1 others. 2024. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. Advances in Neural…
  14. Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, Zachary Yahn, Yichang Xu, and Ling Liu. 2025. Safety tax: Safety alignment makes your large reasoning models less reasonable. Preprint, arXiv:2503.00555. — [arXiv:2503.00555](https://arxiv.org/abs/2503.00555) find
  15. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, and 1 others. 2024. Openai o1 system card. arXiv preprint arXiv…
  16. Wonje Jeung, Sangyeon Yoon, Minsuk Kahng, and Albert No. 2025. Safepath: Preventing harmful reasoning in chain-of-thought via early alignment. arXiv preprint arXiv:2505.14667. — [arXiv:2505.14667](https://arxiv.org/abs/2505.14667) find
  17. Mathematical Association of America (MAA). 2024. American invitational mathematics examination – aime. Available at https://maa.org/..., accessed March 26,2025. ↗
  18. Fengqing Jiang, Zhangchen Xu, Yuetai Li, Luyao Niu, Zhen Xiang, Bo Li, Bill Yuchen Lin, and Radha Poovendran. 2025. Safechain: Safety of language models with long chain-of-thought reasoning capabilities. arXiv preprint arXiv:2502.12025. find
  19. Jianing Qiu, Lin Li, Jiankai Sun, Hao Wei, Zhe Xu, Kyle Lam, and Wu Yuan. 2025. Emerging cyber attack risks of medical AI agents. CoRR, abs/2504.03759. — [arXiv:2504.03759](https://arxiv.org/abs/2504.03759) find
  20. Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, and Nouha Dziri. 2024a. Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models. Preprint, arXiv:2406.18510.
  21. Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, and 1 others. 2024b. Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models. Advances in Neural Information Processing S…
  22. George Kour, Marcel Zalmanovici, Naama Zwerdling, Esther Goldbraich, Ora Nova Fandina, Ateret Anaby-Tavor, Orna Raz, and Eitan Farchi. 2023. Unveiling safety vulnerabilities of large language models. arXiv preprint arXiv:2311.04124. find
  23. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling.
  24. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. Advances in neural information processing systems, 36:68539–68551.
  25. Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, and 1 others. 2024. A strongreject for empty jailbreaks. Advances in Neural Information Processing Systems, 37:125416–125440. find
  26. Qwen Team. 2024. Qwen2.5: A party of foundation models. find
  27. Ang Li, Yichuan Mo, Mingjie Li, Yifei Wang, and Yisen Wang. 2025. Are smarter llms safer? exploring safety-reasoning trade-offs in prompting and fine-tuning. arXiv preprint arXiv:2502.09673. — [arXiv:2502.09673](https://arxiv.org/abs/2502.09673) find
  28. Simone Tedeschi, Felix Friedrich, Patrick Schramowski, Kristian Kersting, Roberto Navigli, Huu Nguyen, and Bo Li. 2024. Alert: A comprehensive benchmark for assessing large language models’ safety through red teaming. arXiv preprint arXiv:2404.08676. — [arXiv:2404.08676](https://arxiv.org/abs/2404.08676) find
  29. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. arXiv preprint arXiv:2305.20050.
  30. Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. 2023. Do-not-answer: A dataset for evaluating safeguards in llms. arXiv preprint arXiv:2308.13387.
  31. Zi Lin, Zihan Wang, Yongqi Tong, Yangkun Wang, Yuxin Guo, Yujia Wang, and Jingbo Shang. 2023. Toxicchat: Unveiling hidden challenges of toxicity detection in real-world user-ai conversation. Preprint, arXiv:2310.17389.
  32. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, and 1 others. 2023. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688.
  33. AI @ Meta Llama Team. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  34. Junyu Luo, Weizhi Zhang, Ye Yuan, Yusheng Zhao, Junwei Yang, Yiyang Gu, Bohan Wu, Binqi Chen, Ziyue Qiao, Qingqing Long, and 1 others. 2025. Large language model agent: A survey on methodology, applications and challenges. arXiv preprint arXiv:2503.21460.
  35. Zijun Wang, Haoqin Tu, Yuhan Wang, Juncheng Wu, Jieru Mei, Brian R Bartoldson, Bhavya Kailkhura, and Cihang Xie. 2025. Star-1: Safer alignment of reasoning llms with 1k data. arXiv preprint arXiv:2504.01903. — [arXiv:2504.01903](https://arxiv.org/abs/2504.01903) find
  36. Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, and 1 others. 2024. Sorry-bench: Systematically evaluating large language model safety refusal. arXiv preprint arXiv:2406.14598. find
  37. Zhiyuan Xu, Joseph Gardiner, and Sana Belguith. 2025. The dark deep side of deepseek: Fine-tuning attacks against the safety alignment of cot-enabled models. arXiv preprint arXiv:2502.01225. — [arXiv:2502.01225](https://arxiv.org/abs/2502.01225) find
  38. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  39. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations.
  40. Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. 2025. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503.18892.
  41. Yichi Zhang, Zihao Zeng, Dongbai Li, Yao Huang, Zhijie Deng, and Yinpeng Dong. 2025a. Realsafe-r1: Safety-aligned deepseek-r1 without compromising reasoning capability. arXiv preprint arXiv:2504.10081. — [arXiv:2504.10081](https://arxiv.org/abs/2504.10081) find
  42. Zhexin Zhang, Xian Qi Loye, Victor Shea-Jay Huang, Junxiao Yang, Qi Zhu, Shiyao Cui, Fei Mi, Lifeng Shang, Yingkang Wang, Hongning Wang, and 1 others. 2025b. How should we enhance the safety of large reasoning models: An empirical study. arXiv preprint arXiv:2505.15404.
  43. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623.
  44. Chulun Zhou, Chunkang Zhang, Guoxin Yu, Fandong Meng, Jie Zhou, Wai Lam, and Mo Yu. 2026. Improving multi-step rag with hypergraph-based memory for long-context complex relational modeling. Preprint, arXiv:2512.23959. — [arXiv:2512.23959](https://arxiv.org/abs/2512.23959) find
  45. Kaiwen Zhou, Chengzhi Liu, Xuandong Zhao, Shreed-har Jangam, Jayanth Srinivasa, Gaowen Liu, Dawn Song, and Xin Eric Wang. 2025a. The hidden risks of large reasoning models: A safety assessment of r1. arXiv preprint arXiv:2502.12659.
  46. Kaiwen Zhou, Xuandong Zhao, Gaowen Liu, Jayanth Srinivasa, Aosong Feng, Dawn Song, and Xin Eric Wang. 2025b. Safekey: Amplifying aha-moment insights for safety reasoning. arXiv preprint arXiv:2505.16186.
  47. Junda Zhu, Lingyong Yan, Shuaiqiang Wang, Dawei Yin, and Lei Sha. 2025a. Reasoning-to-defend: Safety-aware reasoning can defend large language models from jailbreaking. arXiv preprint arXiv:2502.12970.
  48. Zihao Zhu, Hongbao Zhang, Ruotong Wang, Ke Xu, Siwei Lyu, and Baoyuan Wu. 2025b. To think or not to think: Exploring the unthinking vulnerability in large reasoning models. arXiv preprint arXiv:2502.12202. — [arXiv:2502.12202](https://arxiv.org/abs/2502.12202) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mao-2026-when-models-outthink-their-safety/