Elamur Research⌕
☰
home / papers / 2025 / The Hidden Risks of Large Reasoning Models: A Safety Assessment of R1

The Hidden Risks of Large Reasoning Models: A Safety Assessment of R1

Paper · 2025 · Kaiwen Zhou, Chengzhi Liu, Xuandong Zhao et al.

Summary

Open-weight large reasoning models such as DeepSeek-R1 have become much stronger. Because they are freely available, misuse is a real concern. This study audits their safety systematically. It compares three open reasoning models (DeepSeek-R1, the Llama 3.3-based distilled DeepSeek-R1-70b, and QwQ) with non-reasoning counterparts (DeepSeek-V3, Llama 3.3-70b) and proprietary systems (o3-mini, Gemini 2.5).

The evaluation covers policy-based harmful requests, cybersecurity tasks, over-refusal, jailbreaks and prompt injection. GPT-4o acts as the safety judge, and multi-attribute reward models score how harmful unsafe outputs are. The authors also judge the hidden reasoning traces separately from the final answers.

Main findings:

  • Open reasoning models lag clearly behind o3-mini, both on benchmarks and under attack.
  • Distillation weakened Llama 3.3's original safety alignment.
  • When stronger reasoners do comply with malicious requests, their answers are more harmful.
  • Safety-aware thinking appears but often breaks down under adversarial prompts.
  • Reasoning traces frequently leak more unsafe content than the final replies.

These results suggest that safety training should explicitly cover intermediate reasoning, not just final outputs.

Key points

  • Benchmarks (7 datasets):
    • AirBench: 5,694 prompts across 16 level-2 categories drawn from regulations and corporate policies.
    • CyberSecEval suites: MITRE (377), Code Interpreter (500), Spear Phishing (200) and prompt Injection (251).
    • XSTest: 250 prompts for over-refusal.
    • WildGuard jailbreak set: 810 prompts.
  • Distillation cost: DeepSeek-R1-70b scored consistently lower on safety than its base model, Llama 3.3-70b, across all tests.
  • Harmfulness scoring: Rather than relying only on binary safe/unsafe labels, the study rated harm levels with reward-model scorers drawn from Wang et al. (2024) and Dorka (2024). Under this scoring, R1 models' unsafe answers proved more harmful than those of non-reasoning models.
  • Reasoning vs. answer: On most benchmarks, the content inside <think> tags was less safe than the final completion. In some cases, the trace spelled out attack techniques that the final reply then refused to give.
  • Proposed remedies:
    • Stronger safety alignment, including rule-based rewards.
    • Explicit safety-reasoning training data.
    • Representation engineering.
    • Reinforcement learning with rewards for the safety of intermediate reasoning steps.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California, BerkeleyBerkeley · United States175 works · 227,440 citations🎓University of California, Santa BarbaraSanta Barbara · United States57 works · 9,752 citations🎓University of California, Santa CruzSanta Cruz · United States36 works · 3,515 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksBias, fairness & safety · 517 works

Cite

Cite as Kaiwen Zhou, Chengzhi Liu, Xuandong Zhao, Shreedhar Jangam, Jayanth Srinivasa, Gaowen Liu, Dawn Song and Xin Eric Wang. 2025. The Hidden Risks of Large Reasoning Models: A Safety Assessment of R1. Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.ijcnlp-long.173

@inproceedings{zhou2025hidden,
  title = {{The Hidden Risks of Large Reasoning Models: A Safety Assessment of R1}},
  author = {Kaiwen Zhou, Chengzhi Liu, Xuandong Zhao, Shreedhar Jangam, Jayanth Srinivasa, Gaowen Liu, Dawn Song and Xin Eric Wang},
  booktitle = {Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.ijcnlp-long.173},
  url = {https://doi.org/10.18653/v1/2025.ijcnlp-long.173}
}

References

38 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, Zico Kolter, Matt Fredrikson, et al. 2024. Agentharm: A benchmark for measuring harmfulness of llm agents. arXiv preprint arXiv:2410.09024. — [arXiv:2410.09024](https://arxiv.org/abs/2410.09024) find
  2. Manish Bhatt, Sahana Chennabasappa, Yue Li, Cyrus Nikolaidis, Daniel Song, Shengye Wan, Faizan Ahmad, Cornelius Aschermann, Yaohui Chen, Dhaval Kapil, et al. 2024. Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models. arXiv preprint arXiv:2404.13161. — [arXiv:2404.13161](https://arxiv.org/abs/2404.13161) find
  3. Nicolai Dorka. 2024. Quantile regression for distributional reward models in rlhf. arXiv preprint arXiv:2409.10164. — [arXiv:2409.10164](https://arxiv.org/abs/2409.10164) find
  4. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  5. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  6. Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. 2024. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms. Preprint, arXiv:2406.18495.
  7. Luxi He, Mengzhou Xia, and Peter Henderson. 2024. What is in your safe data? identifying benign data that breaks safety. arXiv preprint arXiv:2404.01099. — [arXiv:2404.01099](https://arxiv.org/abs/2404.01099) find
  8. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  9. Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, et al. 2024. Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models. arXiv preprint arXiv:2406.18510.
  10. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. 2024. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787. find
  11. Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wangmeng Zuo, Dahua Lin, Yu Qiao, and Jing Shao. 2024a. Salad-bench: A hierarchical and comprehensive safety benchmark for large language models. arXiv preprint arXiv:2402.05044.
  12. Nathaniel Li, Ziwen Han, Ian Steneker, Willow Primack, Riley Goodside, Hugh Zhang, Zifan Wang, Cristina Menghini, and Summer Yue. 2024b. Llm defenses are not robust to multi-turn human jailbreaks yet. arXiv preprint arXiv:2408.15221. — [arXiv:2408.15221](https://arxiv.org/abs/2408.15221) find
  13. Zeyi Liao and Huan Sun. 2024. Amplegcg: Learning a universal and transferable generative model of adversarial suffixes for jailbreaking both open and closed llms. arXiv preprint arXiv:2404.07921. — [arXiv:2404.07921](https://arxiv.org/abs/2404.07921) find
  14. Google DeepMind. 2025. Gemini model thinking updates: March 2025. Accessed: 2025-05-18. find
  15. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024a. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. find
  16. Melody Y Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, et al. 2024. Deliberative alignment: Reasoning enables safer language models. 2024. URL https://arxiv.org/abs/2412.16339. ↗
  17. Xiaogeng Liu, Peiran Li, Edward Suh, Yevgeniy Vorobeychik, Zhuoqing Mao, Somesh Jha, Patrick McDaniel, Huan Sun, Bo Li, and Chaowei Xiao. 2024b. Autodan-turbo: A lifelong agent for strategy self-exploration to jailbreak llms. arXiv preprint arXiv:2410.05295. — [arXiv:2410.05295](https://arxiv.org/abs/2410.05295) find
  18. Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian Kivlichan, Molly Lin, Alex Beutel, John Schulman, and Lilian Weng. 2024. Rule based rewards for language model safety. arXiv preprint arXiv:2411.01111. find
  19. Jingwei Yi, Yueqi Xie, Bin Zhu, Emre Kiciman, Guangzhong Sun, Xing Xie, and Fangzhao Wu. 2023. Benchmarking and defending against indirect prompt injection attacks on large language models. arXiv preprint arXiv:2312.14197. — [arXiv:2312.14197](https://arxiv.org/abs/2312.14197) find
  20. OpenAI. 2025a. O1 system card. find
  21. Yi Zeng, Yu Yang, Andy Zhou, Jeffrey Ziwei Tan, Yuheng Tu, Yifan Mai, Kevin Klyman, Minzhou Pan, Ruoxi Jia, Dawn Song, et al. 2024. find
  22. Air-bench 2024: A safety benchmark based on risk categories from regulations and policies. arXiv preprint arXiv:2407.17436. find
  23. OpenAI. 2025b. O3 mini system card. find
  24. Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2023. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693.
  25. Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. 2023. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. arXiv preprint arXiv:2308.01263.
  26. Qwen Team. 2025. QWQ-32B: A New Frontier for Qwen. https://qwenlm.github.io/blog/ qwq-32b/. Accessed: 2025-05-18. ↗
  27. Shengye Wan, Cyrus Nikolaidis, Daniel Song, David Molnar, James Crnkovich, Jayson Grace, Manish Bhatt, Sahana Chennabasappa, Spencer Whitman, Stephanie Ding, Vlad Ionescu, Yue Li, and Joshua Saxe. 2024a. Cyberseceval 3: Advancing the evaluation of cybersecurity risks and capabilities in large langua… — [arXiv:2408.01605](https://arxiv.org/abs/2408.01605) find
  28. Shengye Wan, Cyrus Nikolaidis, Daniel Song, David Molnar, James Crnkovich, Jayson Grace, Manish Bhatt, Sahana Chennabasappa, Spencer Whitman, Stephanie Ding, et al. 2024b. Cyberseceval 3: Advancing the evaluation of cybersecurity risks and capabilities in large language models. arXiv preprint arXiv:… — [arXiv:2408.01605](https://arxiv.org/abs/2408.01605) find
  29. Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. 2024. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. arXiv preprint arXiv:2406.12845.
  30. Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. 2024. Injecagent: Benchmarking indirect prompt injections in tool-integrated large language model agents. arXiv preprint arXiv:2403.02691.
  31. Hanrong Zhang, Jingyuan Huang, Kai Mei, Yifei Yao, Zhenting Wang, Chenlu Zhan, Hongwei Wang, and Yongfeng Zhang. 2024. Agent security bench (asb): Formalizing and benchmarking attacks and defenses in llm-based agents. arXiv preprint arXiv:2410.02644. — [arXiv:2410.02644](https://arxiv.org/abs/2410.02644) find
  32. Kaiwen Zhou, Xuandong Zhao, Jayanth Srinivasa, Gaowen Liu, Aosong Feng, Dawn Song, and Xin Eric Wang. 2025. Safekey: Amplifying aha-moment insights for safety reasoning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 25407–25423.
  33. Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. 2024. Autodan: interpretable gradient-based adversarial attacks on large language models. In First Conference on Language Modeling. find
  34. Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, J Zico Kolter, Matt Fredrikson, and Dan Hendrycks. 2024. Improving alignment and robustness with circuit breakers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  35. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.
  36. Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. 2023. Do-not-answer: A dataset for evaluating safeguards in llms. arXiv preprint arXiv:2308.13387.
  37. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2024. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36.
  38. Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, et al. 2024. Sorry-bench: Systematically evaluating large language model safety refusal behaviors. arXiv preprint arXiv:2406.14598. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2025-hidden-risks-large-reasoning-models/