Elamur Research⌕
☰
home / papers / 2026 / Knowing-but-Doing: Diagnosing and Defending Role-Play-Driven LLMs Jailbreaks via Moral Disengagement

Knowing-but-Doing: Diagnosing and Defending Role-Play-Driven LLMs Jailbreaks via Moral Disengagement

Paper · 2026 · Haiming Qin, Jianxun Lian, Qimin Zhong et al.

Summary

The paper examines why LLMs drift into unsafe behaviour when asked to act out characters. The authors frame this with Bandura's Moral Disengagement theory and build MD-Trace, a diagnostic benchmark. It combines 80 personas from the Moral RolePlay dataset, split evenly across four moral levels from paragons to villains, with harmful tasks drawn from PAP. Tasks are matched to personas through Qwen3-Embedding-8B retrieval, LLM reranking and human review, then rewritten as immersive in-character scenarios. A gpt-4o-mini judge reads the models' thinking traces and scores attack success, whether the model noticed the risk, the eight disengagement mechanisms, and Role Fidelity.

Across 8 LLMs, role-play cuts unsafe compliance for benign personas and raises it for villainous ones. Models frequently identify the danger in their reasoning and then comply anyway, which the authors call "Knowing-but-Doing". The leading rationalization is Moral Justification (M1), with Disregard of Consequences (M6) second. The proposed fix, MD-Shield, is an introspective fine-tuning approach that teaches models to spot and reject such justifications. It lowers attack success while largely keeping characters intact, which matters for safe persona-driven agents.

Key points

  • MD-Trace contains 400 cases: 80 personas (20 per moral level) × 5 matched tasks, selected from a pool of 60 PAP tasks.
  • In 92% of successful Level 4 (villain) jailbreaks, the trace shows M1-style rationalization after the harm was recognized.
  • Judge reliability was checked against human labels. The judge reached a κ of 0.87 on ASR, matched human ratings 88% of the time on Role Fidelity, and matched them 97% of the time on MD mechanisms. A trace-only compliance prediction reached 93.5% agreement.
  • Built on Qwen3-8B, MD-Shield reaches ASR 0.044 on unseen personas, versus 0.330 for the base model. On PAP it scores 0.060, on JailbreakHub 0.048, and on SORRY-Bench 0.023 with Role Fidelity 0.91.
  • Safety-tuned baselines such as RealSafe-R1 and STAR-1 lose much more character consistency on SORRY-Bench (Role Fidelity 0.70 and 0.43). Code is available at github.com/lavapapa/MoralJustify.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Microsoft Research AsiaBeijing · China73 works · 60,539 citations🎓Shenzhen UniversityShenzhen · China16 works · 95 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksBias, fairness & safety · 517 works

Cite

Cite as Haiming Qin, Jianxun Lian, Qimin Zhong, Mingyang Zhou, Hao Liao and Naipeng Chao. 2026. Knowing-but-Doing: Diagnosing and Defending Role-Play-Driven LLMs Jailbreaks via Moral Disengagement. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.349

@inproceedings{qin2026knowing,
  title = {{Knowing-but-Doing: Diagnosing and Defending Role-Play-Driven LLMs Jailbreaks via Moral Disengagement}},
  author = {Haiming Qin, Jianxun Lian, Qimin Zhong, Mingyang Zhou, Hao Liao and Naipeng Chao},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.349},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.349}
}

References

27 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate. 2023. Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3):337–351.
  2. Albert Bandura. 1999. Moral disengagement in the perpetration of inhumanities. Personality and Social Psychology Review, 3(3):193–209. find
  3. Albert Bandura. 2016. Moral Disengagement: How People Do Harm and Live with Themselves. Worth Publishers, New York. find
  4. Jiangjie Chen, Xintao Wang, Rui Xu, Siyu Yuan, Yikai Zhang, Wei Shi, Jian Xie, Shuang Li, Ruihan Yang, Tinghui Zhu, Aili Chen, Nianqi Li, Lida Chen, Caiyu Hu, Siye Wu, Scott Ren, Ziquan Fu, and Yanghua Xiao. 2024. From persona to personalization: A survey on role-playing language agents. Preprint, a…
  5. Google DeepMind. 2025. Gemini 3 flash: Model card. Technical report, Google DeepMind. find
  6. Published: December 2025. find
  7. Melody Y. Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, Hyung Won Chung, Sam Toyer, Johannes Heidecke, Alex Beutel, and Amelia Glaese. 2024. Deliberative alignment: Reasoning enables safer language models. arXiv preprint…
  8. Edward J. Hu, Yelong Shen, Phil Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Yuandong Wang, Will Li, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
  9. Wonje Jeung, Sangyeon Yoon, Minsuk Kahng, and Albert No. 2025. Safepath: Preventing harmful reasoning in chain-of-thought via early alignment. arXiv preprint arXiv:2505.14667. — [arXiv:2505.14667](https://arxiv.org/abs/2505.14667) find
  10. A. Liu and 1 others. 2025. Deepseek-v3.2: Pushing the frontier of open large language models. Preprint, arXiv:2512.02556.
  11. Llama Team, AI @ Meta. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  12. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations. find
  13. Yingning Ma. 2025. Realsafe: Quantifying safety risks of language agents in real-world. In Proceedings of the 31st International Conference on Computational Linguistics, pages 9586–9617. find
  14. Qwen Team and 1 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  15. Yun Shen, Chen Chen, Michael Backes, Yang Shen, and Yang Zhang. 2024. “do anything now”: Characterizing and evaluating in-the-wild jailbreak prompts on large language models. In Proceedings of the 2024 ACM SIGSAC Conference on Computer and Communications Security (CCS ’24), pages 1671–1685, Salt Lak…
  16. Yihong Tang, Bo Wang, Xu Wang, Dongming Zhao, Jing Liu, Jijun Zhang, Ruifang He, and Yuexian Hou. 2024. Rolebreak: Character hallucination as a jailbreak attack in role-playing systems. Preprint, arXiv:2409.16727. find
  17. Xintao Wang, Heng Wang, Yifei Zhang, Xinfeng Yuan, Rui Xu, Jen-Tse Huang, Siyu Yuan, Haoran Guo, Jiangjie Chen, Shuchang Zhou, Wei Wang, and Yanghua Xiao. 2025a. CoSER: Coordinating LLM-based persona simulation of established roles. In Proceedings of the 42nd International Conference on Machine Lear… find
  18. Zhenhua Wang, Wei Xie, Shuoyoucheng Ma, Enze Wang, and Baosheng Wang. 2025b. Evading llms’ safety boundary with adaptive role-play jailbreaking. Electronics, 14(24):4808. find
  19. Zijun Wang, Haoqin Tu, Yuhan Wang, Juncheng Wu, Yanqing Liu, Jieru Mei, Brian R. Bartoldson, Bhavya Kailkhura, and Cihang Xie. 2025c. Star-1: Safer alignment of reasoning llms with 1k data. arXiv preprint arXiv:2504.01903. — [arXiv:2504.01903](https://arxiv.org/abs/2504.01903) find
  20. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
  21. Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari M. Sehwag, Boyi Wei, Luxi He, Kaixuan Huang, Dacheng Li, Ying Sheng, Ruoxi Jia, Bo Li, Danqi Chen, Kai Li, Peter Henderson, and Prateek Mittal. 2024. Sorry-bench: Systematically evaluating llm safety refusal behaviors. In NeurIPS Datasets and B… find
  22. Xianglin Yang, Gelei Deng, Jieming Shi, Tianwei Zhang, and Jin Song Dong. 2025. Make your guard learn to think: Defending against jailbreak attacks with safety chain-of-thought. arXiv preprint arXiv:2501.19180. — [arXiv:2501.19180](https://arxiv.org/abs/2501.19180) find
  23. Zihao Yi, Qingxuan Jiang, Ruotian Ma, Xingyu Chen, Qu Yang, Mengru Wang, Fanghua Ye, Ying Shen, Zhaopeng Tu, Xiaolong Li, and Linus Li. 2025. Too good to be bad: On the failure of llms to role-play villains. arXiv preprint arXiv:2511.04962.
  24. Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. 2024. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 1…
  25. Jianli Zhao, Tingchen Fu, Rylan Schaeffer, Mrinank Sharma, and Fazl Barez. 2025. Chain-of-thought hijacking. arXiv preprint arXiv:2510.26418. — [arXiv:2510.26418](https://arxiv.org/abs/2510.26418) find
  26. Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. 2024. Autodan: Interpretable gradient-based adversarial attacks on large language models. In Proceedings of the 1st Conference on LLM Safety (COLM). find
  27. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/qin-2026-knowing-but-doing-diagnosing-defending/