Elamur Research⌕
☰
home / papers / 2025 / MTSA: Multi-turn Safety Alignment for LLMs through Multi-round Red-teaming

MTSA: Multi-turn Safety Alignment for LLMs through Multi-round Red-teaming

Paper · 2025 · Weiyang Guo, Jing Li, Wenya Wang et al.

Summary

Jailbreak research and safety training have mostly targeted single-turn prompts. In multi-turn chats, an attacker can spread harmful intent across several innocent-looking turns, and LLMs are more likely to give in. MTSA is a two-stage framework for multi-turn safety.

In the first stage, the authors write a few seed dialogues for four attack strategies. In these dialogues, the attacker reasons about the conversation before each move. This forms the Think-before-attack dataset, which is used to selectively fine-tune an initial red-team model.

In the second stage, the red-team and target models talk repeatedly. Sampled trajectories from these exchanges train both sides: the attacker gets stronger, and the defender becomes harder to break. The target model is trained with a multi-turn RL method that scores each turn by the rewards of later dialogue states, instead of tuning only the final risky turn.

The resulting red-team model reaches state-of-the-art attack success. After three iterations, the target model is safer on both single- and multi-turn benchmarks without losing general ability or refusing too often. Code, a multi-turn jailbreak corpus and the red-team model are released.

Key points

  • The four attack categories are Purpose Inversion, Query Decomposition, Role Play and Mixed Mode. Removing the reasoning ("thought") steps from the training data makes the red-team model clearly less toxic.
  • The multi-turn RLHF method with future rewards gives the target model (MTSA-T3) higher safety and helpfulness win rates than SFT, DPO or IPO.
  • For the MTSA-R3 attacker, toxicity is highest at turn 3 (3.49). Diversity falls from 1.287 at turn 1 to 0.603 at turn 8, and attacks become less effective after about 5 turns.
  • A red-team model based on Qwen2.5-7B-Instruct produces more varied attacks than one based on zephyr-beta, with similar success rate and toxicity.
  • Safety was judged by GPT-4o on CoSafe and BeaverTails. The authors list three limitations:
    • these benchmarks are static;
    • the attacks depend on the initial templates;
    • iterative online sampling is computationally expensive.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Harbin Institute of TechnologyHarbin · China158 works · 28,841 citations🎓Nanyang Technological UniversitySingapore · Singapore156 works · 25,644 citations🎓Zhejiang UniversityHangzhou · China172 works · 12,988 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksBias, fairness & safety · 517 works

Cite

Cite as Weiyang Guo, Jing Li, Wenya Wang, Yu Li, Daojing He, Jun Yu and Min Zhang. 2025. MTSA: Multi-turn Safety Alignment for LLMs through Multi-round Red-teaming. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.1282

@inproceedings{guo2025mtsa,
  title = {{MTSA: Multi-turn Safety Alignment for LLMs through Multi-round Red-teaming}},
  author = {Weiyang Guo, Jing Li, Wenya Wang, Yu Li, Daojing He, Jun Yu and Min Zhang},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1282},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1282}
}

References

56 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2024. Claude-3.5-sonnet. Accessed: 2024-01-01. find
  2. Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Rémi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024. A general theoretical paradigm to understand learning from human preferences. In Proceedings of the International Conference on Artificial Intelligence (ICAI), pages 4447…
  3. Rishabh Bhardwaj, Do Duc Anh, and Soujanya Poria. 2024. Language models are homer simpson! safety re-alignment of fine-tuned language models through task arithmetic. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 14138–14149.
  4. Rishabh Bhardwaj and Soujanya Poria. 2023. Red-teaming large language models using chain of utterances for safety-alignment. arXiv preprint arXiv:2308.09662. find
  5. Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258.
  6. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. 2023. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419.
  7. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot arena: An open platform for evaluating llms by human preference. In Proceedings of the Forty-first Internati…
  8. Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. 2024. Safe RLHF: safe reinforcement learning from human feedback. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR).
  9. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing chat language models by scaling high-quality instructional conversations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3029–…
  10. Guodong Du, Zitao Fang, Jing Li, Junlin Li, Runhua Jiang, Shuyang Yu, Yifei Guo, Yangneng Chen, Sim Kuan Goh, Ho-Kin Tang, Daojing He, Honghai Liu, and Min Zhang. 2025a. Neural parameter search for slimmer fine-tuned models and better transfer. arXiv preprint arXiv:2505.18713.
  11. Guodong Du, Junlin Lee, Jing Li, Runhua Jiang, Yifei Guo, Shuyang Yu, Hanting Liu, Sim Kuan Goh, Ho-Kin Tang, Daojing He, and Min Zhang. 2024a. Parameter competition balancing for model merging. In The Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS). find
  12. Guodong Du, Jing Li, Hanting Liu, Runhua Jiang, Shuyang Yu, Yifei Guo, Sim Kuan Goh, and Ho-Kin Tang. 2024b. Knowledge fusion by evolving weights of language models. In Findings of the Association for Computational Linguistics: ACL 2024. find
  13. Guodong Du, Xuanning Zhou, Junlin Li, Zhuo Li, Zesheng Shi, Wanyu Lin, Ho-Kin Tang, Xiucheng Li, Fangming Liu, Wenya Wang, Min Zhang, and Jing Li. 2025b. Knowledge grafting of large language models. arXiv preprint arXiv:2505.18502. — [arXiv:2505.18502](https://arxiv.org/abs/2505.18502) find
  14. Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacafarm: A simulation framework for methods that learn from human feedback. In Processing of the Advances in Neural Information Processing Systems (…
  15. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. 2022. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858.
  16. Zhaolin Gao, Jonathan Daniel Chang, Wenhao Zhan, Owen Oertell, Gokul Swamy, Kianté Brantley, Thorsten Joachims, J. Andrew Bagnell, Jason D. Lee, and Wen Sun. 2024a. REBEL: Reinforcement learning via regressing relative rewards. In Proceedings of the International Conference on Machine Learning (ICML… find
  17. Zhaolin Gao, Wenhao Zhan, Jonathan D Chang, Gokul Swamy, Kianté Brantley, Jason D Lee, and Wen Sun. 2024b. Regressing the relative future: Efficient policy optimization for multi-turn rlhf. arXiv preprint arXiv:2410.04612. — [arXiv:2410.04612](https://arxiv.org/abs/2410.04612) find
  18. Suyu Ge, Chunting Zhou, Rui Hou, Madian Khabsa, Yi-Chia Wang, Qifan Wang, Jiawei Han, and Yuning Mao. 2024. MART: Improving LLM safety with multi-round automatic red-teaming. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (ACL).
  19. Zhang-Wei Hong, Idan Shenfeld, Tsun-Hsuan Wang, Yung-Sung Chuang, Aldo Pareja, James Glass, Akash Srivastava, and Pulkit Agrawal. 2024. Curiosity-driven red-teaming for large language models. In Proceedings of the International Conference on Learning Representations (ICLR). find
  20. Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. 2023. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674.
  21. Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Qiu, Boxun Li, and Yaodong Yang. 2024. Pku-saferlhf: Towards multi-level safety alignment for llms with human preference. arXiv preprint arXiv:2406.15513.
  22. Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023. Beavertails: Towards improved safety alignment of LLM via a human-preference dataset. In Processing of the Advances in Neural Information Processing Systems (NeurIPS).
  23. Yifan Jiang, Kriti Aggarwal, Tanmay Laud, Kashif Munir, Jay Pujara, and Subhabrata Mukherjee. 2024. Red queen: Safeguarding large language models against concealed multi-turn jailbreaking. arXiv preprint arXiv:2409.17458. — [arXiv:2409.17458](https://arxiv.org/abs/2409.17458) find
  24. Ron Kohavi. 1995. A study of cross-validation and bootstrap for accuracy estimation and model selection. In Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence (IJCAI), pages 1137–1145. find
  25. Wai-Chung Kwan, Xingshan Zeng, Yuxin Jiang, Yufei Wang, Liangyou Li, Lifeng Shang, Xin Jiang, Qun Liu, and Kam-Fai Wong. 2024. Mt-eval: A multi-turn capabilities evaluation benchmark for large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing…
  26. Junlin Lee, Yequan Wang, Jing Li, and Min Zhang. 2024. Multimodal reasoning with multimodal knowledge graph. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). find
  27. Junlin Li, Guodong DU, Jing Li, Sim Kuan Goh, Wenya Wang, Yequan Wang, Fangming Liu, Ho-Kin Tang, Saleh Alharbi, Daojing He, and Min Zhang. 2025. Multi-modality expansion and retention for llms through parameter merging and decoupling. find
  28. Nathaniel Li, Ziwen Han, Ian Steneker, Willow Primack, Riley Goodside, Hugh Zhang, Zifan Wang, Cristina Menghini, and Summer Yue. 2024. LLM defenses are not robust to multi-turn human jailbreaks yet. arXiv preprint arXiv:2408.15221. — [arXiv:2408.15221](https://arxiv.org/abs/2408.15221) find
  29. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David A. Forsyth, and Dan Hendrycks. 2024. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. In Proceedings of the Forty-first Interna…
  30. OpenAI. 2023. Gpt-3.5 turbo. Accessed: 2023-04-01. find
  31. OpenAI. 2024a. Gpt-4o system card. Accessed: 2024-01-01.
  32. Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2024. Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations (ICLR).
  33. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pages 36–44.
  34. Feiliang Ren, Longhui Zhang, Xiaofeng Zhao, Shujuan Yin, Shilei Liu, and Bochao Li. 2022. A simple but effective bidirectional framework for relational triple extraction. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining. find
  35. Qibing Ren, Chang Gao, Jing Shao, Junchi Yan, Xin Tan, Yu Qiao, Wai Lam, and Lizhuang Ma. 2024a. Exploring safety generalization challenges of large language models via code. arXiv preprint arXiv:2403.07865. — [arXiv:2403.07865](https://arxiv.org/abs/2403.07865) find
  36. Qibing Ren, Hao Li, Dongrui Liu, Zhanxu Xie, Xiaoya Lu, Yu Qiao, Lei Sha, Junchi Yan, Lizhuang Ma, and Jing Shao. 2024b. Derail yourself: Multi-turn LLM jailbreak attack through self-discovered clues. arXiv preprint arXiv:2410.10700. — [arXiv:2410.10700](https://arxiv.org/abs/2410.10700) find
  37. Paul Röttger, Hannah Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. 2024. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational L…
  38. Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang. 2023. On second thought, let‘s not think step by step! bias and toxicity in zero-shot reasoning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL).
  39. Zesheng Shi and Yucheng Zhou. 2023. Topic-selective graph network for topic-focused summarization. In Advances in Knowledge Discovery and Data Mining - 27th Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD). find
  40. Zesheng Shi, Yucheng Zhou, Jing Li, Yuxin Jin, YU LI, Daojing He, Fangming Liu, Saleh Alharbi, Jun Yu, and Min Zhang. 2025. Safety alignment via constrained knowledge unlearning. In The 63rd Annual Meeting of the Association for Computational Linguistics (ACL).
  41. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, and Amjad Almahairi etc. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  42. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. 2023. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944. find
  43. Fengxiang Wang, Ranjie Duan, Peng Xiao, Xiaojun Jia, YueFeng Chen, Chongwen Wang, Jialing Tao, Hang Su, Jun Zhu, and Hui Xue. 2024a. Mrj-agent: An effective jailbreak agent for multi-round dialogue. arXiv preprint arXiv:2411.03814. — [arXiv:2411.03814](https://arxiv.org/abs/2411.03814) find
  44. Haoran Wang and Kai Shu. 2024. Trojan activation attack: Red-teaming large language models using steering vectors for safety-alignment. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management (CIKM), pages 2347–2357. find
  45. Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. 2024b. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. arXiv preprint arXiv:2406.12845.
  46. Zeguan Xiao, Yan Yang, Guanhua Chen, and Yun Chen. 2024. Distract large language models for automatic jailbreak attack. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 16230–16244. find
  47. Xikang Yang, Xuehai Tang, Songlin Hu, and Jizhong Han. 2024. Chain of attack: a semantic-driven contextual multi-turn attacker for LLM. arXiv preprint arXiv:2405.05610. — [arXiv:2405.05610](https://arxiv.org/abs/2405.05610) find
  48. Erxin Yu, Jing Li, Ming Liao, Siqi Wang, Zuchen Gao, Fei Mi, and Lanqing Hong. 2024. Cosafe: Evaluating large language model safety in multi-turn dialogue coreference. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 17494–17508. find
  49. Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. 2024. GPT-4 is too smart to be safe: Stealthy chat with llms via cipher. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR).
  50. Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. 2024. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge AI safety by humanizing llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational (ACL), pages 14322– 14350.
  51. Jinchuan Zhang, Yan Zhou, Yaxin Liu, Ziming Li, and Songlin Hu. 2024a. Holistic automated red teaming for large language models through top-down test case generation and multi-turn interaction. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1… find
  52. Longhui Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, Meishan Zhang, and Min Zhang. 2024b. A two-stage adaptation of large language models for text ranking. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). find
  53. Rui Zheng, Hongyi Guo, Zhihan Liu, Xiaoying Zhang, Yuanshun Yao, Xiaojun Xu, Zhaoran Wang, Zhiheng Xi, Tao Gui, Qi Zhang, et al. 2024. Toward optimal llm alignments using two-player games. arXiv preprint arXiv:2406.10977. find
  54. Zhenhong Zhou, Jiuyang Xiang, Haopeng Chen, Quan Liu, Zherui Li, and Sen Su. 2024. Speak out of turn: Safety vulnerability of large language models in multi-turn dialogue. arXiv preprint arXiv:2402.17262. — [arXiv:2402.17262](https://arxiv.org/abs/2402.17262) find
  55. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023a. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.
  56. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. 2023b. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/guo-2025-mtsa-multi-turn-safety-alignment/