Elamur Research⌕
☰
home / papers / 2026 / Breaking the Script: Do Role-Playing Agents Maintain Goal Alignment under Distraction?

Breaking the Script: Do Role-Playing Agents Maintain Goal Alignment under Distraction?

Summary

LLMs are increasingly used as role-playing partners in professional and educational training simulations. A key risk is that a learner's off-topic or competing input pulls the agent away from the scenario's pedagogical objectives. The authors treat this "goal-competing distraction" as a distinct failure mode and build a controlled multi-agent framework to study it. An LLM-based user simulator injects scripted distractors at the start of each scenario phase, the role-playing agent replies in character, and an LLM annotator labels each reply. A branching policy then decides whether the episode continues. They compare a stateless agent with one whose prompt includes an explicit natural-language dialogue state: current phase, active objective, and topics already covered.

Behaviour depends on model scale. Larger models stay socially responsive and often take up the distractor, while the smaller model resists more rigidly. Redirection is the usual first reaction, but redirected episodes frequently drift into full engagement later. Dialogue state stabilises the small model but adds little for larger ones. The authors conclude that training agents need explicit control over the balance between responsiveness and goal adherence.

Key points

  • Models tested: Gemini-2.0-Flash, Llama-3.3-70B-Instruct and Llama-3.1-8B-Instruct.
  • Scenarios: two professional training scenarios, leadership coaching and feedback delivery.
  • Distractor taxonomy: adapted from Hample (2016), with three goal types: Relational maintenance (RE), Manage own face (MA) and Instrumental redirection (IN). Each type has two variants differing in contextual relatedness, giving six distractors per scenario.
  • Episode design:
    • Each agent reply is labelled RESIST, REDIRECT or ENGAGE.
    • Only REDIRECT replies trigger a further simulator turn.
    • Episodes are capped at five exchanges, a limit chosen because MiniLM embedding similarity to the original distractor declined across turns.
  • Findings by distractor type:
    • RE distractions were the easiest for all models to deflect.
    • Llama models engaged more with MA distractors, while Gemini engaged more with IN distractors.
    • A small exploratory human study, using workshop logs collected at Utrecht University with DialogueTrainer, showed that such distractions also occur in real training sessions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.sigdial-1.8

@inproceedings{lu2026breaking,
  title = {{Breaking the Script: Do Role-Playing Agents Maintain Goal Alignment under Distraction?}},
  author = {Lu, Dongxu and Gatt, Albert and Jeuring, Johan},
  booktitle = {Proceedings of the 27th Annual Meeting of the Special Interest Group on Discourse and Dialogue},
  year = {2026},
  doi = {10.18653/v1/2026.sigdial-1.8},
  url = {https://doi.org/10.18653/v1/2026.sigdial-1.8}
}

References

31 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rauno Arike, Elizabeth Donoway, Henning Bartsch, and Marius Hobbhahn. 2025. Technical report: evaluating goal drift in language model agents. Preprint, arXiv:2505.02709. — [arXiv:2505.02709](https://arxiv.org/abs/2505.02709) find
  2. Ryan S. J. d. Baker, Albert T. Corbett, Kenneth R. Koedinger, Shelley Evenson, Ido Roll, Angela Z. Wagner, Meghan Naim, Jay Raspat, Daniel J. Baker, and Joseph E. Beck. 2006. Adapting to when students game an intelligent tutoring system. In Intelligent Tutoring Systems: 8th International Conference… find
  3. Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. 2018. Multiwoz: a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language P… find
  4. Rafael Carranza and Mateo Alejandro Rojas. 2025. Interpretable and robust dialogue state tracking via natural language summarization with llms. Preprint, arXiv:2503.08857. — [arXiv:2503.08857](https://arxiv.org/abs/2503.08857) find
  5. Adam J. Coscia, Shunan Guo, Eunyee Koh, and Alex Endert. 2025. Ongoal: tracking and visualizing conversational goals in multi-turn dialogue with large language models. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology, Busan, Republic of Korea. Association for… find
  6. Vardhan Dongre, Ryan A. Rossi, Viet Dac Lai, David Seunghyun Yoon, Dilek Hakkani-Tür, and Trung Bui. 2025. Drift no more? context equilibria in multi-turn llm interactions. arXiv preprint arXiv:2510.07777. — [arXiv:2510.07777](https://arxiv.org/abs/2510.07777) find
  7. Jiayi Geng, Howard Chen, Ryan Liu, Manoel Horta Ribeiro, Robb Willer, Graham Neubig, and Thomas L. Griffiths. 2025. Accumulating context changes the beliefs of language models. Preprint, arXiv:2511.01805. — [arXiv:2511.01805](https://arxiv.org/abs/2511.01805) find
  8. Google DeepMind. 2024. Introducing gemini 2.0: our new ai model for the agentic era. Accessed: 2026-04-20. find
  9. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, and et al. Pandey, Abhinav. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. find
  10. Shengyue Guan, Jindong Wang, Jiang Bian, Bin Zhu, Jian-Guang Lou, and Haoyi Xiong. 2026. Evaluating llm-based agents for multi-turn conversations: A survey. ACM Transactions on Intelligent Systems and Technology, 17(4). find
  11. Dale Hample. 2016. A theory of interpersonal goals and situations. Communication Research, 43(3):344– 371. find
  12. Young-Woo Jo, Myungeun Lee, and Hyung-Jeong Yang. 2025. Large language model-based virtual patient simulations in medical and nursing education: A review. Applied Sciences, 15(22):11917. find
  13. Junseok Kim, Nakyeong Yang, and Kyomin Jung. 2025. Persona is a double-edged sword: Rethinking the impact of role-play prompts in zero-shot reasoning tasks. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of… find
  14. Chin-Yew Lin. 2004. Rouge: a package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. find
  15. Dongxu Lu, Johan Jeuring, and Albert Gatt. 2025. Evaluating llm-generated versus human-authored responses in role-play dialogues. In Proceedings of the 18th International Natural Language Generation Conference, pages 20–40, Hanoi, Vietnam. Association for Computational Linguistics. find
  16. Rachel Ma, Jingyi Qu, Andreea Bobu, and Dylan Hadfield-Menell. 2024. Goal inference from open-ended dialog. Preprint, arXiv:2410.13957. of the 20th Conference on Natural Language Processing (KONVENS 2024), pages 249–263, Vienna, Austria. Association for Computational Linguistics. — [arXiv:2410.13957](https://arxiv.org/abs/2410.13957) find
  17. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744. find
  18. Anyi Wang, Dong Shu, Yifan Wang, Yunpu Ma, and Mengnan Du. 2025a. Improving llm reasoning through interpretable role-playing steering. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 731–751, Suzhou, China. Association for Computational Linguistics. find
  19. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational… find our card
  20. Lei Wang, Jianxun Lian, Yi Huang, Yanqi Dai, Haoxuan Li, Xu Chen, Xing Xie, and Ji-Rong Wen. 2025b. Characterbox: evaluating the role-playing capabilities of llms in text-based virtual worlds. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computa… find
  21. Changyong Qi, Longwei Zheng, Anna He, Haoxin Xu, Linzhao Jia, Yuang Wei, Bingqian Jiang, and Xiaoqing Gu. 2025. Simulating student learning behaviors with llm-based role-playing agents: A data-driven and cognitively inspired framework. Expert Systems with Applications, 304:130753. find
  22. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728– 53741. find
  23. Janarthanan Rajendran, Jatin Ganhotra, Satinder Singh, and Lazaros Polymenakos. 2018. Learning end-to-end goal-oriented dialog with multiple answers. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3834–3843, Brussels, Belgium. Association for Computa… find
  24. Amirhossein Razavi, Mina Soltangheis, Negar Arabzadeh, Sara Salamat, Morteza Zihayat, and Ebrahim Bagheri. 2025. Benchmarking prompt sensitivity in large language models. In Advances in Information Retrieval: 47th European Conference on Information Retrieval (ECIR 2025), Lecture Notes in Computer Sc… find
  25. Traian Rebedea, Makesh Sreedhar, Shaona Ghosh, Jiaqi Zeng, and Christopher Parisien. 2024. Canttalkaboutthis: Aligning language models to stay on topic in dialogues. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 12232– 12252, Miami, Florida, USA. Association for Com… find
  26. Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. 2021. Minilmv2: multi-head self-attention relation distillation for compressing pre-trained transformers. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2140–2151, Online. Association for Computation… find
  27. Xingguang Wang, Xuxin Cheng, Juntong Song, Tong Zhang, and Cheng Niu. 2024. Enhancing dialogue state tracking models through llm-backed user-agents simulation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8724–8741, Bangkok… find
  28. Hao Xiang, Tianyi Tang, Yang Su, Bowen Yu, An Yang, Fei Huang, Yichang Zhang, Yaojie Lu, Hongyu Lin, Xianpei Han, and 1 others. 2025. Rmtbench: Benchmarking llms through multi-turn user-centric role-playing. In Findings of the Association for Computational Linguistics: EMNLP 2025, Suzhou, China. Ass… find
  29. Langqi Yang, Tianhang Zheng, Yixuan Chen, Kedong Xiu, Hao Zhou, Di Wang, Puning Zhao, Zhan Qin, and Kui Ren. 2025. Harmmetric eval: benchmarking metrics and judges for llm harmfulness assessment. Preprint, arXiv:2509.24384. — [arXiv:2509.24384](https://arxiv.org/abs/2509.24384) find
  30. Murray Shanahan, Kyle McDonell, and Laria Reynolds. 2023. Role play with large language models. Nature, 623(7987):493–498. find our card
  31. Don Tuggener, Teresa Schneider, Ariana Huwiler, Tobias Kreienbühl, Simon Hischier, Pius von Däniken, and Susanna Niehaus. 2024. Role-playing llms in professional communication training: The case of investigative interviews with children. In Proceedings find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/lu-2026-breaking-script-do-role-playing/