Generation of Dialogue Breakdown Repair Utterance in Non-task Oriented Conversation
Summary
Even capable LLM-based chat systems sometimes produce "dialogue breakdowns," such as factual mistakes or contextually odd replies. Earlier repair methods only covered a few predefined error types, often with hand-crafted rules. This paper asks whether a general LLM can learn to recover smoothly from many kinds of breakdown in casual, open-domain Japanese conversation.
The authors use the Dialogue Breakdown Repair Corpus (DBR Corpus). In it, crowd workers who saw a dialogue ending in a system error wrote both a plausible user reaction and the repair reply they would expect from the system.
They first compared zero-shot GPT-4o-mini with two open Japanese-capable models; GPT-4o-mini scored best. They then fine-tuned GPT-4o-mini on the corpus. The fine-tuned model matched human references more closely on BLEU, BERTScore and S-BERT. Its outputs were also shorter and less verbose than the over-polite, information-heavy zero-shot replies.
The work shows that a modest, targeted dataset can push an LLM toward natural, human-like repair behaviour in casual dialogue.
Key points
-
DBR Corpus: Japanese, built on the Higashinaka et al. (2022) breakdown taxonomy. It covers 140 breakdown scenarios with 3,990 responses from 57 crowd workers. Data was split 8:1:1, and 399 test instances were used for evaluation.
-
Zero-shot baselines: GPT-4o-mini outperformed llm-jp-3-3.7b-instruct and DeepSeek-R1-Distill-Qwen-7B-Japanese on all metrics. On BERTScore it reached 0.8889, versus 0.8740 and 0.8487. On S-BERT it reached 0.4795, versus 0.4034 and 0.2956.
-
Fine-tuning gains: GPT-4o-mini was fine-tuned for 3 epochs with a learning-rate multiplier of 1.0. Compared with zero-shot prompting, BLEU rose from 6.769 to 12.843, BERTScore from 0.889 to 0.914, and S-BERT from 0.480 to 0.522.
-
Utterance length (sentences / morphemes):
- Fine-tuned model: 1.22 / 9.18
- Crowd workers: 1.33 / 10.57
- Zero-shot prompting: 2.66 / 27.68
Friedman and Wilcoxon tests showed significant differences between the methods.
-
LLM-as-a-Judge and limitations: A GPT-4o judge rated zero-shot replies as more redundant, polite and informative. Fine-tuned outputs scored closer to the human ones. Stated limitations are that the data is Japanese only and that no human evaluation was carried out.
Cite
@inproceedings{tsubokura2026generation,
title = {{Generation of Dialogue Breakdown Repair Utterance in Non-task Oriented Conversation}},
author = {Tsubokura, Kazuya and Iribe, Yurie and Kitaoka, Norihide},
booktitle = {Proceedings of the 27th Annual Meeting of the Special Interest Group on Discourse and Dialogue},
year = {2026},
doi = {10.18653/v1/2026.sigdial-1.38},
url = {https://doi.org/10.18653/v1/2026.sigdial-1.38}
}References
- Timothy Bickmore, Ha Trinh, Reza Asadi, and Stefan Olafsson. 2018. Safety First: Conversational Agents for Health Care, pages 33–57. Springer International Publishing, Cham. find
- Chayan Chakrabarti and George F. Luger. 2015. Artificial conversations for customer service chatter bots: Architecture, algorithms, and evaluation metrics. Expert Systems with Applications, 42(20):6878– 6897. find
- Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. CoRR, abs/1911.02116. find
- Ryuichiro Higashinaka, Masahiro Araki, Hiroshi Tsukahara, and Masahiro Mizukami. 2022. Classification of utterances that lead to dialogue breakdowns in chat-oriented dialogue systems (in japanese). Journal of Natural Language Processing, 29(2):443–466. find
- Michimasa Inaba and Kenichi Takahashi. 2019. Applying dialogue breakdown detection to dialogue systems (in japanese). Transactions of the Japanese Society for Artificial Intelligence, 34(3):B–I64_1–8. find
- Bilyana Martinovsky and David Traum. 2003. The error is the clue: breakdown in human-machine interaction. In Error Handling in Spoken Dialogue Systems, pages 11–16. find
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational… find our card
- Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,… find
- Hiroaki Sugiyama, Toyomi Meguro, Yuichiro Yoshikawa, and Junji Yamato. 2018. Avoiding breakdown of conversational dialogue through inter-robot coordination. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS ’18, page 2256–2258, Richland, SC. Inter… find
- Kazuya Tsubokura, Yurie Iribe, and Norihide Kitaoka. 2026. A corpus for personalized dialogue breakdown repair in japanese open-domain conversations. In Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026), pages 2899–2912, Palma, Mallorca, Spain. European Language R… find our card
- Takahisa Uchida, Takashi Minato, Tora Koyama, and Hiroshi Ishiguro. 2019. Who is responsible for a dialogue breakdown? an error recovery strategy that promotes cooperative intentions from humans by mutual attribution of responsibility in human-robot dialogues. Frontiers in Robotics and AI, 6. find
- Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. find
This page in Markdown → https://md.research.elamur.ai/free/en/papers/tsubokura-2026-generation-dialogue-breakdown-repair-utterance/