Elamur Research⌕
☰
home / papers / 2026 / TRACER: Early Failure Detection for Task-Oriented Dialogue

TRACER: Early Failure Detection for Task-Oriented Dialogue

Summary

Most evaluation of task-oriented dialogue (TOD) systems judges a conversation only after it ends, when it is too late to rescue it. TRACER reframes this as forecasting: given only a prefix of a dialogue, it estimates the probability that the whole interaction will fail. Each visible prefix becomes a separate training example labeled with the dialogue's final outcome. The model has two input streams. One encodes the serialized belief state as text with RoBERTa. The other processes a sequence of turn-level trajectory features describing how the dialogue state evolves. The fused representation produces a risk score, and a threshold tuned on development data decides when to launch a recovery action. Experiments use both gold (oracle) and LLM-generated belief states, showing the model 25%, 50%, 75% or 100% of each dialogue. TRACER finds useful failure signals as early as the first quarter of a conversation. It outperforms heuristic, classical and single-stream baselines. This makes early warning a practical tool for live dialogue systems rather than a post-hoc diagnostic.

Key points

  • The trajectory stream uses a 5-dimensional vector per turn:
    • oscillation score (slots whose values keep changing),
    • coverage rate of required slots,
    • cumulative conflict count,
    • fill velocity,
    • number of domains visited.
  • The text stream encodes the cumulative belief state with RoBERTa, serialized as "domain-slot: value" pairs following the MultiWOZ ontology.
  • In the generated setting, belief states are predicted turn by turn by Qwen3-30B-A3B instead of being taken from gold annotations.
  • Zero-shot baselines prompt Llama-3.1-8B, Mistral-7B-Instruct and Qwen2.5-7B to output a JSON success/failure prediction with a confidence score, which is mapped to a failure probability for AUC.
  • For recovery, a feature-driven template method beats a fine-tuned T5 generator at targeting the actual source of instability, even though T5 produces fluent text. Code is available at github.com/erfan-nourbakhsh/TRACER.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Dialogue systems

Institutions

Also on the byline: The University of Texas at San Antonio.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.sigdial-1.44

@inproceedings{nourbakhsh2026tracer,
  title = {{TRACER: Early Failure Detection for Task-Oriented Dialogue}},
  author = {Nourbakhsh, Erfan and Slavin, Rocky and Yang, Ke and Rios, Anthony},
  booktitle = {Proceedings of the 27th Annual Meeting of the Special Interest Group on Discourse and Dialogue},
  year = {2026},
  doi = {10.18653/v1/2026.sigdial-1.44},
  url = {https://doi.org/10.18653/v1/2026.sigdial-1.44}
}

References

21 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Matthew Henderson, Blaise Thomson, and Jason D. Williams. 2014. The second dialog state tracking challenge. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 263–272, Philadelphia, PA, U.S.A. Association for Computational Linguistics. find
  2. Ryuichiro Higashinaka, Kotaro Funakoshi, Yuka Kobayashi, and Michimasa Inaba. 2016. The dialogue breakdown detection challenge: Task description, datasets, and evaluation metrics. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3146– 3150, P… find
  3. Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020. A simple language model for task-oriented dialogue. In Advances in Neural Information Processing Systems, volume 33, pages 20179–20191. Curran Associates, Inc. find
  4. Vojtěch Hudeček and Ondrej Dusek. 2023. Are large language models all you need for task-oriented dialogue? In Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 216–228, Prague, Czechia. Association for Computational Linguistics. find
  5. Wenqiang Lei, Xisen Jin, Min-Yen Kan, Zhaochun Ren, Xiangnan He, and Dawei Yin. 2018. Sequicity: Simplifying task-oriented dialogue systems with single sequence-to-sequence architectures. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper… find our card
  6. Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016. Deep reinforcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1192– 1202, Austin, Texas. Association for Computational Ling… find our card
  7. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  8. Elisabeth Maier, Norbert Reithinger, and Jan Alexandersson. 1997. Clarification dialogues as measure to increase robustness in a spoken dialogue system. In Interactive Spoken Dialog Systems: Bringing Speech and NLP Together in Real Applications. find
  9. Susan W. McRoy and Graeme Hirst. 1995. The repair of speech act misunderstandings by abductive inference. Computational Linguistics, 21(4):435–478. find our card
  10. Anh Ngo, Dirk Heylen, Nicolas Rollet, Catherine Pelachaud, and Chloé Clavel. 2024. Exploration of human repair initiation in task-oriented dialogue: A linguistic feature-based approach. In Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 603–609,… find
  11. Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayandeh, Lars Liden, and Jianfeng Gao. 2021. Soloist: Building task bots at scale with transfer learning and machine teaching. Transactions of the Association for Computational Linguistics, 9:807–824. find
  12. Libo Qin, Wenbo Pan, Qiguang Chen, Lizi Liao, Zhou Yu, Yue Zhang, Wanxiang Che, and Min Li. 2023. End-to-end task-oriented dialogue: A survey of tasks, methods, and future directions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5925–5941, Singapor… find
  13. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. find our card
  14. Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8689–8696. find
  15. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. find our card
  16. Marilyn A. Walker, Diane J. Litman, Candace A. Kamm, and Alicia Abella. 1997. PARADISE: A framework for evaluating spoken dialogue agents. In 35th Annual Meeting of the Association for Computational Linguistics and 8th Conference of the European Chapter of the Association for Computational Linguisti… find our card
  17. Tsung-Hsien Wen, David Vandyke, Nikola Mrkšić, Milica Gašić, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the European Chapter of the Association for Computation… find
  18. Chien-Sheng Wu, Steven C.H. Hoi, Richard Socher, and Caiming Xiong. 2020. TOD-BERT: Pre-trained natural language understanding for task-oriented dialogue. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 917–929, Online. Association for Computa… find our card
  19. Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transferable multi-domain state generator for task-oriented dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 808–819, Florence… find
  20. Steve Young, Milica Gašić, Blaise Thomson, and Jason D. Williams. 2013. Pomdp-based statistical spoken dialog systems: A review. Proceedings of the IEEE, 101(5):1160–1179. find
  21. A Experimental Setup A.1 Datasets We train and evaluate TRACER primarily on MultiWOZ 2.4 (Budzianowski et al., 2018), which provides multi-domain dialogues with explicit belief states and rich opportunities for gradual failure through missing information, slot inconsistency, or domain drift. Partial… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nourbakhsh-2026-tracer-early-failure-detection-task/