Elamur Research
☰
home / papers / 2020 / MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines

MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.nlp4convai-1.13.pdf

Summary

MultiWOZ, a multi-domain task-oriented dialogue corpus of more than 10,000 conversations across 8 domains, is a standard benchmark for dialogue state tracking. Even after the MultiWOZ 2.1 cleanup, its state labels remained noisy. The authors list the remaining problems. Some values appear in the state without support in the conversation, such as entities the agent proposes later, entries copied from the database, typos, and silently adjusted times. Others are inconsistent because they come from different sources, are paraphrased, or reflect differing annotator tracking choices. MultiWOZ 2.2 addresses these issues. It corrects state labels in 17.3% of utterances and rebuilds the ontology as a schema that separates categorical slots from non-categorical ones. It also replaces open-vocabulary slot lists with span annotations, so models no longer depend on ad-hoc string matching. Active intents and requested slots are labelled for every user turn. Three trackers were benchmarked on the new version and scored roughly as they did on 2.1. The release offers a cleaner and more comparable benchmark, along with practical guidance for collecting future dialogue data.

Key points

  • Hallucinated state values were found in 3128 turns across 948 dialogues of MultiWOZ 2.1. In addition, 21.0% of ontology values had no exact-string match in the database.
  • The new schema splits slots into categorical and non-categorical types. Non-categorical slots, such as restaurant name and booking time, receive span annotations in both user and system turns.
  • Joint goal accuracy dropped slightly or stayed close when moving from 2.1 to 2.2:
    • TRADE: 0.460 → 0.454
    • SGD-baseline: 0.434 → 0.420
    • DS-DST: 0.512 → 0.517
  • TRADE and SGD-baseline perform clearly better on non-categorical slots than on categorical ones. The authors attribute this to the corrections, which guarantee that non-categorical values appear in the dialogue history.
  • The authors recommend several data-collection practices:
    • Fix the schema before collection begins.
    • Make the interface restrict categorical slots to schema values and non-categorical slots to values already mentioned in the dialogue.
    • Run validation checks after annotation.
  • The dataset is released at github.com/budzianowski/multiwoz.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

147 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 146 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.nlp4convai-1.13

@inproceedings{zang2020multiwoz,
  title = {{MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines}},
  author = {Zang, Xiaoxue and Rastogi, Abhinav and Sunkara, Srinivas and Gupta, Raghav and Zhang, Jianguo and Chen, Jindong},
  booktitle = {Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI},
  year = {2020},
  doi = {10.18653/v1/2020.nlp4convai-1.13},
  url = {https://doi.org/10.18653/v1/2020.nlp4convai-1.13}
}

References

19 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. 2018. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the schema-guided dialogue dataset. arXiv:1909.05855. 2018 Con… find
  2. Bill Byrne, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Ben Goodrich, Daniel Duckworth, Semih Yavuz, Amit Dubey, Kyu-Young Kim, and Andy Cedilnik. 2019. Taskmaster-1: Toward a realistic and diverse dialog dataset. In Proceedings of the 2019 Conference on Empirical Methods in Nat… find
  3. Guan-Lin Chao and Ian Lane. 2019. Bert-dst: Scalable end-to-end dialogue state tracking with bidirectional encoder representations from transformer. arXiv preprint arXiv:1907.03040. — [arXiv:1907.03040](https://arxiv.org/abs/1907.03040) find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
  5. Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyag Gao, and Dilek Hakkani-Tur. 2019. Multiwoz 2.1: Multi-domain dialogue state corrections and state tracking baselines. arXiv preprint arXiv:1907.01669. find
  6. Shuyang Gao, Abhishek Sethi, Sanchit Agarwal, Tagyoung Chung, and Dilek Hakkani-Tur. 2019. Dialog state tracking: A neural reading comprehension approach. Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue. find
  7. Rahul Goel, Shachi Paul, and Dilek Hakkani-Tür. 2019. Hyst: A hybrid approach for flexible and accurate dialogue state tracking. find
  8. Matthew Henderson, Blaise Thomson, and Jason D Williams. 2014. The second dialog state tracking challenge. In Proceedings of the 15th annual meeting of the special interest group on discourse and dialogue, pages 263–272. find
  9. J. F. Kelley. 1984. An iterative design methodology for user-friendly natural language office information applications. ACM Trans. Inf. Syst., page 26–41. find
  10. Sungjin Lee, Qi Zhu, Ryuichi Takanobu, Zheng Zhang, Yaoqin Zhang, Xiang Li, Jinchao Li, Baolin Peng, Xiujun Li, Minlie Huang, et al. 2019. Convlab: Multi-domain end-to-end dialog system platform. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demon… find
  11. Abhinav Rastogi, Dilek Hakkani-Tür, and Larry Heck. 2017. Scalable multi-domain dialogue state tracking. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop, pages 561–568. IEEE. find
  12. Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2019. Towards scalable multi-domain conversational agents: The arXiv preprint Liliang Ren, Kaige Xie, Lu Chen, and Kai Yu. 2018. Towards universal dialogue state tracking. Proceedings of the 2018 Conference on Empiric… find
  13. Pararth Shah, Dilek Hakkani-Tür, Gokhan Tür, Abhinav Rastogi, Ankur Bapna, Neha Nayak, and Larry Heck. 2018. Building a conversational agent overnight with dialogue self-play. arXiv preprint arXiv:1801.04871. — [arXiv:1801.04871](https://arxiv.org/abs/1801.04871) find
  14. Tsung-Hsien Wen, David Vandyke, Nikola Mrkšić, Milica Gasic, Lina M Rojas Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017a. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the European Chapter of the Association for Computation… find
  15. Tsung-Hsien Wen, David Vandyke, Nikola Mrkšić, Milica Gasic, Lina M. Rojas Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017b. A network-based end-to-end trainable task-oriented dialogue system. Proceedings of the 15th Conference of the European Chapter of the Association for Computational… find
  16. Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transferable multi-domain state generator for task-oriented dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. find
  17. Puyang Xu and Qi Hu. 2018. An end-to-end approach for handling unknown slot values in dialogue state tracking. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 1448–1457. find
  18. Jian-Guo Zhang, Kazuma Hashimoto, Chien-Sheng Wu, Yao Wan, Philip S Yu, Richard Socher, and Caiming Xiong. 2019. Find or classify? dual strategy for slot-value predictions on multi-domain dialog state tracking. arXiv preprint arXiv:1910.03544. find
  19. Li Zhou and Kevin Small. 2019. Multi-domain dialogue state tracking as dynamic knowledge graph enhanced question answering. arXiv preprint arXiv:1911.06192. — [arXiv:1911.06192](https://arxiv.org/abs/1911.06192) find