Elamur Research⌕
☰
home / papers / 2020 / CrossWOZ: A Large-Scale Chinese Cross-Domain Task-Oriented Dialogue Dataset

CrossWOZ: A Large-Scale Chinese Cross-Domain Task-Oriented Dialogue Dataset

Paper · 2020 · Qi Zhu, Kaili Huang, Zheng Zhang et al.

Summary

Task-oriented dialogue research has relied mainly on English data. Even multi-domain corpora such as MultiWOZ link domains only through fixed constraints set in advance, and they lack user-side annotation. CrossWOZ addresses both gaps. It is the first large-scale Chinese cross-domain dialogue corpus, collected in a Wizard-of-Oz, human-to-human setup that simulates tourists planning trips in Beijing.

A goal generator built on a crawled entity database creates linked sub-goals. For example, a user might need a hotel close to an attraction picked earlier, then a taxi between the two. Choices made in one domain therefore shape later domains as the conversation unfolds. Crowdworkers played both roles and recorded user and system states. Dialogue acts were then derived by rules, and an expert check showed the annotations to be of high quality.

The authors also release a rule-based user simulator and baselines for each pipeline module. Their experiments show that cross-domain goals are much harder to complete. The resource provides a Chinese testbed for state tracking, policy learning and user simulation.

Key points

  • Size and scope: about 6K dialogues and 102K utterances across 5 domains: hotel, restaurant, attraction, metro and taxi. Roughly 60% of the dialogues involve cross-domain goals.
  • Complexity versus MultiWOZ: the training split has 5,012 dialogues, averaging 3.24 domains and 16.9 turns per dialogue. MultiWOZ averages 1.80 domains.
  • Database: 465 attractions, 951 restaurants and 1,133 hotels. Annotation includes dialogue states and dialogue acts for both user and system.
  • Baselines: BERTNLU, RuleDST, SL policy, TemplateNLG and SC-LSTM, plus a rule-based simulator. System-level results are limited mainly by RuleDST and SL policy. TemplateNLG has a lower BLEU score but outperforms SC-LSTM in simulation.
  • Availability: data and code are at github.com/thu-coai/CrossWOZ, and the paper is licensed CC-BY 4.0.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China137 works · 83,262 citations🔬Beijing National Research Center for Information Science and TechnologyBeijing · China21 works · 5,843 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Dialogue systems · 401 works

Cite

Cite as Qi Zhu, Kaili Huang, Zheng Zhang, Xiaoyan Zhu and Minlie Huang. 2020. CrossWOZ: A Large-Scale Chinese Cross-Domain Task-Oriented Dialogue Dataset. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00314

@article{zhu2020crosswoz,
  title = {{CrossWOZ: A Large-Scale Chinese Cross-Domain Task-Oriented Dialogue Dataset}},
  author = {Qi Zhu, Kaili Huang, Zheng Zhang, Xiaoyan Zhu and Minlie Huang},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.1162/tacl\_a\_00314},
  url = {https://doi.org/10.1162/tacl_a_00314}
}

References

22 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pawel Budzianowski, Iñigo Casanueva, Bo-Hsiang Tseng, and Milica Gasic. 2018a. Towards end-to-end multi-domain dialogue modelling. find
  2. Charles T. Hemphill, John J. Godfrey, and George R. Doddington. 1990. The ATIS spoken language systems pilot corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27,1990. find
  3. Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018b, MultiWOZ -a large-scale multi-domain wizard-of-Oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Langua… find
  4. Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu. 2019. Pre-training with whole word masking for chinese bert. arXiv preprint arXiv:1906.08101. our card
  5. Matthew Henderson, Blaise Thomson, and Jason D. Williams. 2014, The second dialog state tracking challenge. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 263–272. Philadelphia, PA. Association for Computational Linguistics. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019, Jun. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo… our card
  7. Wenqiang Lei, Xisen Jin, Min-Yen Kan, Zhaochun Ren, Xiangnan He, and Dawei Yin. 2018. Sequicity: Simplifying task-oriented dialogue systems with single sequence-to-sequence architectures. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper… our card
  8. Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. 2017. Deal or no deal? end-to-end learning of negotiation dialogues. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2443–2453. Copenhagen, Denmark. Association for Computational Li… our card
  9. Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 100–110. Hong Kong, China. Association for Computational Linguistics. Nikola Mrkšić, Diarmuid Ó Séaghdha, Tsung-Hsien Wen, Blaise Thomson… find
  10. Baolin Peng, Xiujun Li, Lihong Li, Jianfeng Gao, Asli Celikyilmaz, Sungjin Lee, and Kam-Fai Wong. 2017. Composite task-completion dialogue policy learning via hierarchical deep reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 22… find
  11. Zhongyu Wei, Qianlong Liu, Baolin Peng, Huaixiao Tou, Ting Chen, Xuanjing Huang, Kam-fai Wong, and Xiangying Dai. 2018. Task-oriented dialogue system for automatic diagnosis. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 20… our card
  12. Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, Pei-Hao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned LSTM-based natural language generation for spoken dialogue systems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1711–1721.… our card
  13. Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2019. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. arXiv preprint arXiv:1909.05855. find
  14. Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, and Steve Young. 2007. Agenda-based user simulation for bootstrapping a POMDP dialogue system. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volum… our card
  15. Pararth Shah, Dilek Hakkani-Tür, Gokhan Tür, Abhinav Rastogi, Ankur Bapna, Neha Nayak, and Larry Heck. 2018. Building a conversational agent overnight with dialogue self-play. arXiv preprint arXiv:1801.04871. find
  16. Ryuichi Takanobu, Hanlin Zhu, and Minlie Huang. 2019. Guided dialog policy learning: Reward estimation for multi-domain task-oriented dialog. In Proceedings of the 2019 find
  17. Tsung-Hsien Wen, David Vandyke, Nikola Mrkšić, Milica Gašić, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the European Chapter of the Association for Computati… find
  18. Jason Williams, Antoine Raux, Deepak Ramachandran, and Alan Black. 2013. The dialog state tracking challenge. In Proceedings of the SIGDIAL 2013 Conference, pages 404–413. Metz, France. Association for Computational Linguistics. find
  19. Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019, Transferable multi-domain state generator for task-oriented dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 808–819. find
  20. Florence, Italy. Association for Computational Linguistics. find
  21. Qi Zhu, Zheng Zhang, Yan Fang, Xiang Li, Ryuichi Takanobu, Jinchao Li, Baolin Peng, Jianfeng Gao, Xiaoyan Zhu, and Minlie Huang. 2020. Convlab-2: An open-source toolkit for building, evaluating, and diagnosing dialogue systems. arXiv preprint arXiv:2002.04793. find
  22. Kaisheng Yao, Geoffrey Zweig, Mei-Yuh Hwang, Yangyang Shi, and Dong Yu. 2013. Recurrent neural networks for language understanding. In Interspeech, pages 2524–2528. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhu-2020-crosswoz-large-scale-chinese-cross/