Elamur Research⌕
☰
home / papers / 2019 / Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset

Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset

Summary

Data-driven dialog systems have been held back by a shortage of high-quality, goal-oriented conversational data. Existing corpora often come from deployed systems that carry their limitations, or they rely on small knowledge bases and tight scripts. The authors built Taskmaster-1, a corpus of 13,215 task-based English dialogs across six everyday service domains, using two collection methods. In the first, crowdworkers spoke with trained human operators who posed as an automated assistant (a spoken Wizard of Oz setup). In the second, individual crowdworkers wrote both sides of a conversation from a scenario ("self-dialogs"). Instead of full dialog-state or dialog-act schemes, the dialogs are annotated with API calls and their arguments, which is cheaper and easier to apply consistently. Compared with MultiWOZ, Taskmaster-1 has a larger vocabulary and more real-world entities, and language models find it harder to fit. Baselines range from n-gram models to a Transformer. Automatic metrics broadly agreed with human judgments. The resource supports research on task-oriented dialog and on spoken versus written conversation.

Key points

  • Composition: 5,507 spoken two-person (WOz) dialogs and 7,708 written self-dialogs.
    • Six domains: pizza ordering, auto repair appointments, ride service, movie tickets, coffee orders, and restaurant reservations.
    • Dialogs average roughly 22–23 utterances.
  • Comparison with MultiWOZ: The self-dialogs contain 21,894 unique words versus 19,175 in MultiWOZ, along with many more unique named entities.
  • Response generation: Baselines included 3/4-gram, LSTM (with and without attention), fconv Convolution, Transformer, and GPT-2 (117M).
    • The Transformer achieved the best automatic scores (PPL 18.19, BLEU 6.11) and was ranked first by human judges.
    • LSTM-attention received the highest LIKERT rating (3.51).
  • Human evaluation reliability: Krippendorff's alpha was 0.29 for ranking and 0.21 for rating, so ranking produced more consistent judgments.
  • API argument prediction: A Transformer with a copy mechanism reached 51.79% micro F1, compared with 48.73% for the plain Transformer. The dataset is available at g.co/dataset/taskmaster-1.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

105 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 99 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1459

@inproceedings{byrne2019taskmaster,
  title = {{Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset}},
  author = {Byrne, Bill and Krishnamoorthi, Karthik and Sankar, Chinnadhurai and Neelakantan, Arvind and Goodrich, Ben and Duckworth, Daniel and Yavuz, Semih and Dubey, Amit and Kim, Kyu-Young and Cedilnik, Andy},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1459},
  url = {https://doi.org/10.18653/v1/d19-1459}
}

References

33 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amazon. 2019. Alexa skills. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. ICLR. find our card
  3. Antoine Bordes, Y-Lan Boureau, and Jason Weston. 2017. Learning end-to-end goal-oriented dialog. ICLR. find
  4. Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. 2018. Multiwoz - a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. EMNLP. find
  5. Wallace Chafe and Deborah Tannen. 1987. The relation between written and spoken language. Annual Review of Anthropology. find
  6. J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N. Dauphin. 2017. Convolutional Sequence to Sequence Learning. Arxiv. find our card
  7. Ryan Lowe, Nissan Pow, Iulian V. Serban, and Joelle Pineau. 2015. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. SIGDIAL. find our card
  8. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. ICLR. find our card
  9. Nikita Moghe, Siddhartha Arora, Suman Banerjee, and Mitesh M. Khapra. 2018. Towards exploiting background knowledge for building conversation systems. EMNLP. find our card
  10. Aron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alexander Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. 2016. Wavenet: A generative model for raw audio. Arxiv. find our card
  11. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. NAACL Demonstrations. find our card
  12. Google. 2019. Actions on google. find
  13. Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. ACL. find our card
  14. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Arxiv. find
  15. Matthew Henderson, Blaise Thomson, and Steve Young. 2013. Deep neural network approach for the dialog state tracking challenge. SIGDIAL. find
  16. Antoine Raux, Brian Langner, Alan W Black, and Maxine Eskenazi. 2003. Lets go: Improving spoken dialog systems for the elderly and non-natives. Eurospeech. find
  17. Geoffrey Hinton, Li Deng, Dong Yu, George Dahl, Abdel rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara Sainath, and Brian Kingsbury. 2012. Deep neural networks for acoustic modeling in speech recognition. Signal Processing Magazine. find
  18. Lina Maria Rojas-Barahona, Milica Gasic, Nikola Mrksic, Pei-Hao Su, Stefan Ultes, Tsung-Hsien Wen, Steve J. Young, and David Vandyke. 2017. A network-based end-to-end trainable task-oriented dialogue system. EACL. find
  19. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation. find
  20. Norman P. Jouppi, Cliff Young, Nishant Patil, David Patterson, and Gaurav Agrawal et al. 2017. In-datacenter performance analysis of a tensor processing unit. ISCA. find
  21. John F Kelley. 1984. An iterative design methodology for user-friendly natural language office information applications. ACM Transactions on Information Systems (TOIS). find
  22. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. ICLR. find
  23. Ben Krause, Marco Damonte, Mihai Dobre, Daniel Duma, Joachim Fainberg, Federico Fancellu, Emmanuel Kahembwe, Jianpeng Cheng, and Bonnie Webber. 2017. Edina: Building an open domain socialbot with self-dialogues. Arxiv. find
  24. Yann Lecun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature. find
  25. Iulian Vlad Serban, Ryan Lowe, Peter Henderson, Laurent Charlin, and Joelle Pineau. 2017. A survey of available corpora for building data-driven dialogue systems. nnn. find
  26. Dave Steinkrau, Patrice Y. Simard, and Ian Buck. 2005. Using gpus for machine learning algorithms. IC-DAR. find
  27. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. NeurIPS. find
  28. Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018. Tensor2tensor for neural machine translation. CoRR. find our card
  29. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. NeurIPS. find
  30. Oriol Vinyals and Quoc V. Le. 2015. A neural conversational model. Arxiv. find our card
  31. Joseph Weizenbaum. 1966. Eliza a computer program for the study of natural language communication between man and machine. Computational Linguistics. find
  32. Jason Williams, Antoine Raux, and Matthew and Henderson. 2016. The dialog state tracking challenge series: A review. Dialog and Discourse. find
  33. Jason D. Williams and Steve J. Young. 2007. Partially observable markov decision processes for spoken dialog systems. Computer Speech & Language. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/byrne-2019-taskmaster-1-realistic-diverse-dialog/