Elamur Research
☰
home / papers / 2017 / DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset

DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1710.03957

Summary

Training data for open-domain chatbots tends to be either small and tied to a single task, or large but flawed. Social-media post–reply corpora are noisy and short. Subtitle and telephone corpora such as OpenSubtitles and Switchboard stretch one "conversation" over hundreds of turns and several topics. To fill this gap, the authors built DailyDialog, a corpus of English two-person dialogues about everyday life. They collected the dialogues from websites where English learners practise conversation, then removed duplicates, dropped dialogues with more than two speakers, and fixed spelling automatically. Three expert annotators labelled every utterance for communicative intention (dialog act) and emotion. The final corpus is short, topic-focused and relatively clean, and it shows natural exchange patterns such as a question followed by an answer, or a proposal followed by a commitment. The authors tested existing retrieval-based and generation-based models on it. Pre-training on OpenSubtitles gave lower perplexity but worse BLEU, which they attribute to the domain mismatch. The resource matters because it supports research on dialog management, emotion-aware response generation and domain adaptation.

Key points

  • Size and shape: 13,118 dialogues, averaging 7.9 speaker turns, 114.7 tokens per dialogue and 14.6 tokens per utterance.
  • Dialog act labels: four classes (Inform, Questions, Directives, Commissive), following Amanova et al. (2016), which is compatible with ISO 24617-2.
    • Their shares are 45.2%, 28.6%, 16.8% and 9.4% respectively.
  • Emotion labels: seven categories, consisting of Ekman's "Big Six" plus an extra class for other emotions.
    • The three annotators reached 78.9% inter-annotator agreement.
  • Topics: ten categories, led by Relationship (33.33%), Ordinary Life (28.26%) and Work (14.49%).
  • Pre-training finding: models pre-trained on OpenSubtitles converged faster and reached lower PPL, but scored worse on BLEU.
    • They also produced responses less relevant to the context than models trained only on DailyDialog.
    • The dataset is available at yanran.li/dailydialog.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

549 papers of our library cite it (of the 140673 whose bibliographies we read); 15 of them have a page here, 534 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1710.03957 · arXiv 1710.03957

@misc{li2017dailydialog,
  title = {{DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset}},
  author = {Li, Yanran and Su, Hui and Shen, Xiaoyu and Li, Wenjie and Cao, Ziqiang and Niu, Shuzi},
  year = {2017},
  doi = {10.48550/arxiv.1710.03957},
  eprint = {1710.03957},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1710.03957}
}

References

32 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martı́n Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Gregory S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian J. Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Józefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh… find our card
  2. Rami Al-Rfou’, Marc Pickett, Javier Snaider, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2016. Conversational contextual cues: The case of personalization and history for response ranking. CoRR, abs/1606.00372. — [arXiv:1606.00372](https://arxiv.org/abs/1606.00372) find
  3. Dilafruz Amanova, Volha Petukhova, and Dietrich Klakow. 2016. Creating annotated dialogue resources: Cross-domain dialogue act classification. In LREC. find
  4. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473. find our card
  5. Antoine Bordes and Jason Weston. 2016. ing end-to-end goal-oriented dialog. abs/1605.07683. find
  6. Learn-CoRR, Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259. find our card
  7. Jesse Dodge, Andreea Gane, Xiang Zhang, Antoine Bordes, Sumit Chopra, Alexander H. Miller, Arthur Szlam, and Jason Weston. 2015. Evaluating prerequisite qualities for learning end-to-end dialog systems. CoRR, abs/1511.06931. find
  8. Paul Ekman. 1992. An argument for basic emotions. Cognition & emotion, 6(3-4):169–200. find our card
  9. John J Godfrey, Edward C Holliman, and Jane Mc-Daniel. 1992. Switchboard: Telephone speech corpus for research and development. In Acoustics, Speech, and Signal Processing, 1992. ICASSP-92., 1992 IEEE International Conference on, volume 1, pages 517–520. IEEE. find
  10. Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In NIPS. find our card
  11. Sina Jafarpour, Christopher JC Burges, and Alan Ritter. 2010. Filter, rank, and transfer the knowledge: Learning to chat. Advances in Ranking, 10. find
  12. Jörg Tiedemann. 2009. News from OPUS A Collection of Multilingual Parallel Corpora with Tools and Interfaces. find
  13. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. International Conference for Learning Representations. find
  14. Jiwei Li, Will Monroe, Alan Ritter, Daniel Jurafsky, Michel Galley, and Jianfeng Gao. 2016. Deep reinforcement learning for dialogue generation. In EMNLP. find our card
  15. Jiwei Li, Will Monroe, Tianlin Shi, Alan Ritter, and Dan Jurafsky. 2017. Adversarial learning for neural dialogue generation. arXiv preprint arXiv:1701.06547. find
  16. Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In EMNLP. find
  17. Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. In SIGDIAL Conference. find our card
  18. Chuwei Luo and Wenjie Li. 2016. A combination of similarity and rule-based method of polyu for ntcir-12 stc task. In Proceedings of the 12th NTCIR Conference on Evaluation of Information Access Technologies. find
  19. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In EMNLP. find
  20. Hongyuan Mei, Mohit Bansal, and Matthew R. Walter. 2017. Coherent dialogue with attention-based language models. In AAAI. find
  21. Atsushi Otsuka, Toru Hirano, Chiaki Miyazaki, Ryuichiro Higashinaka, Toshiro Makino, and Yoshihiro Matsuo. 2017. Utterance selection using discourse relation filter for chat-oriented dialogue systems. In Dialogues with Social Robots, pages 355– 365. Springer. find
  22. Volha Petukhova. 2011. Emultidimensional dialogue modelling. phd dissertation. Tilburg University, The Netherlands. find
  23. Volha Petukhova, Martin Gropp, Dietrich Klakow, Anna Schmidt, Gregor Eigner, Mario Topf, Stefan Srb, Petr Motlicek, Blaise Potard, John Dines, et al. 2014. The dbox corpus collection of spoken human-human and human-machine dialogues. In Proceedings of the Ninth International Conference on Language R… find
  24. Eric K Ringger, James F Allen, Bradford W Miller, and Teresa Sikorski. 1996. A robust system for natural spoken dialogue. find
  25. Alan Ritter, Colin Cherry, and William B Dolan. 2011. Data-driven response generation in social media. In Proceedings of the conference on empirical methods in natural language processing, pages 583–593. Association for Computational Linguistics. find
  26. Iulian Serban, Alessandro Sordoni, Yoshua Bengio, Aaron C. Courville, and Joelle Pineau. 2016a. Building end-to-end dialogue systems using generative hierarchical neural network models. In AAAI. find our card
  27. Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2016b. A hierarchical latent variable encoder-decoder model for generating dialogues. arXiv preprint arXiv:1605.06069. find our card
  28. Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. Neural responding machine for short-text conversation. In ACL. find our card
  29. Alessandro Sordoni, Yoshua Bengio, Hossein Vahabi, Christina Lioma, Jakob Grue Simonsen, and Jian-Yun Nie. 2015. A hierarchical recurrent encoder-decoder for generative context-aware query suggestion. In Cikm’15 Proceedings of the 24th Acm International on Conference on Information and Knowledge Man… find
  30. Hao Wang, Zhengdong Lu, Hang Li, and Enhong Chen. 2013. A dataset for research on short-text conversations. In EMNLP, pages 935–945. find
  31. Zhao Yan, Nan Duan, Junwei Bao, Peng Chen, Ming Zhou, Zhoujun Li, and Jianshe Zhou. 2016. Doc-chat: an information retrieval approach for chatbot engines using unstructured documents. In ACL. find
  32. Hao Zhou, Minlie Huang, Tianyang Zhang, Xiaoyan Zhu, and Bing Liu. 2017. Emotional chatting machine: Emotional conversation generation with internal and external memory. arXiv preprint arXiv:1704.01074. — [arXiv:1704.01074](https://arxiv.org/abs/1704.01074) find