Elamur Research
☰
home / papers / 2015 / The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems

The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1506.08909

Summary

Neural dialogue systems have advanced more slowly than vision or speech systems. The authors attribute this mainly to the lack of large public conversational datasets. Slot-filling corpora such as DSTC are small, and Twitter/Weibo collections are mostly unreleased and dominated by very short exchanges. To address this, the authors build the Ubuntu Dialogue Corpus from Ubuntu technical-support channels on the Freenode IRC network. These channels interleave many parallel conversations, so the authors use a heuristic disentanglement step. Each message becomes a (time, sender, recipient, utterance) tuple. Recipients are detected by matching the first word against recently seen usernames while excluding common English words. Two-person dialogues are then reassembled from these tuples. The authors also define a next-response selection task that requires no manual labeling and report TF-IDF, RNN and LSTM baselines, with the LSTM performing best. The corpus matters because it pairs the multi-turn nature of DSTC data with the open-ended style of microblogs, at a scale large enough to train neural models.

Key points

  • Scale: The corpus contains roughly 930,000 dyadic dialogues, about 7.1 million utterances and around 100 million words. Dialogues average 8 turns, with at least 3 turns each.
  • Source: The data comes from Ubuntu IRC chat logs covering 2004–2015. It differs from the earlier Ubuntu Chat Corpus, which keeps the multi-party structure instead of isolating two-person conversations.
  • Task and baselines: The evaluation task is response selection, where the model ranks the true reply against randomly chosen false candidates (up to 1-in-10) and is scored with Recall@k. The baselines are TF-IDF, an RNN and an LSTM with one hidden layer of 200 units, and the LSTM is significantly better than the others.
  • Why selection instead of generation: The authors argue that BLEU/METEOR-style scoring of generated replies is unreliable for dialogue. Selection difficulty can also be tuned by changing the number of distractors or the value of k.
  • Availability: The code is at github.com/npow/ubottu. An updated, bug-fixed version of the dataset is at github.com/rkadlec/ubuntu-ranking-dataset-creator, and its results should not be compared directly with those on the original dataset.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

275 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 263 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1506.08909 · arXiv 1506.08909

@misc{lowe2015ubuntu,
  title = {{The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems}},
  author = {Lowe, Ryan and Pow, Nissan and Serban, Iulian and Pineau, Joelle},
  year = {2015},
  doi = {10.48550/arxiv.1506.08909},
  eprint = {1506.08909},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1506.08909}
}

References

33 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 35(8):1798–1828, 2013. find our card
  2. A. Bordes, J. Weston, and N. Usunier. Open question answering with weakly supervised embedding models. In MLKDD, pages 165– 180. Springer, 2014. find
  3. J. Boyd-Graber, B. Satinoff, H. He, and H. Daume. Besting the quiz master: Crowdsourcing incremental classification games. In EMNLP, 2012. find
  4. K. Cho, B. van Merrienboer, C. Gulcehre, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014. find
  5. J. Deng, W. Dong, R. Socher, L.J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. find
  6. M. Elsner and E. Charniak. You talking to me? a corpus and algorithm for conversation disentanglement. In ACL, pages 834– 842, 2008. find
  7. M. Galley, C. Brockett, A. Sordoni, Y. Ji, M. Auli, C. Quirk, M. Mitchell, J. Gao, and B. Dolan. deltableu: A discriminative metric for generation tasks with intrinsically diverse targets. arXiv preprint arXiv:1506.06863, 2015. find
  8. J.J. Godfrey, E.C. Holliman, and J. Mc-Daniel. Switchboard: Telephone speech corpus for research and development. In ICASSP, 1992. find
  9. M. Henderson, B. Thomson, and J. Williams. Dialog state tracking challenge 2 & 3, 2014. find
  10. M. Henderson, B. Thomson, and J. Williams. The second dialog state tracking challenge. In SIGDIAL, page 263, 2014. find
  11. M. Henderson, B. Thomson, and S. Young. Word-based dialog state tracking with recurrent neural networks. In SIGDIAL, page 292, 2014. find
  12. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. find
  13. S. Jafarpour, C. Burges, and A. Ritter. Filter, rank, and transfer the knowledge: Learning to chat. Advances in Ranking, 10, 2010. find
  14. D.P. Kingma and J. Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014. find
  15. A. Lavie and M.J. Denkowski. The ME-TEOR metric for automatic evaluation of Machine Translation. Machine Translation, 23(2-3):105–115, 2009. find
  16. L.R. Medsker and L.C. Jain. Recurrent neural networks. Design and Applications, 2001. find
  17. K. Papineni, S. Roukos, T. Ward, and W.J. Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL, 2002. find our card
  18. J. Pennington, R. Socher, and C.D. Manning. GloVe: Global Vectors for Word Representation. In EMNLP, 2014. find
  19. J. Ramos. Using tf-idf to determine word relevance in document queries. In ICML, 2003. find
  20. A. Ritter, C. Cherry, and W. Dolan. Unsupervised modeling of twitter conversations. 2010. find
  21. A. Ritter, C. Cherry, and W. Dolan. Data-driven response generation in social media. In EMNLP, pages 583–593, 2011. find
  22. A.M. Saxe, J.L. McClelland, and S. Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013. find
  23. J. Schatzmann, K. Georgila, and S. Young. Quantitative evaluation of user simulation techniques for spoken dialogue systems. In SIGDIAL, 2005. find
  24. L. Shang, Z. Lu, and H. Li. Neural responding machine for short-text conversation. arXiv preprint arXiv:1503.02364, 2015. find our card
  25. B. A. Shawar and E. Atwell. Chatbots: are they really useful? In LDV Forum, volume 22, pages 29–49, 2007. find
  26. S. Singh, D. Litman, M. Kearns, and M. Walker. Optimizing dialogue management with reinforcement learning: Experiments with the NJFun system. Journal of Artificial Intelligence Research, 16:105–133, 2002. find
  27. A. Sordoni, M. Galley, M. Auli, C. Brockett, Y. Ji, M. Mitchell, J.Y. Nie, J. Gao, and W. Dolan. A neural network approach to context-sensitive generation of conversational responses. 2015. find our card
  28. D.C. Uthus and D.W. Aha. Extending word highlighting in multiparticipant chat. Technical report, DTIC Document, 2013. find
  29. D.C. Uthus and D.W Aha. The ubuntu chat corpus for multiparticipant chat analysis. In AAAI Spring Symposium on Analyzing Microtext, pages 99–102, 2013. find
  30. H. Wang, Z. Lu, H. Li, and E. Chen. A dataset for research on short-text conversations. In EMNLP, 2013. find
  31. J. Williams, A. Raux, D. Ramachandran, and A. Black. The dialog state tracking challenge. In SIGDIAL, pages 404–413, 2013. find
  32. L. Yu, K. M. Hermann, P. Blunsom, and S. Pulman. Deep learning for answer sentence selection. arXiv preprint arXiv:1412.1632, 2014. — [arXiv:1412.1632](https://arxiv.org/abs/1412.1632) find
  33. M.D. Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012. find our card