Elamur Research⌕
☰
home / papers / 2015 / Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems

Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems

Paper · 2015 · Jesse Dodge, Andreea Gane, Xiang Zhang et al.

Summary

End-to-end neural dialog systems trained on raw human conversations are usually judged by BLEU, crowdsourcing or a few human ratings. These measures show whether replies sound plausible, not whether a system can reach a goal. The synthetic bAbI tasks are diagnostic but too small to reward scalable methods. Researchers at Facebook AI Research address this with the Movie Dialog dataset, built from OMDb, MovieLens and Reddit. It contains four movie-domain tasks plus a joint one:

  • factoid QA over a knowledge base of triples
  • personalized recommendation
  • three-turn dialogs mixing QA and recommendation
  • picking the right reply in Reddit threads

Models rank candidate answers and are scored with hits@k, so no human judges are needed. Compared systems include supervised embeddings, LSTMs, IR baselines, SVD, a dedicated QA system and Memory Networks (MemN2N). MemN2N was the strongest general-purpose model, especially where dialog context matters. LSTMs struggled on Reddit, and performing all tasks in one joint model proved harder. MemN2N also clearly outperformed the original baselines on the Ubuntu Dialog Corpus.

Key points

  • The dataset covers about 75k movie entities and about 3.5M training examples. It is available via fb.ai/babi.
  • Task 1 (QA) uses about 96k/10k/10k train/dev/test questions across 11 question types. Templates come from SimpleQuestions, and the knowledge base has 8 relation types. Test hits@1 was 90.7 for the QA system of Bordes et al. (2014), 78.9 for MemN2N and 50.9 for supervised embeddings.
  • Task 2 (Recommendation) draws on MovieLens ratings of about 11k movies and about 110k training users. It provides 1M generated training examples and is scored with hits@100.
  • Task 3 (QA+Recommendation) is scored with hits@10. MemN2N with 2 hops reached 76.8, compared with 56.0 for supervised embeddings and 19.9 for an LSTM.
  • Task 4 (Reddit) is also scored with hits@10. MemN2N reached about 29, against 27.6 for supervised embeddings and 23.7 for the best IR baseline. LSTMs tended to favor short replies until hidden states were averaged.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MetaMenlo Park · United States214 works · 241,492 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,341 worksDialogue systems · 654 works

Cite

Cite as Jesse Dodge, Andreea Gane, Xiang Zhang, Antoine Bordes, Sumit Chopra, Alexander Miller, Arthur Szlam and Jason Weston. 2015. Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems. arXiv:1511.06931.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1511.06931 · arXiv 1511.06931

@misc{dodge2015evaluating,
  title = {{Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems}},
  author = {Jesse Dodge, Andreea Gane, Xiang Zhang, Antoine Bordes, Sumit Chopra, Alexander Miller, Arthur Szlam and Jason Weston},
  year = {2015},
  doi = {10.48550/arxiv.1511.06931},
  eprint = {1511.06931},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1511.06931}
}

References

46 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio, Yoshua. Neural machine translation by jointly learning to align and translate. ICLR 2015, 2015. 11 Results for the baselines from (Lowe et al., 2015) differ to that from the v3 of the arxiv paper, because the corpus has been updated since then. All resu…
  2. V ALIDATION ( HITS @1) N / A N / A N / A 57.23 64.28 64.31 64.01 T EST ( HITS @1) 48.81 37.91 55.22 56.25 63.51 63.72 62.82 Bai, Bing, Weston, Jason, Grangier, David, Collobert, Ronan, Sadamasa, Kunihiko, Qi, Yanjun, Chapelle, Olivier, and Weinberger, Kilian. Supervised semantic indexing. In Proceed… find
  3. Berant, Jonathan and Liang, Percy. Semantic parsing via paraphrasing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL’14), Baltimore, USA, 2014.
  4. Berant, Jonathan, Chou, Andrew, Frostig, Roy, and Liang, Percy. Semantic parsing on freebase from question-answer pairs. In EMNLP, pp. 1533–1544, 2013.
  5. Bordes, Antoine, Chopra, Sumit, and Weston, Jason. Question answering with subgraph embeddings. In Proc. EMNLP, 2014. find
  6. Bordes, Antoine, Usunier, Nicolas, Chopra, Sumit, and Weston, Jason. Large-scale simple question answering with memory networks. arXiv preprint arXiv:1506.02075, 2015. find
  7. Cremonesi, Paolo, Koren, Yehuda, and Turrin, Roberto. Performance of recommender algorithms on top-n recommendation tasks. In Proceedings of the fourth ACM conference on Recommender systems, pp. 39–46. ACM, 2010. find
  8. Fader, Anthony, Zettlemoyer, Luke, and Etzioni, Oren. Open question answering over curated and extracted knowledge bases. In Proceedings of 20th SIGKDD Conference on Knowledge Discovery and Data Mining (KDD’14), New York City, USA, 2014. ACM. find
  9. Graves, Alex et al. Supervised sequence labelling with recurrent neural networks, volume 385. Springer, 2012. find
  10. Griol, David, Hurtado, Lluı́s F, Segarra, Encarna, and Sanchis, Emilio. A statistical approach to spoken dialog systems design and evaluation. Speech Communication, 50(8):666–682, 2008. find
  11. Henderson, Matthew. Machine learning for dialog state tracking: A review. In Proceedings of The First International Workshop on Machine Learning in Spoken Language Processing, 2015. find
  12. Henderson, Matthew, Thomson, Blaise, and Williams, Jason. The second dialog state tracking challenge. In 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 263, 2014. find
  13. Hermann, Karl Moritz, Kočiský, Tomáš, Grefenstette, Edward, Espeholt, Lasse, Kay, Will, Suleyman, Mustafa, and Blunsom, Phil. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems (NIPS), 2015. URL http://arxiv.org/abs/1506.03340. ↗
  14. Hill, Felix, Bordes, Antoine, Chopra, Sumit, and Weston, Jason. The goldilocks principle: Reading children’s books with explicit memory representations. arXiv preprint arXiv:1511.02301, 2015.
  15. Hirschman, Lynette, Dahl, Deborah A, McKay, Donald P, Norton, Lewis M, and Linebarger, Marcia C. Beyond class a: A proposal for automatic evaluation of discourse. Technical report, DTIC Document, 1990. find
  16. Hochreiter, Sepp and Schmidhuber, Jürgen. Long short-term memory. Neural computation, 9(8): 1735–1780, 1997. find
  17. Isbell, Charles Lee, Kearns, Michael, Kormann, Dave, Singh, Satinder, and Stone, Peter. Cobot in lambdamoo: A social statistics agent. In AAAI/IAAI, pp. 36–41, 2000. find
  18. Jafarpour, Sina, Burges, Christopher JC, and Ritter, Alan. Filter, rank, and transfer the knowledge: Learning to chat. Advances in Ranking, 10, 2010. find
  19. Joulin, Armand and Mikolov, Tomas. Inferring algorithmic patterns with stack-augmented recurrent nets. arXiv preprint: 1503.01007, 2015. find
  20. Kolomiyets, Oleksandr and Moens, Marie-Francine. A survey on question answering technology from an information retrieval perspective. Information Sciences, 181(24):5412–5434, 2011. find
  21. Koren, Yehuda. Factorization meets the neighborhood: a multifaceted collaborative filtering model. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 426–434. ACM, 2008. find
  22. Koren, Yehuda and Bell, Robert. Advances in collaborative filtering. In Recommender systems handbook, pp. 145–186. Springer, 2011. find
  23. Kwiatkowski, Tom, Choi, Eunsol, Artzi, Yoav, and Zettlemoyer, Luke. Scaling semantic parsers with on-the-fly ontology matching. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP’13), Seattle, USA, October 2013.
  24. Lowe, Ryan, Pow, Nissan, Serban, Iulian, and Pineau, Joelle. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. arXiv preprint arXiv:1506.08909, 2015.
  25. Mikolov, Tomas and Zweig, Geoffrey. Context dependent recurrent neural network language model. In SLT, pp. 234–239, 2012. find
  26. Mikolov, Tomáš, Kombrink, Stefan, Burget, Lukáš, Černockỳ, Jan Honza, and Khudanpur, Sanjeev. Extensions of recurrent neural network language model. In Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEE International Conference on, pp. 5528–5531. find
  27. Mikolov, Tomas, Chen, Kai, Corrado, Greg, and Dean, Jeffrey. Efficient estimation of word representations in vector space. arXiv:1301.3781, 2013.
  28. Narasimhan, Karthik, Kulkarni, Tejas, and Barzilay, Regina. Language understanding for text-based games using deep reinforcement learning. arXiv preprint arXiv:1506.08941, 2015.
  29. Paek, Tim. Empirical methods for evaluating dialog systems. In Proceedings of the workshop on Evaluation for Language and Dialogue Systems-Volume 9, pp. 2. Association for Computational Linguistics, 2001. find
  30. Ramachandran, Deepak, Yeh, Peter Z, Jarrold, William, Douglas, Benjamin, Ratnaparkhi, Adwait, Provine, Ronald, Mendel, Jeremy, and Emfield, Adam. An end-to-end dialog system for tv program discovery. In Spoken Language Technology Workshop (SLT), 2014 IEEE, pp. 602–607. find
  31. Ritter, Alan, Cherry, Colin, and Dolan, William B. Data-driven response generation in social media. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 583–593. Association for Computational Linguistics, 2011.
  32. Rojas-Barahona, Lina M, Lorenzo, Alejandra, and Gardent, Claire. An end-to-end evaluation of two situated dialog systems. In Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 10–19. Association for Computational Linguistics, 2012. find
  33. Rush, Alexander M, Chopra, Sumit, and Weston, Jason. A neural attention model for abstractive sentence summarization. Proceedings of EMNLP, 2015.
  34. Shang, Lifeng, Lu, Zhengdong, and Li, Hang. Neural responding machine for short-text conversation. arXiv preprint arXiv:1503.02364, 2015.
  35. Sordoni, Alessandro, Galley, Michel, Auli, Michael, Brockett, Chris, Ji, Yangfeng, Mitchell, Margaret, Nie, Jian-Yun, Gao, Jianfeng, and Dolan, Bill. A neural network approach to context-sensitive generation of conversational responses. Proceedings of NAACL, 2015.
  36. Sukhbaatar, Sainbayar, Szlam, Arthur, Weston, Jason, and Fergus, Rob. End-to-end memory networks. Proceedings of NIPS, 2015. find
  37. Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc VV. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104–3112, 2014.
  38. Vinyals, Oriol and Le, Quoc. A neural conversational model. arXiv preprint arXiv:1506.05869, 2015.
  39. Walker, Marilyn A, Litman, Diane J, Kamm, Candace A, and Abella, Alicia. Paradise: A framework for evaluating spoken dialogue agents. In Proceedings of the eighth conference on European chapter of the Association for Computational Linguistics, pp. 271–280. Association for Computational Linguistics,…
  40. Walker, Marilyn A, Prasad, Rashmi, and Stent, Amanda. A trainable generator for recommendations in multimodal dialog. In INTERSPEECH, 2003. find
  41. Weston, J., Bordes, A., Chopra, S., and Mikolov, T. Towards AI-complete question answering: A set of prerequisite toy tasks. arXiv preprint: 1502.05698, 2015a.
  42. Weston, Jason, Yee, Hector, and Weiss, Ron J. Learning to rank recommendations with the k-order statistic loss. In Proceedings of the 7th ACM conference on Recommender systems, pp. 245–248. ACM, 2013. find
  43. Weston, Jason, Bordes, Antoine, Chopra, Sumit, and Mikolov, Tomas. Towards ai-complete question answering: a set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698, 2015b.
  44. Weston, Jason, Chopra, Sumit, and Bordes, Antoine. Memory networks. Proceedings of ICLR, 2015c. find
  45. Whittaker, Steve, Walker, Marilyn A, and Moore, Johanna D. Fish or fowl: A wizard of oz evaluation of dialogue strategies in the restaurant domain. In LREC, 2002. find
  46. Williams, Jason, Raux, Antoine, Ramachandran, Deepak, and Black, Alan. The dialog state tracking challenge. In Proceedings of the SIGDIAL 2013 Conference, pp. 404–413, 2013. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dodge-2015-evaluating-prerequisite-qualities-learning-end/