Elamur Research⌕
☰
home / papers / 2018 / Wizard of Wikipedia: Knowledge-Powered Conversational agents

Wizard of Wikipedia: Knowledge-Powered Conversational agents

Paper · 2018 · Emily Dinan, Stephen Roller, Kurt Shuster et al.

Summary

The authors address a long-standing problem in open-domain dialogue: sequence-to-sequence chatbots tend to produce generic phrases that the model has simply memorized in its weights, and they rarely draw on external knowledge. One reason is the lack of a training set in which the conversation is explicitly grounded in a source of facts. The Facebook AI Research team therefore crowdsourced the Wizard of Wikipedia corpus. In each conversation, two people play different roles: a curious "apprentice" asks questions, while the "wizard" sees automatically retrieved passages from Wikipedia and marks the sentence on which their reply is based. On this data, the authors trained the Transformer Memory Networks architecture in two variants, retrieval and generative. It combines the memory of Memory Networks, from which the model selects the relevant knowledge, with a Transformer that encodes the text and generates the response. The best models conduct knowledgeable conversations on open-domain topics, as confirmed by both automatic metrics and human evaluations. However, the example dialogues contain clear factual errors. The main value of the work is that it provides an open benchmark for knowledge-grounded chatbots.

Key points

  • The corpus contains 22,311 dialogues and 201,999 utterances on 1,365 topics, each linked to a Wikipedia article. The split is as follows: 166,787 utterances in train, 17,715 in valid, and 17,497 in test.
  • The test set has two parts. Test Seen covers 533 topics that already appeared in training, while Test Unseen covers 58 entirely new topics.
  • The knowledge base comprises 5.4M articles and 93M sentences. Passages for the "wizard" were selected by the TF-IDF retriever from Open-SQuAD (Chen et al., 2017): the top seven articles for the two most recent utterances plus the article on the initial topic.
  • The Transformer MemNet models (retrieval, two-stage generative, etc.) were compared with baseline Memory Networks and a Transformer without access to knowledge.
  • The data and tasks are published in ParlAI (parl.ai/projects/wizard_of_wikipedia).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Dialogue systems · 624 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Chatbots · 54 works

Cite

Cite as Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli and Jason Weston. 2018. Wizard of Wikipedia: Knowledge-Powered Conversational agents. arXiv:1811.01241.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1811.01241 · arXiv 1811.01241

@misc{dinan2018wizard,
  title = {{Wizard of Wikipedia: Knowledge-Powered Conversational agents}},
  author = {Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli and Jason Weston},
  year = {2018},
  doi = {10.48550/arxiv.1811.01241},
  eprint = {1811.01241},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1811.01241}
}

References

28 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Niels O Bernsen, Hans Dybkjær, and Laila Dybkjær. Designing interactive speech systems: From first ideas to user testing. Springer Science & Business Media, 2012. find
  2. Antoine Bordes, Y-Lan Boureau, and Jason Weston. Learning end-to-end goal-oriented dialog. In Proceedings of the International Conference on Learning Representations (ICLR), 2017. find
  3. Daniel Cer, Yinfei Yang, Sheng yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. Universal sentence encoder. arXiv preprint arXiv:1803.11175, 2018.
  4. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pp. 1870–1879. Association for Computational Linguistics, 2017.
  5. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. QuAC: Question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 2018.
  6. Jesse Dodge, Andreea Gane, Xiang Zhang, Antoine Bordes, Sumit Chopra, Alexander Miller, Arthur Szlam, and Jason Weston. Evaluating prerequisite qualities for learning end-to-end dialog systems. In Proceedings of the International Conference on Learning Representations (ICLR), 2016. find
  7. Layla El Asri, Hannes Schulz, Shikhar Sharma, Jeremie Zumer, Justin Harris, Emery Fine, Rahul Mehrotra, and Kaheer Suleman. Frames: a corpus for adding memory to goal-oriented dialogue systems. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pp. 207–219, Saarbrücken, Ge…
  8. Angela Fan, David Grangier, and Michael Auli. Controllable abstractive summarization. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pp. 45–54. Association for Computational Linguistics, 2018.
  9. Marjan Ghazvininejad, Chris Brockett, Ming-Wei Chang, Bill Dolan, Jianfeng Gao, Wen-tau Yih, and Michel Galley. A knowledge-grounded neural conversation model. In Proceedings of the Conference on Association for the Advancement of Artificial Intelligence (AAAI), 2018. find
  10. Matthew Henderson, Blaise Thomson, and Jason D Williams. The second dialog state tracking challenge. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pp. 263–272, 2014. find
  11. Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun-hsuan Sung, L’aszl’o Luk’acs, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. Efficient Natural Language Response Suggestion for Smart Reply. arXiv preprint arXiv:1705.00652, 2017. find
  12. Jiwei Li, Will Monroe, and Daniel Jurafsky. A simple, fast diverse decoding algorithm for neural generation. arXiv preprint arXiv:1611.08562, 2016. find
  13. Pierre-Emmanuel Mazaré, Samuel Humeau, Martin Raison, and Antoine Bordes. Training millions of personalized dialogue agents. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 2018. Association for Computational Linguistics.
  14. Nikita Moghe, Siddhartha Arora, Suman Banerjee, and Mitesh M. Khapra. Towards exploiting background knowledge for building conversation systems. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2322–2332. Association for Computational Linguistics, 2018.…
  15. Prasanna Parthasarathi and Joelle Pineau. Extending neural generative conversational model using external knowledge sources. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 2018. find
  16. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383–2392, Austin, Texas, November 2016. Association for Computational Lingui…
  17. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pp. 1715–1725, Berlin, Germany, August 2016. Association for Computational Linguistics.
  18. Iulian Vlad Serban, Ryan Lowe, Laurent Charlin, and Joelle Pineau. Generative deep neural networks for dialogue: A short review. arXiv preprint arXiv:1611.06216, 2016. — [arXiv:1611.06216](https://arxiv.org/abs/1611.06216) find
  19. Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. A neural network approach to context-sensitive generation of conversational responses. arXiv preprint arXiv:1506.06714, 2015.
  20. Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. End-to-end memory networks. In Advances in Neural Information Processing Systems, pp. 2440–2448, 2015. find
  21. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pp. 3104–3112, 2014.
  22. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pp. 5998–6008, 2017.
  23. Ashwin K. Vijayakumar, Michael Cogswell, Ramprasaath R. Selvaraju, Qing Sun, Stefan Lee, David J. Crandall, and Dhruv Batra. Diverse beam search: Decoding diverse solutions from neural sequence models. arXiv preprint arXiv:1610.02424, 2016.
  24. Oriol Vinyals and Quoc Le. A neural conversational model. In Proceedings of the 31st International Conference on Machine Learning, Deep Learning Workshop, Lille, France, 2015.
  25. Tsung-Hsien Wen, David Vandyke, Nikola Mrksic, Milica Gasic, Lina M Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. A network-based end-to-end trainable task-oriented dialogue system. arXiv preprint arXiv:1604.04562, 2016.
  26. Ledell Wu, Adam Fisch, Sumit Chopra, Keith Adams, Antoine Bordes, and Jason Weston. Starspace: Embed all the things! arXiv preprint arXiv:1709.03856, 2017. — [arXiv:1709.03856](https://arxiv.org/abs/1709.03856) find
  27. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pp. 2204–2213, Melbourne, Australia, July 2018. Assoc…
  28. Hao Zhou, Tom Young, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. Commonsense knowledge aware conversation generation with graph attention. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI), pp. 4623–4629. International Joint Conferen… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dinan-2018-wizard-wikipedia-knowledge-powered-conversational/