Elamur Research
☰
home / papers / 2020 / Towards a Human-like Open-Domain Chatbot

Towards a Human-like Open-Domain Chatbot

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2001.09977

Summary

Open-domain chatbots at the time were either built on elaborate hand-engineered pipelines or tended to give replies that made no sense or said nothing in particular. This Google Brain paper introduces Meena, a 2.6B-parameter generative model trained end-to-end. Its only objective is to predict the next token over about 40B words of filtered public-domain social media dialogue. The model is a seq2seq Evolved Transformer that reads up to seven previous turns and generates the reply. To measure quality, the authors design a crowd-sourced metric called Sensibleness and Specificity Average (SSA). Raters first judge whether a response makes sense in context. If it does, they judge whether it fits that particular context or is a generic reply. Across many model variants, lower perplexity closely tracks higher SSA, unlike the weak links reported for metrics such as BLEU. Base Meena reaches 72% SSA, and adding a filtering step with tuned decoding raises it to 79%. Humans score 86%, and none of the other chatbots tested scored above 56%. The results suggest that better perplexity alone could bring chatbots close to human-level conversation.

Key points

  • Meena configuration: 2.6B parameters, Evolved Transformer seq2seq, 8K BPE subword vocabulary, and a test perplexity of 10.2 after training on 40B words of social media conversations.
  • SSA penalizes bland answers. A trivial GenericBot that always replies "I don't know" or "ok" is rated 70% sensible, above DialoGPT's 62%, but its SSA is only 35% compared with DialoGPT's 51%. Crowd worker agreement was 76% for sensibleness and 66% for specificity (Krippendorff's alpha 0.42 and 0.30).
  • Perplexity and interactive SSA correlate strongly across Meena versions (R² = 0.93). SSA also lines up with separate "human-like" judgments (R² = 0.96).
  • Interactive SSA: humans 86%, full Meena 79%, base Meena 72%, Mitsuku and Cleverbot 56% each, DialoGPT 48%, XiaoIce 31%. Static evaluation used a set of 1,477 multi-turn conversations. Chat transcripts are published in the google-research GitHub repository (meena folder).
  • Limitations:
    • Static contexts are only 1–3 turns long and mostly casual, Turing-test-style talk.
    • Interactive chats last only 14–28 turns.
    • The metric leaves out qualities such as empathy, humor and deep reasoning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

342 papers of our library cite it (of the 140673 whose bibliographies we read); 21 of them have a page here, 321 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2001.09977 · arXiv 2001.09977

@misc{adiwardana2020human,
  title = {{Towards a Human-like Open-Domain Chatbot}},
  author = {Adiwardana, Daniel and Luong, Minh-Thang and So, David R. and Hall, Jamie and Fiedel, Noah and Thoppilan, Romal and Yang, Zi and Kulshreshtha, Apoorv and Nemade, Gaurav and Lu, Yifeng and Le, Quoc V.},
  year = {2020},
  doi = {10.48550/arxiv.2001.09977},
  eprint = {2001.09977},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2001.09977}
}

References

59 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR. find our card
  2. Arun Chaganty, Stephen Mussmann, and Percy Liang. 2018. The price of debiasing automatic metrics in natural language evalaution. In ACL. find
  3. Chun-Yen Chen, Dian Yu, Weiming Wen, Yi Mang Yang, Jiaping Zhang, Mingyang Zhou, Kevin Jesse, Austin Chau, Antara Bhowmick, Shreenath Iyer, Giritheja Sreenivasulu, Runxiang Cheng, Ashwin Bhandare, and Zhou Yu. 2018. Gunrock: Building a human-like social bot by leveraging large scale real user data.… find
  4. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical Neural Story Generation. arXiv e-prints, page arXiv:1805.04833. find
  5. Jianfeng Gao, Michel Galley, and Lihong Li. 2019a. Neural approaches to conversational AI. Foundations and Trends in Information Retrieval, 13(2-3):127–298. find
  6. Xiang Gao, Sungjin Lee, Yizhe Zhang, Chris Brockett, Michel Galley, Jianfeng Gao, and Bill Dolan. 2019b. Jointly optimizing diversity and relevance in neural response generation. In NAACL. find
  7. Asma Ghandeharioun, Judy Hanwen Shen, Natasha Jaques, Craig Ferguson, Noah Jones, Agata Lapedriza, and Rosalind Picard. 2019. Approximating interactive human evaluation with self-play for open-domain dialog systems. In Advances in Neural Information Processing Systems, pages 13658– 13669. find
  8. Sarik Ghazarian, Johnny Tian-Zheng Wei, Aram Galstyan, and Nanyun Peng. 2019. Better automatic evaluation of open-domain dialogue systems with contextualized embeddings. CoRR, abs/1904.10635. find
  9. Xiaodong Gu, Kyunghyun Cho, Jung-Woo Ha, and Sunghun Kim. 2019. DialogWAE: Multimodal response generation with conditional wasserstein auto-encoder. In ICLR. find
  10. Tatsunori B. Hashimoto, Hugh Zhang, and Percy Liang. 2019. Unifying human and statistical evaluation for natural language generation. In NAACL-HLT. find
  11. Jennifer Hill, W. Randolph Ford, and Ingrid G. Far-reras. 2015. Real conversations with artificial intelligence: A comparison between human-human online conversations and human-chatbot conversations. Computers in Human Behavior, 49:245–250. find
  12. Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. ArXiv, abs/1503.02531. find our card
  13. Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. ArXiv, abs/1904.09751. find
  14. Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. 2019a. Human and automatic detection of generated text. ArXiv, abs/1911.00650. find our card
  15. Daphne Ippolito, Reno Kriz, Joao Sedoc, Maria Kustikova, and Chris Callison-Burch. 2019b. Comparison of diverse decoding methods from conditional language models. In ACL. find
  16. Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for controllable generation. ArXiv, abs/1909.05858. find our card
  17. Wei-Jen Ko, Greg Durrett, and Junyi Jessy Li. 2019. Linguistically-informed specificity and semantic plausibility for dialogue generation. In NAACL. find
  18. Klaus Krippendorff. 2011. Computing krippendorff’s alpha-reliability. https://repository. upenn.edu/asc\_papers/43. find
  19. Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2011. The winograd schema challenge. In KR. find
  20. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A diversity-promoting objective function for neural conversation models. In NAACL-HLT. find our card
  21. Jiwei Li, Michel Galley, Chris Brockett, Georgios Spithourakis, Jianfeng Gao, and Bill Dolan. 2016b. A persona-based neural conversation model. In ACL. find our card
  22. Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. 2017. Adversarial learning for neural dialogue generation. In EMNLP. find
  23. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In ACL workshop on Text Summarization Branches Out. find
  24. Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In EMNLP. find
  25. Ryan Lowe, Michael Noseworthy, Iulian V. Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017. Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses. ACL. find our card
  26. Minh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. 2015. Addressing the rare word problem in neural machine translation. In ACL. find our card
  27. Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. Why We Need New Evaluation Metrics for NLG. EMNLP. find
  28. Kishore Papineni, Salim Roukos, Todd Ward, and Wei jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In ACL. find our card
  29. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9. find
  30. Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2018. I know the feeling: Learning to converse with empathy. CoRR, abs/1811.00207. find
  31. Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. 2018. Regularized evolution for image classifier architecture search. In AAAI. find
  32. Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Quoc V. Le, and Alex Kurakin. 2017. Large-scale evolution of image classifiers. In ICML. find
  33. Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019. What makes a good conversation? how controllable attributes affect human judgments. In NAACL. find our card
  34. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. ACL. find
  35. Iulian Vlad Serban, Chinnadhurai Sankar, Mathieu Germain, Saizheng Zhang, Zhouhan Lin, Sandeep Subramanian, Taesup Kim, Michael Pieper, Sarath Chandar, Nan Rosemary Ke, Sai Mudumba, Alexandre de Brébisson, Jose Sotelo, Dendi Suhubdy, Vincent Michalski, Alexandre Nguyen, Joelle Pineau, and Yoshua Be… — [arXiv:1709.02349](https://arxiv.org/abs/1709.02349) find
  36. Iulian Vlad Serban, Alessandro Sordoni, Yoshua Bengio, Aaron C. Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In AAAI. find our card
  37. Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. Neural responding machine for short-text conversation. In ACL. find our card
  38. Yuanlong Shao, Stephan Gouws, Denny Britz, Anna Goldie, Brian Strope, and Ray Kurzweil. 2017. Generating high-quality and informative conversation responses with sequence-to-sequence models. In EMNLP. find
  39. Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive Learning Rates with Sublinear Memory Cost. ICML. find
  40. David R. So, Chen Liang, and Quoc V. Le. 2019. The evolved transformer. In ICML. find
  41. Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In NAACL-HLT. find our card
  42. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In NeuRIPS. find
  43. Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan. 2017. RUBER: an unsupervised method for automatic evaluation of open-domain dialog systems. CoRR, abs/1701.03079. — [arXiv:1701.03079](https://arxiv.org/abs/1701.03079) find
  44. Alan M. Turing. 1950. Computing machinery and intelligence. Mind, 59(236):433–460. find
  45. Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018. Tensor2tensor for neural machine translation. CoRR, abs/1803.07416. find our card
  46. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeuRIPS. find
  47. Anu Venkatesh, Chandra Khatri, Ashwin Ram, Fenfei Guo, Raefer Gabriel, Ashish Nagar, Rohit Prasad, Ming Cheng, Behnam Hedayatnia, Angeliki Metallinou, Rahul Goel, Shaohua Yang, and Anirudh Raju. 2018. On evaluating and comparing conversational agents. CoRR, abs/1801.03625. — [arXiv:1801.03625](https://arxiv.org/abs/1801.03625) find
  48. Oriol Vinyals and Quoc V. Le. 2015. A neural conversational model. In ICML Deep Learning Workshop. find our card
  49. Di Wang, Nebojsa Jojic, Chris Brockett, and Eric Nyberg. 2017. Steering output style and topic in neural response generation. In EMNLP. find
  50. Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. 2019. Transfertransfo: A transfer learning approach for neural network based conversational agents. CoRR, abs/1901.08149. — [arXiv:1901.08149](https://arxiv.org/abs/1901.08149) find
  51. Steve Worswick. 2018. Mitsuku wins Loebner Prize 2018! https://medium.com/pandorabots-blog/mitsuku-wins-loebner-prize-2018-3e8d98c5f2a7. [Online; written on September 13, 2018]. ↗
  52. Chen Xing, Wei Wu, Yu Wu, Jie Liu, Yalou Huang, Ming Zhou, and Wei-Ying Ma. 2017. Topic aware neural response generation. In AAAI. find
  53. Zhen Xu, Nan Jiang, Bingquan Liu, Wenge Rong, Bowen Wu, Baoxun Wang, Zhuoran Wang, and Xiaolong Wang. 2018. LSDSCC: a large scale domain-specific conversational corpus for response generation with diversity oriented evaluation metrics. In NAACL. find
  54. Ruqing Zhang, Jiafeng Guo, Yixing Fan, Yanyan Lan, Jun Xu, and Xueqi Cheng. 2018a. Learning to control the specificity in neural response generation. In ACL. find
  55. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018b. Personalizing dialogue agents: I have a dog, do you have pets too? In ACL. find our card
  56. Yizhe Zhang, Michel Galley, Jianfeng Gao, Zhe Gan, Xiujun Li, Chris Brockett, and Bill Dolan. 2018c. Generating informative and diverse conversational responses via adversarial information maximization. In NeuRIPS. find
  57. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2019. Dialogpt: Large-scale generative pre-training for conversational response generation. CoRR, abs/1911.00536. find our card
  58. Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In ACL. find
  59. Li Zhou, Jianfeng Gao, Di Li, and Heung-Yeung Shum. 2018. The design and implementation of xiaoice, an empathetic social chatbot. CoRR, abs/1812.08989. find