Elamur Research
☰
home / papers / 2020 / Unsupervised Evaluation of Interactive Dialog with DialoGPT

Unsupervised Evaluation of Interactive Dialog with DialoGPT

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.sigdial-1.28.pdf

Summary

Automatically judging open-domain chatbots is difficult. Word-overlap metrics such as BLEU penalize valid replies that differ from a reference. Dialog quality has many dimensions. Interactive conversations with real users also have no ground-truth reply to compare against. Mehri and Eskenazi address these issues with the FED metric, which needs no references, no training data and no supervision. FED uses the off-the-shelf DialoGPT model as a stand-in for the user. It scores a system turn by how likely DialoGPT would be to answer it with hand-written follow-ups tied to a particular quality, such as an expression of interest or of confusion. To test the metric, the authors built the FED dataset by crowd-annotating conversations released with Meena, covering Meena, Mitsuku and human partners. Annotators rated eighteen fine-grained qualities at the turn and dialog levels. FED correlates moderately to strongly with human ratings, without any fine-tuning. This suggests that large-scale pre-training gives DialoGPT an implicit sense of what makes a dialog good. The approach is cheap to apply across many qualities, but the authors stress that it complements human evaluation rather than replacing it.

Key points

  • The FED dataset contains 124 conversations (40 Human–Meena, 44 Human–Mitsuku, 40 Human–Human). Each was rated by five Amazon Mechanical Turk workers, giving 4712 data points: 3348 turn-level and 1364 dialog-level.
  • Annotations cover 8 turn-level qualities (e.g., interesting, relevant, fluent) and 10 dialog-level qualities (e.g., coherent, consistent, inquisitive), each with an overall-impression score. The dataset is meant for evaluating metrics only, not for training.
  • The metric was tested with DialoGPT variants of 345M and 762M parameters, and agreement with humans was measured using Spearman correlation.
  • FED works best for qualities that vary widely in Reddit data, such as interesting and engaging. It is weaker on consistency and understandability, which the authors attribute to DialoGPT's Reddit pre-training.
  • Stated limitations: the metric has only been tested on chit-chat, not goal-oriented dialog, and models could game it by steering toward specific follow-up replies.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

112 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 110 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.sigdial-1.28

@inproceedings{mehri2020unsupervised,
  title = {{Unsupervised Evaluation of Interactive Dialog with DialoGPT}},
  author = {Mehri, Shikib and Eskenazi, Maxine},
  booktitle = {Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue},
  year = {2020},
  doi = {10.18653/v1/2020.sigdial-1.28},
  url = {https://doi.org/10.18653/v1/2020.sigdial-1.28}
}

References

27 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977. find our card
  2. Jan Deriu, Alvaro Rodrigo, Arantxa Otegi, Guillermo Echegoyen, Sophie Rosset, Eneko Agirre, and Mark Cieliebak. 2019. Survey on evaluation methods for dialogue systems. arXiv preprint arXiv:1905.04071. — [arXiv:1905.04071](https://arxiv.org/abs/1905.04071) find
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
  4. Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, et al. 2019. The second conversational intelligence challenge (convai2). arXiv preprint arXiv:1902.00098. find our card
  5. Maxine Eskenazi, Shikib Mehri, Evgeniia Razumovskaia, and Tiancheng Zhao. 2019. Beyond turing: Intelligent agents centered on the user. arXiv preprint arXiv:1901.06613. — [arXiv:1901.06613](https://arxiv.org/abs/1901.06613) find
  6. Michel Galley, Chris Brockett, Alessandro Sordoni, Yangfeng Ji, Michael Auli, Chris Quirk, Margaret Mitchell, Jianfeng Gao, and Bill Dolan. 2015. deltaBLEU: A discriminative metric for generation tasks with intrinsically diverse targets. In Proceedings of the 53rd Annual Meeting of the Association f… find
  7. Asma Ghandeharioun, Judy Hanwen Shen, Natasha Jaques, Craig Ferguson, Noah Jones, Agata Lapedriza, and Rosalind Picard. 2019. Approximating interactive human evaluation with self-play for open-domain dialog systems. In Advances in Neural Information Processing Systems, pages 13658– 13669. find
  8. Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, Dilek Hakkani-Tür, and Amazon Alexa AI. 2019. Topical-chat: Towards knowledge-grounded open-domain conversations. find
  9. Proc. Interspeech 2019, pages 1891–1895. find
  10. Fenfei Guo, Angeliki Metallinou, Chandra Khatri, Anirudh Raju, Anu Venkatesh, and Ashwin Ram. 2018. Topic-based evaluation for conversational bots. arXiv preprint arXiv:1801.03622. — [arXiv:1801.03622](https://arxiv.org/abs/1801.03622) find
  11. Prakhar Gupta, Shikib Mehri, Tiancheng Zhao, Amy Pavel, Maxine Eskenazi, and Jeffrey P Bigham. 2019. Investigating evaluation of open-domain dialogue systems with human generated multiple references. arXiv preprint arXiv:1907.10568. find
  12. Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu, Yordan Yordanov, and Thomas Lukasiewicz. 2019. A surprisingly robust trick for winograd schema challenge. arXiv preprint arXiv:1905.06290. — [arXiv:1905.06290](https://arxiv.org/abs/1905.06290) find
  13. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. find
  14. Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. arXiv preprint arXiv:1603.08023. find
  15. Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017. Towards an automatic Turing test: Learning to evaluate dialogue responses. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P… find our card
  16. Shikib Mehri and Maxine Eskenazi. 2020. Usr: An unsupervised and reference free evaluation metric for dialog generation. arXiv preprint arXiv:2005.00456. find our card
  17. Libo Qin, Yijia Liu, Wanxiang Che, Haoyang Wen, Yangming Li, and Ting Liu. 2019. Entity-consistent end-to-end task-oriented dialogue system with kb retriever. arXiv preprint arXiv:1909.06762. find
  18. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assets/research-covers/languageunsupervised/language understanding paper. pdf. find
  19. Ashwin Ram, Rohit Prasad, Chandra Khatri, Anu Venkatesh, Raefer Gabriel, Qing Liu, Jeff Nunn, Behnam Hedayatnia, Ming Cheng, Ashish Nagar, et al. 2018. Conversational ai: The science behind the alexa prize. arXiv preprint arXiv:1801.03604. — [arXiv:1801.03604](https://arxiv.org/abs/1801.03604) find
  20. Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019. What makes a good conversation? how controllable attributes affect human judgments. arXiv preprint arXiv:1902.08654. find our card
  21. Jamin Shin, Peng Xu, Andrea Madotto, and Pascale Fung. 2019. Happybot: Generating empathetic dialogue responses by improving user experience lookahead. arXiv preprint arXiv:1906.08487. — [arXiv:1906.08487](https://arxiv.org/abs/1906.08487) find
  22. Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and William B. Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In HLT-NAACL. find our card
  23. Anu Venkatesh, Chandra Khatri, Ashwin Ram, Fenfei Guo, Raefer Gabriel, Ashish Nagar, Rohit Prasad, Ming Cheng, Behnam Hedayatnia, Angeliki Metallinou, et al. 2018. On evaluating and comparing open domain dialog systems. arXiv preprint arXiv:1801.03625. — [arXiv:1801.03625](https://arxiv.org/abs/1801.03625) find
  24. Marilyn A Walker, Diane J Litman, Candace A Kamm, and Alicia Abella. 1997. Paradise: A framework for evaluating spoken dialogue agents. arXiv preprint cmp-lg/9704004. find
  25. Sanghyun Yi, Rahul Goel, Chandra Khatri, Tagyoung Chung, Behnam Hedayatnia, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tur. 2019. Towards coherent and engaging spoken dialog response generation using automatic conversation evaluators. arXiv preprint arXiv:1904.13015. find
  26. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2019. Dialogpt: Large-scale generative pre-training for conversational response generation. arXiv preprint arXiv:1911.00536. find our card
  27. Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. arXiv preprint arXiv:1703.10960. find