Elamur Research
☰
home / papers / 2017 / Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses

Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1708.07149

Summary

Open-domain chatbots are still commonly scored with word-overlap metrics such as BLEU, which often disagree with human ratings. A response can be perfectly sensible yet share no words with the single reference reply, and these metrics also ignore the conversation context. The authors treat evaluation as a supervised learning task. They gathered appropriateness ratings from Amazon Mechanical Turk workers for Twitter Corpus replies produced by several systems, then trained ADEM, a hierarchical RNN (LSTM) evaluator trained semi-supervised. ADEM encodes the context, the reference reply and the candidate reply, and combines them through learned bilinear terms to output a human-like score. ADEM's predictions track human judgements significantly, and far better than BLEU and similar metrics, both for individual responses and when ranking whole systems. It also stays useful for dialogue models whose outputs were held out from training. This makes it a cheap first step toward an "automatic Turing test" that reduces the need for repeated, costly human evaluation during model development.

Key points

  • Dataset size: 4,104 rated (context, model response, reference response, score) examples covering 1,026 Twitter contexts. The split is 2,872 / 616 / 616 for train, validation and test, with inter-annotator κ of 0.63 and one rating per response.
  • Candidate sources: responses came from a TF-IDF retriever, the Dual Encoder (DE), the generative HRED model, and freshly written human replies that differ from the corpus reference.
  • Scoring function: score = (cᵀMr̂ + rᵀNr̂ − α)/β. M and N are learned matrices initialised to identity, and α and β map outputs into the 1–5 range.
  • Rating criterion: ratings target appropriateness only. Other criteria either showed low inter-annotator agreement or correlated strongly with appropriateness.
  • Known limitation: ADEM inherits the human tendency to reward generic responses. Domain transfer, for example to Reddit, is left for future work.
  • Availability: code and pretrained weights are released at github.com/mike-n-7/ADEM.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

150 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 143 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1708.07149 · arXiv 1708.07149

@misc{lowe2017automatic,
  title = {{Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses}},
  author = {Lowe, Ryan and Noseworthy, Michael and Serban, Iulian V. and Angelard-Gontier, Nicolas and Bengio, Yoshua and Pineau, Joelle},
  year = {2017},
  doi = {10.48550/arxiv.1708.07149},
  eprint = {1708.07149},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1708.07149}
}

References

53 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Joshua Albrecht and Rebecca Hwa. 2007. Regression for sentence-level mt evaluation with pseudo references. In ACL. find
  2. Ron Artstein, Sudeep Gandhe, Jillian Gerten, Anton Leuski, and David Traum. 2009. Semi-formal evaluation of conversational characters. In Languages: From Formal to Natural, Springer, pages 22–35. find
  3. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 . find our card
  4. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. volume 29, pages 65–72. find our card
  5. Yoshua Bengio, Patrice Simard, and Paolo Frasconi. 1994. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks 5(2):157–166. find
  6. Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. COLING . find our card
  7. Chris Callison-Burch, Philipp Koehn, Christof Monz, and Omar F Zaidan. 2011. Findings of the 2011 workshop on statistical machine translation. In Proceedings of the Sixth Workshop on Statistical Machine Translation. Association for Computational Linguistics, pages 22–64. find
  8. Boxing Chen and Colin Cherry. 2014. A systematic comparison of smoothing techniques for sentence-level bleu. find our card
  9. Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin 70(4):213. find
  10. Tim Cooijmans, Nicolas Ballas, César Laurent, and Aaron Courville. 2016. Recurrent batch normalization. arXiv preprint arXiv:1603.09025 . — [arXiv:1603.09025](https://arxiv.org/abs/1603.09025) find
  11. David DeVault, Anton Leuski, and Kenji Sagae. 2011. Toward learning and evaluation of dialogue policies with text examples. In Proceedings of the SIG-DIAL 2011 Conference. Association for Computational Linguistics, pages 39–48. find
  12. Bhuwan Dhingra, Zhong Zhou, Dylan Fitzpatrick, Michael Muehl, and William W Cohen. 2016. Tweet2vec: Character-based distributed representations for social media. arXiv preprint arXiv:1605.03481 . find
  13. Harrison Edwards and Amos Storkey. 2016. Censoring representations with an adversary. ICLR . find
  14. Salah El Hihi and Yoshua Bengio. 1995. Hierarchical recurrent neural networks for long-term dependencies. In NIPS. Citeseer, volume 400, page 409. find
  15. Philip Gage. 1994. A new algorithm for data compression. The C Users Journal 12(2):23–38. find
  16. Michel Galley, Chris Brockett, Alessandro Sordoni, Yangfeng Ji, Michael Auli, Chris Quirk, Margaret Mitchell, Jianfeng Gao, and Bill Dolan. 2015. deltableu: A discriminative metric for generation tasks with intrinsically diverse targets. arXiv preprint arXiv:1506.06863 . find
  17. Sudeep Gandhe and David Traum. 2016. A semi-automated evaluation metric for dialogue model coherence. In Situated Dialog in Speech-Based Human-Computer Interaction, Springer, pages 217– 225. find
  18. Rohit Gupta, Constantin Orasan, and Josef van Genabith. 2015. Reval: A simple and effective machine translation evaluation metric based on recurrent neural networks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  19. Ryuichiro Higashinaka, Toyomi Meguro, Kenji Imamura, Hiroaki Sugiyama, Toshiro Makino, and Yoshihiro Matsuo. 2014. Evaluating coherence in open domain conversational systems. In INTER-SPEECH. pages 130–134. find
  20. Sepp Hochreiter. 1991. Untersuchungen zu dynamischen neuronalen netzen. Diploma, Technische Universität München page 91. find
  21. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9(8):1735– 1780. find
  22. Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 . find
  23. Anjuli Kannan, Karol Kurach, Sujith Ravi, Tobias Kaufmann, Andrew Tomkins, Balint Miklos, Greg Corrado, László Lukács, Marina Ganea, Peter Young, et al. 2016. Smart reply: Automated response suggestion for email. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (… find
  24. Anjuli Kannan and Oriol Vinyals. 2017. Adversarial evaluation of dialogue models. arXiv preprint arXiv:1701.08198 . — [arXiv:1701.08198](https://arxiv.org/abs/1701.08198) find
  25. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 . find
  26. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in Neural Information Processing Systems. pages 3276–3284. find
  27. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2015. A diversity-promoting objective function for neural conversation models. arXiv preprint arXiv:1510.03055 . find our card
  28. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A persona-based neural conversation model. arXiv preprint arXiv:1603.06155 . find our card
  29. Jiwei Li, Will Monroe, and Dan Jurafsky. 2017. Learning to decode for future success. arXiv preprint arXiv:1701.06549 . — [arXiv:1701.06549](https://arxiv.org/abs/1701.06549) find
  30. Jiwei Li, Will Monroe, Alan Ritter, and Dan Jurafsky. 2016b. Deep reinforcement learning for dialogue generation. arXiv preprint arXiv:1606.01541 . find our card
  31. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out: Proceedings of the ACL-04 workshop. Barcelona, Spain, volume 8. find
  32. Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. arXiv preprint arXiv:1603.08023 . find
  33. Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. arXiv preprint arXiv:1506.08909 . find our card
  34. Matouš Machácek and Ondrej Bojar. 2014. Results of the wmt14 metrics shared task. In Proceedings of the Ninth Workshop on Statistical Machine Translation. Citeseer, pages 293–301. find our card
  35. J. Markoff and P. Mozur. 2015. For sympathetic ear, more chinese turn to smartphone program. NY Times . find
  36. Sebastian Möller, Roman Englert, Klaus-Peter Engelbrecht, Verena Vanessa Hafner, Anthony Jameson, Antti Oulasvirta, Alexander Raake, and Norbert Reithinger. 2006. Memo: towards automatic usability evaluation of spoken dialogue services by user error simulations. In INTERSPEECH. find
  37. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics. Association for Computational Linguistics, pages 311–318. find our card
  38. Karl Pearson. 1901. Principal components analysis. The London, Edinburgh and Dublin Philosophical Magazine and Journal 6(2):566. find
  39. Alan Ritter, Colin Cherry, and William B Dolan. 2011. Data-driven response generation in social media. In Proceedings of the conference on empirical methods in natural language processing. Association for Computational Linguistics, pages 583–593. find
  40. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909 . find
  41. Iulian Vlad Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016a. Building end-to-end dialogue systems using generative hierarchical neural network models. In AAAI. pages 3776–3784. find our card
  42. Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2016b. A hierarchical latent variable encoder-decoder model for generating dialogues. arXiv preprint arXiv:1605.06069 . find our card
  43. Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. Neural responding machine for short-text conversation. arXiv preprint arXiv:1503.02364 . find our card
  44. Lifeng Shang, Tetsuya Sakai, Zhengdong Lu, Hang Li, Ryuichiro Higashinaka, and Yusuke Miyao. 2016. Overview of the ntcir-12 short text conversation task. Proceedings of NTCIR-12 pages 473–484. find
  45. Alessandro Sordoni, Yoshua Bengio, Hossein Vahabi, Christina Lioma, Jakob Grue Simonsen, and Jian-Yun Nie. 2015a. A hierarchical recurrent encoder-decoder for generative context-aware query suggestion. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management… find
  46. Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015b. A neural network approach to context-sensitive generation of conversational responses. arXiv preprint arXiv:1506.06714 . find our card
  47. Miloš Stanojevic, Amir Kamran, Philipp Koehn, and Ondrej Bojar. 2015. Results of the wmt15 metrics shared task. In Proceedings of the Tenth Workshop on Statistical Machine Translation. pages 256–273. find
  48. Alan M Turing. 1950. Computing machinery and intelligence. Mind 59(236):433–460. find
  49. Oriol Vinyals and Quoc Le. 2015. A neural conversational model. arXiv preprint arXiv:1506.05869 . find our card
  50. Marilyn A Walker, Diane J Litman, Candace A Kamm, and Alicia Abella. 1997. Paradise: A framework for evaluating spoken dialogue agents. In Proceedings of the eighth conference on European chapter of the Association for Computational Linguistics. Association for Computational Linguistics, pages 271–2… find
  51. J. Weizenbaum. 1966. ELIZAa computer program for the study of natural language communication between man and machine. Communications of the ACM 9(1):36–45. find
  52. Yang Xiang, Yaoyun Zhang, Xiaoqiang Zhou, Xiaolong Wang, and Yang Qin. 2014. Problematic situation analysis and automatic recognition for chi-nese online conversational system. Proc. CLP pages 43– 51. find
  53. Zhou Yu, Ziyu Xu, Alan W Black, and Alex I Rudnicky. 2016. Strategy and policy learning for non-task-oriented conversational systems. In 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue. page 404. find