Elamur Research
☰
home / papers / 2020 / USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation

USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.acl-main.64.pdf

Summary

Automatic evaluation of open-domain dialog is unreliable. Word-overlap scores such as BLEU, F-1 and METEOR compare output against a single reference, even though many different replies can be valid. They also reduce dialog quality to one opaque number. Mehri and Eskenazi propose USR, a metric that needs no reference response and no human quality labels for training. Instead, unsupervised models score individual qualities of a response: a masked language model sub-metric (USR-MLM) and dialog-retrieval sub-metrics (USR-DR) conditioned on either the context or the grounding fact/persona. A regression then merges these sub-scores into an overall score, and the weights can be adjusted for different tasks. To test it, the authors collected expert human ratings of system and human responses on Topical-Chat and PersonaChat, and they release these annotations. USR tracks human Overall Quality judgments much better than word-overlap and embedding-based baselines, at both turn and system level. The authors recommend it for model selection and tuning, used alongside human evaluation rather than in place of it.

Key points

  • Annotation set: 60 contexts per corpus were rated by six dialog researchers, with three annotators per response.
    • Topical-Chat: 360 responses (four Transformer decodings, a new human reply and the original ground truth).
    • PersonaChat: 300 responses (Seq2Seq, LM and KV-MemNN models from ParlAI/ConvAI2).
  • Rated qualities: Each response was scored on six qualities. Five covered whether it was easy to comprehend, sounded human-like, stayed consistent with the conversation so far, was engaging, and drew on the provided knowledge, and the sixth was Overall Quality. Inter-annotator agreement on Overall Quality was 0.71 (Topical-Chat) and 0.66 (PersonaChat).
  • Turn-level results (Overall Quality, Spearman):
    • Topical-Chat: USR reaches 0.42, ahead of METEOR (0.34) and BERTScore (0.32).
    • PersonaChat: USR's table value is 0.469 (rounded to 0.48 in the abstract). The single sub-metric USR-DR (x = c) slightly exceeds it at 0.481.
  • System-level results: USR achieves a Spearman correlation of 1.0 on both corpora. Its Pearson correlation is 0.92 on Topical-Chat and 0.82 on PersonaChat.
  • Stated limitations: USR may not suit non-generative models. It is also a concern when the evaluated model closely resembles USR's own sub-models, such as DR. The annotations are available at shikib.com/usr.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

136 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 126 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.64

@inproceedings{mehri2020usr,
  title = {{USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation}},
  author = {Mehri, Shikib and Eskenazi, Maxine},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.64},
  url = {https://doi.org/10.18653/v1/2020.acl-main.64}
}

References

31 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 376–380. find our card
  2. Jan Deriu, Alvaro Rodrigo, Arantxa Otegi, Guillermo Echegoyen, Sophie Rosset, Eneko Agirre, and Mark Cieliebak. 2019. Survey on evaluation methods for dialogue systems. arXiv preprint arXiv:1905.04071. — [arXiv:1905.04071](https://arxiv.org/abs/1905.04071) find
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
  4. Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, et al. 2019. The second conversational intelligence challenge (convai2). arXiv preprint arXiv:1902.00098. find our card
  5. Gabriel Forgues, Joelle Pineau, Jean-Marie Larchevêque, and Réal Tremblay. 2014. Bootstrapping dialog systems with word embeddings. In Nips, modern machine learning and natural language processing workshop, volume 2. find
  6. Michel Galley, Chris Brockett, Alessandro Sordoni, Yangfeng Ji, Michael Auli, Chris Quirk, Margaret Mitchell, Jianfeng Gao, and Bill Dolan. 2015. deltaBLEU: A discriminative metric for generation tasks with intrinsically diverse targets. In Proceedings of the 53rd Annual Meeting of the Association f… find
  7. Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, Dilek Hakkani-Tür, and Amazon Alexa AI. 2019. Topical-chat: Towards knowledge-grounded open-domain conversations. find
  8. Proc. Interspeech 2019, pages 1891–1895. find
  9. Prakhar Gupta, Shikib Mehri, Tiancheng Zhao, Amy Pavel, Maxine Eskenazi, and Jeffrey P Bigham. 2019. Investigating evaluation of open-domain dialogue systems with human generated multiple references. arXiv preprint arXiv:1907.10568. find
  10. Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751. find
  11. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in neural information processing systems, pages 3294–3302. find
  12. Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. 2017. Adversarial learning for neural dialogue generation. arXiv preprint arXiv:1701.06547. find
  13. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out. find
  14. Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. arXiv preprint arXiv:1603.08023. find
  15. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  16. Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017. Towards an automatic Turing test: Learning to evaluate dialogue responses. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P… find our card
  17. Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. arXiv preprint arXiv:1506.08909. find our card
  18. Shikib Mehri, Evgeniia Razumovsakaia, Tiancheng Zhao, and Maxine Eskenazi. 2019. Pretraining methods for dialog context representation learning. arXiv preprint arXiv:1906.00414. find
  19. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318. Association for Computational Linguistics. find our card
  20. Vasile Rus and Mihai Lintean. 2012. A comparison of greedy and optimal assessment of natural language student input using word-to-word similarity metrics. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, pages 157– 162. Association for Computational Linguistics. find
  21. Shikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zumer. 2017. Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation. arXiv preprint arXiv:1706.09799. — [arXiv:1706.09799](https://arxiv.org/abs/1706.09799) find
  22. Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and William B. Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In HLT-NAACL. find our card
  23. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 6000–6010. find
  24. Anu Venkatesh, Chandra Khatri, Ashwin Ram, Fenfei Guo, Raefer Gabriel, Ashish Nagar, Rohit Prasad, Ming Cheng, Behnam Hedayatnia, Angeliki Metallinou, et al. 2018. On evaluating and comparing open domain dialog systems. arXiv preprint arXiv:1801.03625. — [arXiv:1801.03625](https://arxiv.org/abs/1801.03625) find
  25. Marilyn A Walker, Diane J Litman, Candace A Kamm, and Alicia Abella. 1997. Paradise: A framework for evaluating spoken dialogue agents. arXiv preprint cmp-lg/9704004. find
  26. John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2015. Towards universal paraphrastic sentence embeddings. arXiv preprint arXiv:1511.08198. find our card
  27. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtow-icz, and Jamie Brew. 2019a. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. find our card
  28. Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. 2019b. Transfertransfo: A transfer learning approach for neural network based conversational agents. arXiv preprint arXiv:1901.08149. — [arXiv:1901.08149](https://arxiv.org/abs/1901.08149) find
  29. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? arXiv preprint arXiv:1801.07243. find our card
  30. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. find
  31. Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. arXiv preprint arXiv:1703.10960. find