Elamur Research⌕
☰
home / papers / 2019 / Sentence Mover's Similarity: Automatic Evaluation for Multi-Sentence Texts

Sentence Mover's Similarity: Automatic Evaluation for Multi-Sentence Texts

Paper · 2019 · Elizabeth Clark, Asli Celikyilmaz and Noah A. Smith

Summary

Popular metrics such as ROUGE and BLEU score generated text by exact word overlap. As a result, they miss paraphrased or reordered content and correlate weakly with human judgments. The embedding-based word mover's distance (WMD) is more flexible, but it treats a document as an unordered bag of words and ignores how ideas group into sentences. Clark, Celikyilmaz and Smith extend this optimal-transport approach to longer texts. Their metric SMS represents each document as a set of sentence vectors, built by averaging word embeddings and weighted by sentence length. A second metric, S+WMS, puts word and sentence vectors into one shared transport problem. The authors compared the metrics with human ratings of machine-generated summaries (about 3.4 sentences) and human-written essays (about 7.5 sentences). Both new metrics agree with people significantly better than ROUGE-L and WMS, with either type-based or contextual embeddings. Used as a reinforcement learning reward for summarization, they produce summaries that human judges prefer. The work offers a cheap, meaning-aware way to evaluate and train multi-sentence generation.

Key points

  • Two metrics are introduced. SMS uses only sentence embeddings. S+WMS mixes word embeddings (weighted by frequency) and sentence embeddings (weighted by length), with weights normalized by 2|A|. WMD is turned into a similarity via exp(−distance), as in WMS.
  • Gains over ROUGE-L were larger on generated summaries (avg. 3.4 sentences) than on human essays (avg. 7.5 sentences), but significant in both cases. The choice between type-based and contextual embeddings made no significant difference.
  • The RL summarizers used a mixed MLE+RL loss. Human judges compared 100 summaries, with 3 judges per summary, on four criteria: non-redundancy, coherence, focus and overall quality. Models rewarded with WMS, SMS or S+WMS beat ROUGE-L at p < 0.001.
  • SMS achieved the strongest non-redundancy preference over ROUGE-L (69%). However, without word-level matching it can introduce factual slips in its summaries.
  • Code and data are released at github.com/eaclark07/sms, which builds on the wmd-relax implementation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States240 works · 123,537 citations🎓University of WashingtonSeattle · United States263 works · 180,581 citations🔬Allen Institute for Artificial IntelligenceSeattle · United States219 works · 89,712 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

Cite

Cite as Elizabeth Clark, Asli Celikyilmaz and Noah A. Smith. 2019. Sentence Mover's Similarity: Automatic Evaluation for Multi-Sentence Texts. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1264

@inproceedings{clark2019sentence,
  title = {{Sentence Mover's Similarity: Automatic Evaluation for Multi-Sentence Texts}},
  author = {Elizabeth Clark, Asli Celikyilmaz and Noah A. Smith},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1264},
  url = {https://doi.org/10.18653/v1/p19-1264}
}

References

58 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohammed Alshahrani, Spyridon Samothrakis, and Maria Fasli. 2017. Word mover’s distance for affect detection. 2017 International Conference on the Frontiers and Advances in Data Science. find
  2. Kubilay Atasu, Thomas P. Parnell, Celestine Dünner, Manolis Sifalakis, Haralampos Pozidis, Vasileios Vasileiadis, Michail Vlachos, Cesar Berrospi, and Abdel Labbi. 2017. Linear-complexity relaxed word mover’s distance with GPU acceleration. In IEEE International Conference on Big Data. find
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR.
  4. Georgios Balikas, Charlotte Laclau, Ievgen Redko, and Massih-Reza Amini. 2018. Cross-lingual document retrieval using regularized Wasserstein distance. CoRR, abs/1805.04437. — [arXiv:1805.04437](https://arxiv.org/abs/1805.04437) find
  5. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In IEE-valuation@ACL.
  6. Regina Barzilay and Mirala Lapata. 2008. Modeling local coherence: An entity-based approach. In Computational Linguistics.
  7. Anja Belz and Ehud Reiter. 2006. Comparing automatic and human evaluation of NLG systems. In EACL. find
  8. George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Second International Conference on Human Language Technology Research. find
  9. Ondrej Dusek, Jekaterina Novikova, and Verena Rieser. 2019. Evaluating the state-of-the-art of end-to-end natural language generation: The E2E NLG challenge. In Computational Linguistics. find
  10. Sergey Edunov, Myle Ott, Michael Auli, David Grangier, and Marc’Aurelio Ranzato. 2018. Classical structured prediction losses for sequence to sequence learning. In NAACL-HLT.
  11. Desmond Elliott and Frank Keller. 2014. Comparing automatic evaluation measures for image description. In ACL. find
  12. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. In ACL workshop for NLP Open Source Software.
  13. Yoav Goldberg, Graeme Hirst, Yang Liu, and Meng Zhang. 2018. Neural network methods for natural language processing. Computational Linguistics, 44(1). find
  14. Tatsunori B. Hashimoto, Hugh Zhang, and Percy Liang. 2019. Unifying human and statistical evaluation for natural language generation. In NAACL. find
  15. Asli Celikyilmaz, Antoine Bosselut, Xiadong He, and Yejin Choi. 2018. Deep communicating agents for abstractive summarization. In NAACL.
  16. Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In NeurIPS.
  17. Arun Tejasvi Chaganty, Stephen Mussmann, and Percy Liang. 2018. The price of debiasing automatic metrics in natural language evaluation. In ACL. find
  18. Gao Huang, Chuan Guo, Matt J. Kusner, Yu Sun, Fei Sha, and Kilian Q. Weinberger. 2016. Supervised word mover’s distance. In NeurIPS. find
  19. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, and Phillipp Koehn. 2013. One billion word benchmark for measuring progress in statistical language modeling. In INTERSPEECH.
  20. Natasha Jaques, Shixiang Gu, Dxmitry Bahdanau, Jose Miguel Hernandez-Lobato, Richard E. Turner, and Douglas Eck. 2017. Counterfactual multi-agent policy gradients. In ICML. find
  21. Liqun Chen, Yizhe Zhang, Ruiyi Zhang, Chenyang Tao, Zhe Gan, Haichao Zhang, Bai Li, Dinghan Shen, Changyou Chen, and Lawrence Carin. 2019. Improving sequence-to-sequence learning via optimal transport. In ICLR. find
  22. Mert Kilickaya, Aykut Erdem, Nazli Ikizler-Cinbis, and Erkut Erdem. 2017. Re-evaluating automatic metrics for image captioning. In EACL. find
  23. Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by extracting sentences and words. In ACL.
  24. Alexis Conneau and Douwe Kiela. 2018. SentEval: An evaluation toolkit for universal sentence representations. In LREC.
  25. Andrew M. Dai, Christopher Olah, and Quoc V. Le. 2015. Document embedding with paragraph vectors. In NeurIPS Deep Learning Workshop. find
  26. Matt J. Kusner, Yu Sun, Nicholas I. Kolkin, and Kilian Q. Weinberger. 2015. From word embeddings to document distances. ICLR, 37. find
  27. Mirella Lapata and Regina Barzilay. 2005. Automatic evaluation of text coherence: Models and representations. In IJCAI. find
  28. Alon Lavie and Abhaya Agarwal. 2007. METEOR: An automatic metric for MT evaluation with high levels of with human judgments. In Second Workshop on Statistical Machine Translation. find
  29. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In ACL.
  30. Chia-Wei Liu, Ryan Lowe, Iulian Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In EMNLP.
  31. Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In LREC.
  32. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. SummaRuNNer: A recurrent neural network based sequence model for extractive summarization of documents. In AAAI.
  33. Ramesh Nallapati, Bowen Zhou, Cı́cero Nogueira dos Santos, Caglar Gülehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In CoNLL. find
  34. Jekaterina Novikova, Ondrej Dusek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for NLG. In EMNLP.
  35. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL.
  36. Ramakanth Pasunuru and Mohit Bansal. 2017. Reinforced video captioning with entailment rewards. In EMNLP. find
  37. Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In ICLR.
  38. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP.
  39. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL.
  40. Christopher Purdy, Xinyu Wang, Larry He, and Mark O. Riedl. 2018. Predicting generated story quality with quantitative measures. In AIIDE. find
  41. MarcAurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2015. Sequence level training with recurrent neural networks. In ICLR.
  42. Ehud Reiter. 2011. Task-based evaluation of NLG systems: Control vs real-world context. In UC-NLG+Eval. find
  43. Ehud Reiter and Anja Belz. 2006. GENEVAL: A proposal for shared-task evaluation in NLG. In INLG. find
  44. Ehud Reiter and Anja Belz. 2009. An investigation into the validity of some metrics for automatically evaluating natural language generation systems. In CVPR. find
  45. Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel. 2016. Self-critical sequence training for image captioning. In CVPR.
  46. Tomasi C. Rubner, Y. and L. J. Guibas. 1998. A metric for distributions with applications to image databases. In IEEE. find
  47. Manasvi Sagarkar, John Wieting, Lifu Tu, and Kevin Gimpel. 2018. Quality signals in generated stories. In *SEM 2018. find
  48. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In ACL. find
  49. Octavia-Maria Sulea. 2017. Recognizing textual entailment in Twitter using word embeddings. In 2nd Workshop on Evaluating Vector-Space Representations for NLP.
  50. Tsegaye Misikir Tashu and Tomás Horváth. 2018. Pair-Wise: Automatic essay evaluation using word mover’s distance. In CSEDU. find
  51. Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. CIDEr: Consensus-based image description evaluation. In CVPR.
  52. Evan J. Williams. 1959. ume 14. Wiley. find
  53. Regression Analysis, vol-Ronald J. Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. In Machine Learning. find
  54. Lingfei Wu, Ian En-Hsu Yen, Kun Xu, Fangli Xu, Avinash Balakrishnan, Pin-Yu Chen, Pradeep Ravikumar, and Michael J. Witbrock. 2018. Word movers embedding: From word2vec to document embedding. In EMNLP. find
  55. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. ArXiv:1609.08144. find
  56. Hao Zhang and Jie Wang. 2018. Semantic WordRank: Generating finer single-document summarizations. In IDEAL. find
  57. Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. 2017. Earth mover’s distance minimization for unsupervised bilingual lexicon induction. In EMNLP. find
  58. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. BERTScore: Evaluating text generation with BERT. CoRR, abs/1904.09675. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/clark-2019-sentence-mover-s-similarity-automatic/