Elamur Research⌕
☰
home / papers / 2021 / A Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods

A Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods

Paper · 2021 · Daniel Deutsch, Rotem Dror and Dan Roth

Summary

Automatic summarization metrics are usually judged by how well their scores correlate with human quality ratings. However, the field has no standard way to measure how reliable those correlations are, or to tell whether one metric truly beats another. Deutsch, Dror, and Roth address this with resampling statistics. Correlations are computed over a systems × inputs score matrix, so they adapt bootstrapping and permutation testing to resample matrices instead of vectors. Each variant treats systems, inputs, or both as random.

Two simulation experiments compare the variants. Confidence intervals generalize best to held-out data when both systems and inputs are resampled. The matching permutation test is more powerful than its bootstrap counterpart and than Williams' test, the parametric test common in machine translation.

The authors then apply these tools to many metrics on three sets of English human judgments. They find that the confidence intervals are wide, so our knowledge of metric quality is quite uncertain. Few metrics reliably outperform ROUGE. QAEval and BERTScore are notable exceptions in some settings. The approach also carries over to evaluating metrics for other generation tasks.

Key points

  • Three matrix-resampling schemes are proposed: BOOT-SYSTEMS, BOOT-INPUTS, and BOOT-BOTH, along with analogous permutation tests. They are applied to both system-level and summary-level correlations (Pearson, Spearman, Kendall).
  • The parametric options, the Fisher transformation and Williams' test, assume normally distributed data. That assumption fails for summarization scores, which motivates the nonparametric approach.
  • The human judgments are drawn from three sources: Bhandari et al. (2020), Fabbri et al. (2021), and the TAC 2008 data described by Dang and Owczarzak (2008). The datasets cover single- and multi-document settings, including CNN/DM.
  • Recommended setup: BOOT-BOTH for confidence intervals and the corresponding both-variable permutation test for comparing metrics. Williams' test is not advised for summarization metrics because of its low power.
  • Published in TACL vol. 9 (2021), pp. 1132–1146. Code is available at github.com/CogComp/stat-analysis-experiments.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of PennsylvaniaPhiladelphia · United States170 works · 60,739 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 2 Evaluation & benchmarks · 685 works Summarization · 159 works

Cite

Cite as Daniel Deutsch, Rotem Dror and Dan Roth. 2021. A Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00417

@article{deutsch2021statistical,
  title = {{A Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods}},
  author = {Daniel Deutsch, Rotem Dror and Dan Roth},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.1162/tacl\_a\_00417},
  url = {https://doi.org/10.1162/tacl_a_00417}
}

References

43 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Erfan Sadeqi Azer, Daniel Khashabi, Ashish Sabharwal, and Dan Roth. 2020. Not all claims are created equal: Choosing the right statistical approach to assess hypotheses. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5715–5725, Online. Association f… find
  2. Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. 2020. Re-evaluating evaluation in text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9347–9359, Online. Association for Computational Linguist… our card
  3. D. Bonett and T. A. Wright. 2000. Sample size requirements for estimating Pearson, Kendall and Spearman correlations. Psychometrika, 65:23–28. https://doi.org/10.1007/BF02294183 ↗
  4. Carlo E. Bonferroni. 1936. Teoria Statistica Delle Classi e Calcolo Delle Probabilita, Libreria internazionale Seeber. find
  5. Hoa Trang Dang and Karolina Owczarzak. 2008. Overview of the TAC 2008 Update Summarization Task. In Proc. of the Text Analysis Conference (TAC). find
  6. H. T. Dang and K. Owczarzak. 2009. Overview of the TAC 2009 Summarization track. In Text Analysis Conference (TAC). find
  7. Sashka T. Davis, John M. Conroy, and Judith D. Schlesinger. 2012. OCCAMS–An optimal combinatorial covering algorithm for multi-document summarization. In 2012 IEEE 12th International Conference on Data Mining Workshops, pages 454–463. IEEE. find
  8. Michael J. Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the Ninth Workshop on Statistical Machine Translation, WMT@ACL 2014, June 26–27, 2014, Baltimore, Maryland, USA, pages 376–380. The Association for Compute… our card
  9. Daniel Deutsch, Tania Bedrax-Weiss, and Dan Roth. 2021. Towards question-answering as an automatic metric for evaluating the content quality of a summary. Transactions of the Association for Computational Linguistics, 9. our card
  10. Daniel Deutsch and Dan Roth. 2020. SacreROUGE: An open-source library for using and developing summarization evaluation metrics. In Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS), pages 120–125, Online. Association for Computational Linguistics. https://doi.org /10.18653/v1/20… ↗
  11. Rotem Dror, Gili Baumer, Marina Bogomolov, and Roi Reichart. 2017. Replicability analysis for natural language processing: Testing significance with multiple datasets. Transactions of the Association for Computational Linguistics, 5:471–486. https://doi.org/10.1162/tacl_a_00074 ↗
  12. Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker’s guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383–1392. https://doi.… ↗
  13. Rotem Dror, Lotem Peled-Cohen, Segev Shlomov, and Roi Reichart. 2020. Statistical significance testing for natural language processing. Synthesis Lectures on Human Language Technologies, 13(2):1–116. https://doi.org/10.2200/S00994ED1V01Y202002HLT045 ↗
  14. Olive Jean Dunn and Virginia Clark. 1971. Comparison of tests of the equality of dependent correlation coefficients. Journal of the American Statistical Association, 66(336):904–908. find
  15. Bradley Efron and Robert Tibshirani. 1993. An Introduction to the Bootstrap. Springer. https://doi.org/10.1080/01621459 .1971.10482369 ↗
  16. A. R. Fabbri, Wojciech Kryscinski, Bryan McCann, R. Socher, and Dragomir Radev. 2021. SummEval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409. https://doi.org/10.1162/tacl a 00373 ↗
  17. Ronald Aylmer Fisher. 1992. Statistical methods for research workers, Breakthroughs in Statistics, Springer, pages 66–70. https://doi.org/10.1007/978-1-4612-4380-9 6 — [doi:10.1007/978-1-4612-4380-9](https://doi.org/10.1007/978-1-4612-4380-9) ↗
  18. George Giannakopoulos and Vangelis Karkaletsis. 2010. Summarization system evaluation variations based on n-gram graphs. In Proceedings of the Third Text Analysis Conference, TAC 2010, Gaithersburg, Maryland, USA, November 15–16, 2010. NIST. find
  19. George Giannakopoulos and Vangelis Karkaletsis. 2013. Summary evaluation: Together we stand NPowER-ed. In Computational Linguistics and Intelligent Text Processing - 14th International Conference, CICLing 2013, Samos, Greece, March 24-30, 2013, Proceedings, Part II, volume 7817 of Lecture Notes in C… — [doi:10.1007/978-3-642](https://doi.org/10.1007/978-3-642) find
  20. George Giannakopoulos, Vangelis Karkaletsis, George A. Vouros, and Panagiotis Stamatopoulos. 2008. Summarization system evaluation revisited: N-gram graphs. ACM Transactions on Audio, Speech, and Language Processing, 5(3):5:1–5:39. https://doi.org/10.1145/1410358.1410359 — [doi:10.1145/1410358.1410359](https://doi.org/10.1145/1410358.1410359) ↗
  21. Dan Gillick and Yang Liu. 2010. Non-expert evaluation of summarization systems is risky. In Proceedings of the 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, Los Angeles, USA, June 6, 2010, pages 148–151. Association for Computational Linguistics. our card
  22. Yvette Graham. 2015. Re-evaluating automatic summarization with BLEU and 192 shades of ROUGE. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 128–137, Lisbon, Portugal, Association for Computational Linguistics. https://doi.org/10.18653/v1/D15-1013 ↗ our card
  23. Yvette Graham and Timothy Baldwin. 2014. Testing for significance of increased correlation with human judgment. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 172–176. our card
  24. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. https://doi.org/10 .3115/v1/D14-1020 our card
  25. Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019. Results of the WMT19 Metrics Shared Task: Segment-level and strong MT systems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 62–90. our card
  26. Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. 2020. Results of the WMT20 Metrics Shared Task. In Proceedings of the Fifth Conference on Machine Translation, pages 688–725. our card
  27. Ramesh Nallapati, Bowen Zhou, Cicero Nogueira dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of CoNLL, pages 280–290. find
  28. Eric W. Noreen. 1989. Computer Intensive Methods for Hypothesis Testing: An Introduction. Wiley, New York, 19:21. https://doi.org/10.18653/v1/K16-1028 ↗
  29. Karolina Owczarzak, John M. Conroy, Hoa Trang Dang, and Ani Nenkova. 2012. An assessment of the accuracy of automatic evaluation in summarization. In Proceedings of Workshop on Evaluation Metrics and System Comparison for Automatic Summarization@NACCL-HLT 2012, Montrèal, Canada, June 2012, 2012, pa… find
  30. Karolina Owczarzak and Hoa Trang Dang. 2011. Overview of the TAC 2011 Summarization Track: Guided task and AESOP task. In Proceedings of the Text Analysis Conference (TAC 2011), Gaithersburg, Maryland, USA, November. find
  31. Kishore Papineni, Salim Roukos, Todd Ward, and Wj Zhu. 2011. BLEU: A method for automatic evaluation of machine translation. In ACL, July, pages 311–318. https://doi.org/10.3115/1073083.1073135 ↗ our card
  32. Maxime Peyrard, Teresa Botschen, and Iryna Gurevych. 2017. Learning to score system summaries for better content selection evaluation. In Proceedings of the Workshop on New Frontiers in Summarization, NFiS@EMNLP 2017, Copenhagen, Denmark, September 7, 2017, pages 74–84. Association for Computational… find
  33. Peter Rankel, John Conroy, Eric Slud, and Dianne O’Leary. 2011. Ranking human and machine summarization systems. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 467–473, Edinburgh, Scotland, UK. Association for Computational Linguistics. find
  34. Peter A. Rankel, John M. Conroy, Hoa Trang Dang, and Ani Nenkova. 2013. A decade of automatic content evaluation of news summaries: Reassessing the state of the art. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 131–136, So… find
  35. Peter A. Rankel, John M. Conroy, and Judith D. Schlesinger. 2012. Better metrics to automatically predict the quality of a text summary. Algorithms, 5(4):398–420. https://doi.org/10.3390/a5040398 — [doi:10.3390/a5040398](https://doi.org/10.3390/a5040398) ↗
  36. Nornadiah Mohd Razali and Yap Bee Wah. 2011. Power comparisons of Shapiro-Wilk, Kolmogorov-Smirnov, Lilliefors and Anderson-Darling Tests. Journal of Statistical Modeling and Analytics, 2(1):21–33. find
  37. Ori Shapira, David Gabay, Yang Gao, Hadar Ronen, Ramakanth Pasunuru, Mohit Bansal, Yael Amsterdamer, and Ido Dagan. 2019. Crowdsourcing lightweight pyramids for manual summary evaluation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguis… find
  38. Samuel Sanford Shapiro and Martin B. Wilk. 1965. An Analysis of variance test for normality (complete samples). Biometrika, 52(3/4):591–611. find
  39. Stephen Tratz and Eduard H. Hovy. 2008. Summarization evaluation using transformed basic elements. In Proceedings of the First Text Analysis Conference, TAC 2008, Gaithersburg, Maryland, USA, November 17–19, 2008. NIST. find
  40. Frank Wilcoxon. 1992. Individual comparisons by ranking methods. In Breakthroughs in Statistics, pages 196–202, Springer. https://doi.org/10.1093/biomet/52.3-4.591 — [doi:10.1093/biomet/52.3-4.591](https://doi.org/10.1093/biomet/52.3-4.591) ↗
  41. Evan James Williams. 1959. Regression Analysis, volume 14, Wiley. find
  42. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. find
  43. Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International… our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/deutsch-2021-statistical-analysis-summarization-evaluation-metrics/