Elamur Research
☰
home / papers / 2015 / Re-evaluating Automatic Summarization with BLEU and 192 Shades of ROUGE

Re-evaluating Automatic Summarization with BLEU and 192 Shades of ROUGE

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/D15-1013.pdf

Summary

The paper argues that the way automatic summarization metrics are assessed has drifted away from correlation with human judgment, the standard practice in MT. Earlier studies relied on recall-oriented human coverage scores alone, which ignore linguistic quality and favour recall-based metrics. Other studies used proxy criteria, such as separating human summaries from system summaries, or the accuracy of metric/significance-test pairings. They also examined only a handful of ROUGE configurations and had no proper test for whether one metric beats another. Using DUC-2004, Graham merges human coverage scores with human linguistic quality ratings into one overall human score. She then correlates BLEU and 192 ROUGE variants with this score and compares metrics using the Williams test for differences between dependent correlations. The best-supported ROUGE configuration differs from earlier recommendations. BLEU, despite being precision-based, does about as well as the best ROUGE variant. Re-running the Hong et al. (2014) comparison of state-of-the-art systems with the improved variant noticeably changes the system ranking. The work therefore matters for anyone choosing metrics or drawing conclusions from summarization benchmarks.

Key points

  • Scope and data: BLEU plus 192 ROUGE variants were evaluated on DUC-2004. The variants differ in n-gram order, stemming, stop-word removal, average vs. median aggregation, and precision/recall/F-score. A prior study (Owczarzak et al., 2012) covered just six variants, under 4% of the space.
  • Human score: Coverage scores were averaged with linguistic quality ratings. These ratings cover 7 criteria graded A–E and were mapped to 100%–0%. Geometric and harmonic means were also tested as alternative combinations.
  • Significance testing: The Williams test is presented as the first appropriate significance test for comparing summarization metrics.
  • Main findings: The recommended variant is average ROUGE-2 precision with stemming and stop words removed. BLEU reached the highest correlations overall but was not significantly better than this ROUGE variant.
  • Replication of Hong et al. (2014): The original evaluation used median ROUGE-2 recall with stemming and stop words kept. With the new variant, the original fourth-place system moved to first. DPP, ICSI Summ and RegSum were not significantly outperformed by any other system according to paired t-tests.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation

Topics: Machine translation

Cited by

65 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 60 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d15-1013

@inproceedings{graham2015re,
  title = {{Re-evaluating Automatic Summarization with BLEU and 192 Shades of ROUGE}},
  author = {Graham, Yvette},
  booktitle = {Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1013},
  url = {https://doi.org/10.18653/v1/d15-1013}
}

References

20 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2008. Further meta-evaluation of machine translation. In Proc. 3rd Wkshp. Statistical Machine Translation, pages 70–106, Columbus, Ohio. Association for Computational Linguistics. find
  2. Keelan Evanini, Shasha Xie, and Klaus Zechner. 2013. Prompt-based content scoring for automated spoken language assessment. In Proc. 2013 Conf. North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 157–162, Atlanta, GA. Association for Computational Linguistics. find
  3. Yvette Graham and Timothy Baldwin. 2014. Testing for significance of increased correlation with human judgment. In Proc. 2014 Conf. Empirical Methods in Natural Language Processing, pages 172–176, Doha, Qatar. Association for Computational Linguistics. find
  4. Yvette Graham, Nitika Mathur, and Timothy Baldwin. 2014. Randomized significance tests in machine translation. In Proc. 9th Wkshp. Statistical Machine Translation, pages 266–274, Baltimore, MD. Association for Computational Linguistics. find
  5. Yvette Graham, Nitika Mathur, and Timothy Baldwin. 2015. Accurate evaluation of segment-level machine translation metrics. In Proc. 2015 Conf. North American Chapter of the Association for Computational Linguistics - Human Language Technologies, pages 1183–1191, Denver, CO. Association for Computational Linguistics. find
  6. Yvette Graham. 2015. Improving evaluation of machine translation quality estimation. In Proc. Fifty-Third Annual Meeting of the Association for Computational Linguistics, pages 1804–1813, Beijing, China. Association for Computational Linguistics. find
  7. Kai Hong, John M Conroy, Benoit Favre, Alex Kulesza, Hui Lin, and Ani Nenkova. 2014. A repository of state of the art and competitive baseline summaries for generic news summarization. In Proc. 9th edition of the Language Resources and Evaluation Conference, pages 1608–1616, Reykjavik, Iceland. find
  8. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proc. 45th Annual Meeting of the Association for Computational Linguistics, pages 177–180, Prague, Czech Republic. Association for Computational Linguistics. find
  9. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proc. 2004 Conf. Empirical Methods in Natural Language Processing, pages 388–395, Barcelona, Spain. Association for Computational Linguistics. find
  10. Chin-Yew Lin and Eduard Hovy. 2003. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proc. 2003 Conf. North American Chapter of the Association for Computational Linguistics on Human Language Technology, pages 71–78, Edmonton, Canada. Association for Computational Linguistics. find
  11. Paul Over, Hoa Dang, and Donna Harman. 2007. Duc in context. Information Processing & Management, 43(6):1506–1520. find
  12. Karolina Owczarzak, John M. Conroy, Hoa Trang Dang, and Ani Nenkova. 2012. An assessment of the accuracy of automatic evaluation in summarization. In Proc. Wkshp. on Evaluation Metrics and System Comparison for Automatic Summarization, pages 1–9, Quebec, Canada. Association for Computational Linguistics. find
  13. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU : A method for automatic evaluation of machine translation. In Proc. 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, PA. Association for Computational Linguistics. find our card
  14. Emily Pitler, Annie Louis, and Ani Nenkova. 2010. Automatic evaluation of linguistic quality in multi-document summarization. In Proc. 48th Annual Meeting of the Association for Computational Linguistics, pages 544–554, Uppsala, Sweden. Association for Computational Linguistics. find
  15. Peter Rankel, John M. Conroy, Eric V. Slud, and Dianne P. O’Leary. 2011. Ranking human and machine summarization systems. In Proc. 2011 Conf. Empirical Methods in Natural Language Processing, pages 467–473, Edinburgh, Scotland. Association for Computational Linguistics. find
  16. Patrick Royston. 1982. Algorithm as 181: The W test for normality. Applied Statistics, 31:176–180. find
  17. James H. Steiger. 1980. Tests for comparing elements of a correlation matrix. Psychological Bulletin, 87(2):245–251. find
  18. Evan J. Williams. 1959. Regression analysis, volume 14. Wiley New York. find
  19. Helen Yannakoudakis and Ted Briscoe. 2012. Modeling coherence in ESOL learner texts. In Proc. Seventh Wkshp. on Building Educational Applications Using NLP, pages 33–43, Montreal, Canada. Association for Computational Linguistics. find
  20. Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A new dataset and method for automatically grading ESOL texts. In Proc. 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 180– 189, Portland, OR. Association for Computational Linguistics. find