Elamur Research⌕
☰
home / papers / 2004 / ROUGE: A Package for Automatic Evaluation of Summaries

ROUGE: A Package for Automatic Evaluation of Summaries

Summary

Manual evaluation of automatic summaries is very expensive. A human assesses coherence, grammaticality and content coverage, so even a moderate DUC-scale campaign requires more than 3000 person-hours. The author proposes ROUGE, a software package that evaluates a generated summary by comparing it with one or more human-written reference summaries. The approach is inspired by BLEU from machine translation. However, BLEU is based on precision, whereas ROUGE is based on recall. The paper describes four families of metrics. ROUGE-N counts shared n-grams. ROUGE-L is based on the longest common subsequence (LCS). ROUGE-W weights the LCS and favors consecutive matches. ROUGE-S counts word pairs with arbitrary gaps between them, and the SU variant additionally takes unigrams into account. The metrics were tested on DUC 2001–2003 data: for each one, the Pearson correlation with human judgments was computed, and confidence intervals were obtained using the bootstrap method. For single-document summaries and very short summaries, the correlation is high. For multi-document summaries, it is noticeably lower. ROUGE has become the de facto standard for automatic evaluation of summarization and still serves as a baseline metric in NLP projects that generate condensed text.

Key points

  • The package contains the metrics ROUGE-N, ROUGE-L, ROUGE-W and ROUGE-S; NIST used three of them in the DUC 2004 evaluation.
  • When there are multiple references, the best of the pairwise scores is taken. The implementation also applies jackknifing: scores are computed on M sets of M−1 references each and averaged. This also makes it possible to estimate the average performance of human authors.
  • In a four-word example, ROUGE-L distinguishes between sentences with the same bigrams but different word order (0.75 vs. 0.5). At the same time, it ignores alternative LCSs and shorter common fragments.
  • Findings from the DUC data:
    • for single-document summaries, ROUGE-2, L, W and S work well;
    • for headline-like summaries, ROUGE-1, L, W, SU4 and SU9;
    • for multi-document summaries, ROUGE-1, 2, S4, S9, SU4 and SU9 are acceptable if stopwords are removed.
  • Removing stopwords and using multiple references generally increase the correlation. For multi-document summaries of 200 and 400 words, the correlation exceeds 0.70. The author attributes the instability of results in these tasks to the small sample size: about 30 samples versus more than 100 in the single-document tasks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{lin2004rouge,
  title = {{ROUGE: A Package for Automatic Evaluation of Summaries}},
  author = {Lin, Chin-Yew},
  booktitle = {Text Summarization Branches Out},
  year = {2004},
  url = {https://aclanthology.org/W04-1013/}
}

References

13 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Cormen, T. R., C. E. Leiserson, and R. L. Rivest. 1989. Introduction to Algorithms. The MIT Press. find
  2. Davison, A. C. and D. V. Hinkley. 1997. Bootstrap Methods and Their Application. Cambridge University Press. find
  3. Lin, C.-Y. and E. H. Hovy. 2003. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of 2003 Language Technology Conference (HLT-NAACL 2003), Edmonton, Canada. find our card
  4. Lin, C.-Y. 2004. Looking for a few good metrics: R OUGE and its evaluation. In Proceedings of NTCIR Workshop 2004, Tokyo, Japan. find
  5. Lin, C.-Y. and F. J. Och. 2004. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of 42 nd Annual Meeting of ACL (ACL 2004), Barcelona, Spain. find our card
  6. Mani, I. 2001. Automatic Summarization. John Benjamins Publishing Co. find
  7. Melamed, I. D. 1995. Automatic evaluation and uniform filter cascades for inducing n-best translation lexicons. In Proceedings of the 3 rd Workshop on Very Large Corpora (WVLC3). Boston, U.S.A. find
  8. Melamed, I. D., R. Green and J. P. Turian (2003). Precision and recall of machine translation. In Proceedings of 2003 Language Technology Conference (HLT-NAACL 2003), Edmonton, Canada. find our card
  9. Over, P. and J. Yen. 2003. An introduction to DUC 2003 – Intrinsic evaluation of generic news text summarization systems. AAAAAAAAAA http://www-nlpir.nist.gov/projects/duc/pubs/2003slides/duc2003intro.pdf ↗
  10. Papineni, K., S. Roukos, T. Ward, and W.-J. Zhu. 2001. B LEU : A method for automatic evaluation of machine translation. IBM Research Report RC22176 (W0109-022). find
  11. Saggion H., D. Radev, S. Teufel, and W. Lam. 2002. Meta-evaluation of summaries in a cross-lingual environment using content-based metrics. In Proceedings of COLING-2002, Taipei, Taiwan. find
  12. Radev, D. S. Teufel, H. Saggion, W. Lam, J. Blitzer, A. Gelebi, H. Qi, E. Drabek, and D. Liu. 2002. Evaluation of Text Summarization in a Cross-Lingual Information Retrieval Framework. Technical report, Center for Language and Speech Processing, Johns Hopkins University, Baltimore, MD, USA. find
  13. Van Rijsbergen, C. J. 1979. Information Retrieval. Butterworths. London. find

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/