Elamur Research⌕
☰
home / papers / 2017 / The limits of automatic summarisation according to ROUGE

The limits of automatic summarisation according to ROUGE

Paper · 2017 · Natalie Schluter

Summary

This short paper questions how meaningful ROUGE-based progress in extractive summarisation really is. It first gives what the author presents as the first formal proof that building an extractive summary that maximises ROUGE-n is NP-hard. The proof is a reduction from the max k-dominating set problem on cubic graphs: vertex neighbourhoods become sentences, and the graph's vertex set becomes the reference summary. Dummy-symbol padding extends the result from ROUGE-1 to any n.

Despite this hardness, the empirical comparison tells a different story. Oracle extractive summaries were built greedily and exactly (via an ILP solver) on three benchmarks: duc04 (newswire), echr (court judgments) and wiki (Wikipedia). The two methods reach very similar scores. The best attainable scores are also far below 100% and differ widely between datasets. Averaging over multiple reference summaries makes perfect scores impossible, and human summaries score low against one another.

This matters because ROUGE has no natural upper bound. It also offers little reason to expect gains from switching to exact or supervised methods.

Key points

  • Theoretical result: Theorems 1–2 show that exact oracle extractive summarisation is NP-hard under ROUGE-1 and, more generally, ROUGE-n.
  • Datasets and budgets:
    • duc04: 30 document sets, with budgets of 665 bytes and 100 words.
    • echr: 138 pairs from HUDOC, 805-word budget.
    • wiki: 111 "good article" pairs, 335 budget.
  • Tools:
    • Evaluation used ROUGE-1.5.5.
    • Exact oracles were computed with the GLPK solver.
    • The greedy oracle adds sentences by length-normalised ROUGE gain and clips the final sentence to fit the budget.
  • Greedy vs. exact: Oracle ROUGE-1 is roughly 50 on duc04, about 64 on wiki and about 84 on echr, with greedy and exact within about a point of each other. On duc04, greedy is even slightly better, likely thanks to sentence clipping.
  • Human ceiling: On duc04, scoring one reference summary against the other three yields only 39.92 ROUGE-1 and 9.39 ROUGE-2. In one example set, 136 of 160 required stemmed words can only be partially matched.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CopenhagenCopenhagen · Denmark78 works · 14,228 citations🎓IT University of CopenhagenCopenhagen · Denmark16 works · 2,225 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Summarization · 159 works

Cite

Cite as Natalie Schluter. 2017. The limits of automatic summarisation according to ROUGE. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{schluter2017limits,
  title = {{The limits of automatic summarisation according to ROUGE}},
  author = {Natalie Schluter},
  booktitle = {Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers},
  year = {2017},
  doi = {10.18653/v1/e17-2007},
  url = {https://doi.org/10.18653/v1/e17-2007}
}

References

8 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sumit Chopra, Michael Auli, and Alexander M. Rush. 2016. Abstractive sentence summarization with attentive recurrent neural networks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 93–98, San Di…
  2. M.R. Garey and D. S. Johnson. 1979. Computers and Intractability: A Guide to the Theory of NP-Completeness. Freeman and Company. find
  3. Dan Gillick and Benoit Favre. 2009. A scalable global model for summarization. In Proceedings of ILP, pages 10–18.
  4. Yvette Graham. 2015. Re-evaluating automatic summarization with bleu and 192 shades of rouge. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 128–137, Lisbon, Portugal.
  5. Kai Hong, John M. Conroy, Benoit Favre, Alex Kulesza, Hui Lin, and Ani Nenkova. 2014. A repository of state of the art and competitive baseline summaries for generic news summarization. In Proc of LREC. find
  6. Chin-Yew Lin and Eduard Hovy. 2003. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of HLT-NAACL, Edmonton, AB, Canada.
  7. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pages 74–81, Barcelona, Spain.
  8. Feifan Liu and Yang Liu. 2008. Correlation between rouge and human evaluation of extractive meeting find

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schluter-2017-limits-automatic-summarisation-according-rouge/