Elamur Research⌕
☰
home / papers / 2015 / Better Summarization Evaluation with Word Embeddings for ROUGE

Better Summarization Evaluation with Word Embeddings for ROUGE

Paper · 2015 · Jun-Ping Ng and Viktoria Abrecht

Summary

ROUGE scores summaries by counting exact matches of words and n-grams against human references. As a result, it penalizes system output that conveys the same content in different wording, which is a problem for paraphrase-heavy or abstractive summarizers. Ng and Abrecht propose ROUGE-WE, which keeps the ROUGE framework but swaps the binary match test for a soft similarity. Each word is mapped to a pre-trained word2vec vector, and two units are scored by the dot product of their vectors. Out-of-vocabulary items contribute zero. For bigram and skip-bigram variants, word vectors are combined by element-wise multiplication.

The authors test this on the TAC 2011 AESOP data. They correlate metric scores with human pyramid, responsiveness and readability judgements. ROUGE-WE-1 generally achieves the strongest rank-based (Spearman/Kendall) correlations. On pyramid it also beats the best AESOP 2011 system under Spearman. Standard ROUGE-SU4 still wins on Pearson for pyramid and responsiveness.

The work matters because it shows a cheap, drop-in way to make a dominant evaluation metric sensitive to meaning rather than surface form.

Key points

  • Similarity function. Exact lexical match (1/0) is replaced by the dot product of word2vec vectors. The vectors are the 3 million pre-trained mappings learned from Google News. OOV words score 0.
  • N-gram vectors. Bigram vectors are built by multiplying the constituent word vectors element-wise, following Mitchell and Lapata (2008). This reduces OOV for ROUGE-WE-2 and ROUGE-WE-SU4.
  • Data. The TAC 2011 AESOP setup has 44 topics, each with 10 documents and 4 human model summaries. Summaries come from 51 participating systems. Correlations are reported with Pearson, Spearman and Kendall coefficients.
  • Main results for ROUGE-WE-1.
    • Pyramid: Spearman 0.9138 versus 0.9085 for ROUGE-1.
    • Readability: Spearman 0.4312 and Kendall 0.3216, versus at most 0.3975 and 0.2925 for plain ROUGE.
    • Responsiveness: ROUGE-1 is marginally higher on Kendall (0.6334 vs 0.6308).
    • Pearson: ROUGE-SU4 stays best for pyramid (0.9806) and responsiveness (0.9545).
  • Bigram variants.
    • ROUGE-WE-2 beats ROUGE-2 in most settings.
    • ROUGE-WE-SU4 only beats ROUGE-SU4 on readability.
    • Against AESOP 2011 leaders, ROUGE-WE-1 outranks C_S_IIITH3 on Spearman (0.9138 vs 0.9033) but trails it on Kendall (0.7534 vs 0.7582).

Interesting ideas

  • I1. Soft matching as a drop-in replacement. Turning a hard equality check inside an established metric into a continuous similarity keeps the metric's familiar structure while adding semantic tolerance. The same trick could upgrade other overlap-based scores, for example in MT or QA evaluation, especially for morphologically rich languages like Ukrainian, where surface forms vary a lot.
  • I2. Simple OOV fallback. Unknown items contribute zero, so the metric degrades gracefully to roughly lexical behaviour instead of breaking. This is a pragmatic default when embedding coverage is incomplete.
  • I3. Cheap n-gram composition. Element-wise multiplication of word vectors gives phrase representations with no training. The paper also shows its limit: it works for contiguous bigrams but poorly for skip-bigrams. This is a useful warning when choosing which units to compose.
  • I4. Reporting rank correlations alongside Pearson. The authors argue that two independent scoring schemes need not be linearly related. Reporting Spearman and Kendall avoids rewarding mere linearity, and conclusions can change depending on which coefficient is used.
  • I5. Checking test-set bias. The authors note that AESOP outputs are mostly extractive, so the data may underestimate the benefit of semantic matching. Asking whether the evaluation set contains the phenomena a metric targets is worth doing before trusting any metric comparison.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢BloombergNew York · United States5 works · 1,546 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 685 worksSummarization · 159 works

Cite

Cite as Jun-Ping Ng and Viktoria Abrecht. 2015. Better Summarization Evaluation with Word Embeddings for ROUGE. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d15-1222

@inproceedings{ng2015better,
  title = {{Better Summarization Evaluation with Word Embeddings for ROUGE}},
  author = {Jun-Ping Ng and Viktoria Abrecht},
  booktitle = {Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1222},
  url = {https://doi.org/10.18653/v1/d15-1222}
}

References

24 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A Neural Probabilistic Language Model. The Journal of Machine Learning Research, 3:1137–1155.
  2. Hoa Trang Dang and Karolina Owczarzak. 2009. Overview of the TAC 2009 Summarization Track. find
  3. In Proceedings of the Text Analysis Conference (TAC). find
  4. Paulo C. F. de Oliveira. 2011. CatolicaSC at TAC 2011. In Proceedings of the Text Analysis Conference (TAC). find
  5. George Giannakopoulos and Vangelis Karkaletsis. 2009. AutoSummENG and MeMoG in Evaluating Guided Summaries. In Proceedings of the Text Analysis Conference (TAC). find
  6. Eduard Hovy, Chin-Yew Lin, and Liang Zhou. 2005. find
  7. Evaluating DUC 2005 using Basic Elements. In Proceedings of the Document Understanding Conference (DUC). find
  8. Niraj Kumar, Kannan Srinathan, and Vasudeva Varma. 2011. Using Unsupervised System with Least Linguistic Features for TAC-AESOP Task. In Proceedings of the Text Analysis Conference (TAC). find
  9. Chen Li, Fei Liu, Fuliang Weng, and Yang Liu. 2013. Document Summarization via Guided Sentence Compression. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 490–500. find
  10. Chin-Yew Lin. 2004a. Looking for a Few Good Metrics: ROUGE and its Evaluation. In Working Notes of the 4th NTCIR Workshop Meeting. find
  11. Chin-Yew Lin. 2004b. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop.
  12. Fei Liu, Jeffrey Flanigan, Sam Thomson, Norman Sadeh, and Noah A. Smith. 2015. Toward Abstractive Summarization Using Semantic Representations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pa…
  13. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013a. Distributed Representations of Words and Phrases and their Compositionality. In Proceedings of the 27th Annual Conference on Neural Information Processing Systems (NIPS), pages 3111–3119.
  14. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013b. Linguistic Regularities in Continuous Space Word Representations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 746–751.
  15. Jeff Mitchell and Mirella Lapata. 2008. Vector-based Models of Semantic Composition. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL), pages 236–244. find
  16. Jun-Ping Ng, Yan Chen, Min-Yen Kan, and Zhoujun Li. 2014. Exploiting Timelines to Enhance Multi-document Summarization. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 923–933. find
  17. Paul Over and James Yen. 2004. An Introduction to DUC 2004 Intrinsic Evaluation of Generic New Text Summarization Systems. In Proceedings of the Document Understanding Conference (DUC). find
  18. Karolina Owczarzak and Hoa Trang Dang. 2011. Overview of the TAC 2011 Summarization Track: Guided Task and AESOP Task. In Proceedings of the Text Analysis Conference (TAC). find
  19. Karolina Owczarzak. 2010. Overview of the TAC 2010 Summarization Track. In Proceedings of the Text Analysis Conference (TAC). find
  20. Rebecca J Passonneau, Ani Nenkova, Kathleen McKeown, and Sergey Sigelman. 2005. Applying the Pyramid Method in DUC 2005. In Proceedings of the Document Understanding Conference (DUC). find
  21. Rebecca J Passonneau, Emily Chen, Weiwei Guo, and Dolores Perin. 2013. Automated Pyramid Scoring of Summaries using Distributional Semantics. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL), pages 143–147. find
  22. Josef Steinberger and Karel Ježek. 2012. Evaluation Measures for Text Summarization. Computing and Informatics, 28(2):251–275. find
  23. Wenpeng Yin and Hinrich Schütze. 2014. An Exploration of Embeddings for Generalized Phrases. In Proceedings of the ACL 2014 Student Research Workshop, pages 41–47. find
  24. Mo Yu, Matthew Gormley, and Mark Dredze. 2014. Factor-based Compositional Embedding Models. In Proceedings of the NIPS 2014 Deep Learning and Representation Learning Workshop. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ng-2015-better-summarization-evaluation-word-embeddings/