Elamur Research
☰
home / papers / 2019 / Neural Text Summarization: A Critical Evaluation

Neural Text Summarization: A Critical Evaluation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D19-1051.pdf

Summary

Progress in neural text summarization has stalled. Leading systems trained on the CNN/DailyMail news benchmark barely beat the Lead-3 baseline, which simply copies an article's first three sentences. The authors argue that the research setup itself is partly to blame, and they examine its three components: datasets, evaluation metrics and model outputs.

For datasets, they ran a human annotation study on selecting important sentences. It suggests that one reference summary per document, with no extra guidance, leaves the task too ambiguous. For evaluation, they show that ROUGE, the standard lexical-overlap metric, tracks human judgment only weakly and never checks factual accuracy. A manual audit of abstractive outputs found frequent factual errors. For models, they compared outputs from 13 published systems. The models lean heavily on the lead-heavy layout of news articles and produce strikingly similar summaries.

The paper urges the community to build better-constrained datasets, design models less tied to domain-specific bias, and adopt evaluation that captures properties such as factual consistency.

Key points

  • Manual review of 200 article–summary pairs from randomly chosen abstractive models found factual inconsistencies in 60 of them (30%).
  • On the CNN/DM test set, 10 abstractive and 3 extractive models (e.g., See et al. 2017, Chen and Bansal 2018, Zhou et al. 2018) reach ROUGE-1 of about 39.5–41.6 against the original references.
  • When Lead-3 is used as the reference instead, ROUGE-1 rises to roughly 52–79. This exposes strong layout bias, though a large gap remains to the extractive oracle.
  • Pairwise ROUGE-1/ROUGE-4 between model outputs exceeds model–reference overlap. ROUGE-1 varies much less than ROUGE-4, so the models share vocabulary but arrange it into phrases differently.
  • ROUGE extensions such as ParaEval, ROUGE-WE, ROUGE 2.0 and ROUGE-G have not displaced plain ROUGE as the default metric.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

224 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 214 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1051

@inproceedings{kryscinski2019neural,
  title = {{Neural Text Summarization: A Critical Evaluation}},
  author = {Kryscinski, Wojciech and Keskar, Nitish Shirish and McCann, Bryan and Xiong, Caiming and Socher, Richard},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1051},
  url = {https://doi.org/10.18653/v1/d19-1051}
}

References

82 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Katja Filippova and Yasemin Altun. 2013. Overcoming the lack of parallel data in sentence compression. In Proceedings of EMNLP, pages 1481–1491. Citeseer. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR. find our card
  3. Kavita Ganesan. 2018. ROUGE 2.0: Updated and improved measures for evaluation of summarization tasks. CoRR, abs/1803.01937. — [arXiv:1803.01937](https://arxiv.org/abs/1803.01937) find
  4. Chris Callison-Burch, Cameron S. Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. (meta-) evaluation of machine translation. In WMT@ACL, pages 136–158. Association for Computational Linguistics. find
  5. Sebastian Gehrmann, Yuntian Deng, and Alexander M. Rush. 2018. Bottom-up abstractive summarization. In EMNLP, pages 4098–4109. Association for Computational Linguistics. find our card
  6. Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluation the role of bleu in machine translation research. In EACL. The Association for Computer Linguistics. find
  7. Dimitra Gkatzia and Saad Mahamood. 2015. A snapshot of NLG evaluation practices 2005 - 2014. In ENLG 2015 - Proceedings of the 15th European Workshop on Natural Language Generation, 10-11 September 2015, University of Brighton, Brighton, UK, pages 57–60. find
  8. Danqi Chen, Jason Bolton, and Christopher D. Manning. 2016. A thorough examination of the cnn/daily mail reading comprehension task. In ACL (1). The Association for Computer Linguistics. find our card
  9. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking NLI systems with sentences that require simple lexical inferences. In ACL (2), pages 650–655. Association for Computational Linguistics. find our card
  10. Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In ACL (1), pages 675–686. Association for Computational Linguistics. find our card
  11. Sumit Chopra, Michael Auli, and Alexander M. Rush. 2016. Abstractive sentence summarization with attentive recurrent neural networks. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego Californ… find our card
  12. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In CVPR, pages 6325–6334. IEEE Computer Society. find our card
  13. David Graff and C Cieri. 2003. English gigaword, linguistic data consortium. find
  14. Eric Chu and Peter J. Liu. 2018. Unsupervised neural multi-document abstractive summarization. CoRR, abs/1810.05739. — [arXiv:1810.05739](https://arxiv.org/abs/1810.05739) find
  15. Yvette Graham. 2015. Re-evaluating automatic summarization with BLEU and 192 shades of ROUGE. In EMNLP, pages 128–137. The Association for Computational Linguistics. find our card
  16. Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational… find our card
  17. Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orlean… find our card
  18. Carlos A Colmenares, Marina Litvak, Amin Mantrach, and Fabrizio Silvestri. 2015. Heads: Headline generation as sequence prediction using an abstract feature-rich space. In HLT-NAACL, pages 133–142. find
  19. Han Guo, Ramakanth Pasunuru, and Mohit Bansal. 2018. Soft layer-specific multi-task summarization with entailment and question generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Pape… find
  20. Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit Cheung. 2018. Banditsum: Extractive summarization as a contextual bandit. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018. find
  21. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In NAACL-HLT (2), pages 107–112. Association for Computational Linguistics. find
  22. Bonnie Dorr, David Zajic, and Richard Schwartz. 2003. Hedge trimmer: A parse-and-trim approach to headline generation. In HLT-NAACL. find
  23. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In NIPS. find our card
  24. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Proc. ACL workshop on Text Summarization Branches Out, page 10. find
  25. Zachary C. Lipton and Jacob Steinhardt. 2019. Troubling trends in machine learning scholarship. ACM Queue, 17(1):80. find
  26. Kai Hong and Ani Nenkova. 2014. Improving the estimation of word importance for news multi-document summarization. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2014, April 26-30, 2014, Gothenburg, Sweden. find
  27. Feifan Liu and Yang Liu. 2010. Exploring correlation between ROUGE and human evaluation on meeting summaries. IEEE Trans. Audio, Speech & Language Processing, 18(1):187–196. find
  28. Wan Ting Hsu, Chieh-Kai Lin, Ming-Ying Lee, Kerui Min, Jing Tang, and Min Sun. 2018. A unified model for extractive and abstractive summarization using inconsistency loss. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July… find
  29. Jingyun Liu, Jackie Chi Kit Cheung, and Annie Louis. 2019. What comes next? extractive summarization by next-sentence prediction. CoRR, abs/1901.03859. — [arXiv:1901.03859](https://arxiv.org/abs/1901.03859) find
  30. Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summarizing long sequences. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Pr… find our card
  31. Yichen Jiang and Mohit Bansal. 2018. Closed-book training to improve summarization encoder memory. In EMNLP, pages 4067–4077. Association for Computational Linguistics. find
  32. Divyansh Kaushik and Zachary C. Lipton. 2018. How much reading does reading comprehension require? A critical investigation of popular benchmarks. In EMNLP, pages 5010–5015. Association for Computational Linguistics. find our card
  33. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In AAAI. find our card
  34. Chris Kedzie, Kathleen R. McKeown, and Hal Daumé III. 2018. Content selection in deep learning models of summarization. In EMNLP, pages 1818–1828. Association for Computational Linguistics. find
  35. Ramesh Nallapati, Bowen Zhou, Çağlar Gülçehre, Bing Xiang, et al. 2016a. Abstractive text summarization using sequence-to-sequence rnns and beyond. Proceedings of SIGNLL Conference on Computational Natural Language Learning. find
  36. Byeongchang Kim, Hyunwoo Kim, and Gunhee Kim. 2018. Abstractive summarization of reddit posts with multi-level memory networks. CoRR, abs/1811.00783. find
  37. Ramesh Nallapati, Bowen Zhou, and Mingbo Ma. 2016b. Classify or select: Neural architectures for extractive document summarization. CoRR, abs/1611.04244. — [arXiv:1611.04244](https://arxiv.org/abs/1611.04244) find
  38. Mahnaz Koupaee and William Yang Wang. 2018. Wikihow: A large scale text summarization dataset. CoRR, abs/1810.09305. — [arXiv:1810.09305](https://arxiv.org/abs/1810.09305) find
  39. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018a. Don’t give me the details, just the summary! Topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium. find
  40. Wojciech Kryściński, Romain Paulus, Caiming Xiong, and Richard Socher. 2018. Improving abstraction in text summarization. In EMNLP, pages 1808–1817. Association for Computational Linguistics. find
  41. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018b. Ranking sentences for extractive summarization with reinforcement learning. In NAACL-HLT, pages 1747–1759. Association for Computational Linguistics. find
  42. Moontae Lee, Xiaodong He, Wen-tau Yih, Jianfeng Gao, Li Deng, and Paul Smolensky. 2016. Reasoning in vector space: An exploratory study of question answering. In ICLR. find
  43. Junyi Jessy Li, Kapil Thadani, and Amanda Stent. 2016. The role of discourse units in near-extractive summarization. In Proceedings of the SIGDIAL 2016 Conference, The 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, 13-15 September 2016, Los Angeles, CA, USA. find
  44. Shashi Narayan, Nikos Papasarantopoulos, Mirella Lapata, and Shay B. Cohen. 2017. Neural extractive summarization with side information. CoRR, abs/1704.04530. — [arXiv:1704.04530](https://arxiv.org/abs/1704.04530) find
  45. Ani Nenkova and Rebecca J. Passonneau. 2004. Evaluating content selection in summarization: The pyramid method. In Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, HLT-NAACL 2004, Boston, Massachusetts, USA, May 2-7, 2004. find
  46. Wei Li, Xinyan Xiao, Yajuan Lyu, and Yuanzhuo Wang. 2018. Improving neural abstractive document summarization with structural regularization. In EMNLP, pages 4078–4087. Association for Computational Linguistics. find
  47. Joel Larocca Neto, Alex A Freitas, and Celso AA Kaestner. 2002. Automatic text summarization using a machine learning approach. In Brazilian Symposium on Artificial Intelligence, pages 205–215. Springer. find
  48. PurdueOWL. 2019. Journalism and journalistic writing: The inverted pyramid structure. Accessed: 2019-05-15. find
  49. Ehud Reiter. 2018. A structured review of the validity of BLEU. Computational Linguistics, 44(3). find our card
  50. Jun-Ping Ng and Viktoria Abrecht. 2015. Better summarization evaluation with word embeddings for ROUGE. CoRR, abs/1508.06034. find
  51. Ehud Reiter and Anja Belz. 2009. An investigation into the validity of some metrics for automatically evaluating natural language generation systems. Computational Linguistics, 35(4):529–558. find
  52. Benjamin Nye and Ani Nenkova. 2015. Identification and characterization of newsworthy verbs in world news. In NAACL HLT 2015, The 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Denver, Colorado, USA, May 31 - June 5, 2015. find
  53. Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. Proceedings of EMNLP. find
  54. Evan Sandhaus. 2008. The new york times annotated corpus. Linguistic Data Consortium, Philadelphia, 6(12):e26752. find
  55. Paul Over and James Yen. 2001. An introduction to duc-2001: Intrinsic evaluation of generic news text summarization systems. find
  56. John Schulman, Nicolas Heess, Theophane Weber, and Pieter Abbeel. 2015. Gradient estimation using stochastic computation graphs. In NIPS. find
  57. Paul Over and James Yen. 2002. An introduction to duc-2002: Intrinsic evaluation of generic news text summarization systems. find
  58. Raphael Schumann. 2018. Unsupervised abstractive sentence summarization using length controlled variational autoencoder. CoRR, abs/1809.05233. — [arXiv:1809.05233](https://arxiv.org/abs/1809.05233) find
  59. Paul Over and James Yen. 2003. An introduction to duc-2003: Intrinsic evaluation of generic news text summarization systems. find
  60. D. Sculley, Jasper Snoek, Alexander B. Wiltschko, and Ali Rahimi. 2018. Winner’s curse? on pace, progress, and empirical rigor. In ICLR (Workshop). OpenReview.net. find
  61. Karolina Owczarzak, Peter A. Rankel, Hoa Trang Dang, and John M. Conroy. 2012. Assessing the effect of inconsistent assessors on summarization evaluation. In ACL (2), pages 359–362. The Association for Computer Linguistics. find
  62. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In ACL. find
  63. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL, pages 311– 318. ACL. find our card
  64. Elaheh ShafieiBavani, Mohammad Ebrahimi, Raymond K. Wong, and Fang Chen. 2018. A graph-theoretic summary evaluation for rouge. In EMNLP, pages 762–767. Association for Computational Linguistics. find
  65. Rebecca J. Passonneau, Emily Chen, Weiwei Guo, and Dolores Perin. 2013. Automated pyramid scoring of summaries using distributional semantics. In ACL (2), pages 143–147. The Association for Computer Linguistics. find
  66. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In NIPS. find
  67. Sho Takase, Jun Suzuki, Naoaki Okazaki, Tsutomu Hirao, and Masaaki Nagata. 2016. Neural headline generation on abstract meaning representation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016. find
  68. Ramakanth Pasunuru and Mohit Bansal. 2018. Multi-reward reinforced summarization with saliency and entailment. CoRR, abs/1804.06451. find
  69. Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. In ICLR. find our card
  70. Jiwei Tan, Xiaojun Wan, and Jianguo Xiao. 2017. Abstractive document summarization with a graph-based attentional neural model. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers. find
  71. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In *SEM@NAACL-HLT, pages 180–191. Association for Computational Linguistics. find
  72. Liling Tan, Jon Dehdari, and Josef van Genabith. 2015. An awkward disparity between BLEU / RIBES scores and human judgements in machine translation. In WAT, pages 74–81. Workshop on Asian Translation. find
  73. Matt Post. 2018. A call for clarity in reporting BLEU scores. In WMT, pages 186–191. Association for Computational Linguistics. find our card
  74. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017,… find
  75. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In NIPS. find
  76. Yuxiang Wu and Baotian Hu. 2018. Learning to extract coherent summary via deep reinforcement learning. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educa… find
  77. Yongqin Xian, Bernt Schiele, and Zeynep Akata. 2017. Zero-shot learning - the good, the bad and the ugly. In CVPR, pages 3077–3086. IEEE Computer Society. find
  78. Jiacheng Xu and Greg Durrett. 2019. Neural extractive text summarization with syntactic compression. CoRR, abs/1902.00863. find
  79. Yinfei Yang and Ani Nenkova. 2014. Detecting information-dense texts in multiple news domains. In Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, July 27 -31, 2014, Québec City, Québec, Canada. find
  80. Fangfang Zhang, Jin-ge Yao, and Rui Yan. 2018. On the abstractiveness of neural document summarization. In EMNLP, pages 785–790. Association for Computational Linguistics. find
  81. Liang Zhou, Chin-Yew Lin, Dragos Stefan Munteanu, and Eduard H. Hovy. 2006. Paraeval: Using paraphrases to evaluate summaries automatically. In Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, Proceedings, June 4-9, 2006, New York, N… find
  82. Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang, Ming Zhou, and Tiejun Zhao. 2018. Neural document summarization by jointly learning to score and select sentences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2… find