Elamur Research⌕
☰
home / papers / 2019 / Searching for Effective Neural Extractive Summarization: What Works and What's Next

Searching for Effective Neural Extractive Summarization: What Works and What's Next

Paper · 2019 · Ming Zhong, Pengfei Liu, Danqing Wang et al.

Summary

Neural extractive summarizers have improved quickly, but it is unclear which design choices actually drive their gains. Zhong et al. build a training–testing testbed on CNN/DailyMail to take these systems apart. They keep a CNN sentence encoder fixed and vary four things:

  • the document encoder (LSTM vs. Transformer)
  • the decoder (non auto-regressive sequence labeling, SeqLab, vs. an auto-regressive Pointer network)
  • the transferable knowledge (GloVe, BERT, or supervised pre-training on NEWSROOM)
  • the learning schema (supervised vs. reinforcement learning)

They probe the models with cross-domain evaluation, sentence-shuffling tests, and analyses of repetition, sentence length and position bias. Auto-regressive decoding generally helps. LSTM overfits to its architecture more readily than Transformer. On CNN/DailyMail, success depends heavily on sentence position. Unsupervised BERT knowledge transfers better than NEWSROOM pre-training, which is hurt by domain shift. An LSTM+Pointer model with BERT, refined with RL, sets a new state of the art. The work guides future architecture choices and warns that benchmark gains may partly reflect positional bias.

Key points

  • On CNN/DailyMail, ROUGE scores (R-1 / R-2 / R-L) rise step by step from the LSTM + Pointer baseline:
    • baseline: 41.56 / 18.77 / 37.83
    • with BERT: 42.39 / 19.51 / 38.69
    • with BERT and RL: 42.69 / 19.60 / 38.85
    • strongest prior system listed: about 41.6 R-1
  • In a disentangling test, a Transformer given only positional embeddings, without sentence content, still reached 40.08 R-1. Removing positional information caused a large drop.
  • Encoding sentences one at a time with BERT gave about 41.7 R-1. Encoding the full article and mean-pooling tokens gave about 42.3 R-1. This suggests BERT's benefit comes mainly from document-level context, including relations between sentence positions.
  • Context-independent word vectors (GloVe, word2vec) added little. NEWSROOM pre-training usually helped slightly but hurt CNN-LSTM-Pointer, which is the configuration most sensitive to sentence order.
  • Code was released via the fastNLP repository on GitHub.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China65 works · 19,402 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Summarization · 159 works

Cite

Cite as Ming Zhong, Pengfei Liu, Danqing Wang, Xipeng Qiu and Xuanjing Huang. 2019. Searching for Effective Neural Extractive Summarization: What Works and What's Next. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1100

@inproceedings{zhong2019searching,
  title = {{Searching for Effective Neural Extractive Summarization: What Works and What's Next}},
  author = {Ming Zhong, Pengfei Liu, Danqing Wang, Xipeng Qiu and Xuanjing Huang},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1100},
  url = {https://doi.org/10.18653/v1/p19-1100}
}

References

37 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kristjan Arumae and Fei Liu. 2018. Reinforced extractive summarization with question-focused rewards. In Proceedings of ACL 2018, Student Research Workshop. pages 105–111. find
  2. Asli Celikyilmaz, Antoine Bosselut, Xiaodong He, and Yejin Choi. 2018. Deep communicating agents for abstractive summarization. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). v…
  3. Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). volume 1, pages 675–686.
  4. Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by extracting sentences and words.
  5. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). volume 1, pages 484–494. find
  6. Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2018. Transformer-xl: Language modeling with longer-term dependency . find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 .
  8. Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit Cheung. 2018. Banditsum: Extractive summarization as a contextual bandit. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pages 3739–3748. find
  9. Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pages 4098–4109.
  10. Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). vo…
  11. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems. pages 1693– 1701.
  12. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9(8):1735–1780. find
  13. Masaru Isonuma, Toru Fujino, Junichiro Mori, Yutaka Matsuo, and Ichiro Sakata. 2017. Extractive summarization using multi-task learning with document classification. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. pages 2101–2110. find
  14. Aishwarya Jadhav and Vaibhav Rajan. 2018. Extractive summarization with swap-net: Sentences and words from alternating pointer networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). volume 1, pages 142–151. find
  15. Chris Kedzie, Kathleen McKeown, and Hal Daume III. 2018. Content selection in deep learning models of summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pages 1818–1828.
  16. Technologies, Volume 1 (Long Papers). volume 1, pages 2227–2237. find
  17. Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882 .
  18. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out .
  19. Pengfei Liu, Xipeng Qiu, Jifan Chen, and Xuanjing Huang. 2016a. Deep fusion lstms for text semantic matching. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). volume 1, pages 1034–1043. find
  20. Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016b. Recurrent neural network for text classification with multi-task learning. In Proceedings of IJCAI. pages 2873–2879. find
  21. Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2017. Adversarial multi-task learning for text classification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). volume 1, pages 1–10.
  22. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 .
  23. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In Thirty-First AAAI Conference on Artificial Intelligence.
  24. Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Ça glar Gulçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. find
  25. Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Ranking sentences for extractive summarization with reinforcement learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers…
  26. Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304 .
  27. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). pages 1532–1543.
  28. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Matthew…
  29. Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiskỳ, and Phil Blunsom. 2015. Reasoning about entailment with neural attention. arXiv preprint arXiv:1509.06664 .
  30. Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. pages 379–389. find
  31. Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). volume 1, pages 1073–1083. find
  32. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems. pages 3104–3112. find
  33. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems. pages 5998–6008.
  34. Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. 2015. Order matters: Sequence to sequence for sets. arXiv preprint arXiv:1511.06391 . find
  35. Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. 2018. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pages 7794–7803. find
  36. Yuxiang Wu and Baotian Hu. 2018. Learning to extract coherent summary via deep reinforcement learning. In Thirty-Second AAAI Conference on Artificial Intelligence. find
  37. Qingyu Zhou, Nan Yang, Furu Wei, Shaohan Huang, Ming Zhou, and Tiejun Zhao. 2018. Neural document summarization by jointly learning to score and select sentences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). volume 1, pages 654–6…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhong-2019-searching-effective-neural-extractive-summarization/