Elamur Research⌕
☰
home / papers / 2018 / Neural Document Summarization by Jointly Learning to Score and Select Sentences

Neural Document Summarization by Jointly Learning to Score and Select Sentences

Summary

Extractive summarizers usually work in two separate stages. First they assign each sentence an importance score, then they apply a heuristic such as MMR or ILP to pick sentences for the summary. Zhou et al. merge both stages in NeuSum, an end-to-end neural model with no handcrafted features.

A hierarchical encoder builds the sentence representations. A BiGRU runs over the words of each sentence, and a second BiGRU runs over the sequence of sentences. A GRU-based extractor then builds the summary one sentence at a time. At each step it takes in the most recently chosen sentence, updates its state and rescores the sentences still available. The model is trained to predict how much each candidate would raise the ROUGE F1 of the summary built so far.

On the non-anonymized CNN/Daily Mail dataset, with three sentences extracted per document, NeuSum beats unsupervised baselines and neural extractors, including NN-SE. This matters because conditioning on earlier choices reduces redundancy and makes the model rely less on sentence position.

Key points

  • NeuSum scores 41.59 ROUGE-1, 19.01 ROUGE-2 and 37.98 ROUGE-L F1 on the CNN/Daily Mail test set. Its ROUGE-2 is 1.31 points above LEAD3, and it is significantly better than NN-SE on all three metrics.
  • In human evaluation, three volunteers ranked outputs for 50 sampled documents. NeuSum beat NN-SE on informativeness, redundancy and overall quality, with the largest gain in redundancy.
  • Selection precision by step:
    • At the first step, NeuSum reaches 40.38% and NN-SE 39.18%.
    • At the second step, NeuSum reaches 31.52% and NN-SE 28.28%.
    • Precision falls at later steps, which the authors attribute to error propagation and to the remaining choices being harder.
  • On sentence position, 80.91% of NN-SE's selections come from the first three sentences, compared with 58.64% for NeuSum. NeuSum's distribution is closer to the oracle's.
  • The architecture combines a hierarchical BiGRU document encoder with an RNN sentence extractor. The extractor starts from a zero vector, which stands for the empty summary.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

102 papers of our library cite it (of the 140884 whose bibliographies we read); 10 of them have a page here, 92 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-1061

@inproceedings{zhou2018neural,
  title = {{Neural Document Summarization by Jointly Learning to Score and Select Sentences}},
  author = {Zhou, Qingyu and Yang, Nan and Wei, Furu and Huang, Shaohan and Zhou, Ming and Zhao, Tiejun},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1061},
  url = {https://doi.org/10.18653/v1/p18-1061}
}

References

32 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ziqiang Cao, Furu Wei, Li Dong, Sujian Li, and Ming Zhou. 2015a. Ranking with recursive neural networks and its application to multi-document summarization. In AAAI, pages 2153–2159. find
  2. Ziqiang Cao, Furu Wei, Sujian Li, Wenjie Li, Ming Zhou, and WANG Houfeng. 2015b. Learning summary prior representation for extractive summarization. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Langua… find
  3. Jaime Carbonell and Jade Goldstein. 1998. The use of mmr, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, pages 335–336. ACM. find
  4. Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by extracting sentences and words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 484–494, Berlin, Germany. Association for Computational Linguistics. \#\# Conclusio… find our card
  5. Conventional approaches to extractive document summarization contain two separated steps: sentence scoring and sentence selection. In this paper, we present a novel neural network framework for extractive document summarization by jointly learning to score and select sentences to address this issue.… find
  6. John M Conroy and Dianne P O’leary. 2001. Text summarization via hidden markov models. In Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, pages 406–407. ACM. find
  7. Günes Erkan and Dragomir R Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of Artificial Intelligence Research, 22:457–479. find
  8. Dan Gillick and Benoit Favre. 2009. A scalable global model for summarization. In Proceedings of the Workshop on Integer Linear Programming for Natural Langauge Processing, pages 10–18. Association for Computational Linguistics. \#\# Acknowledgments We thank three anonymous reviewers for their helpful… find our card
  9. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Aistats, volume 9, pages 249–256. find
  10. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1693– 1701. find our card
  11. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. find our card
  12. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. ICML (3), 28:1310–1318. find
  13. Eduard Hovy and Chin-Yew Lin. 1998. Automated text summarization and the summarist system. In Proceedings of a workshop on held at Baltimore, Maryland: October 13-15, 1998, pages 197–214. Association for Computational Linguistics. find
  14. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. find
  15. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532– 1543. find
  16. Hakan Inan, Khashayar Khosravi, and Richard Socher. 2017. Tying word vectors and word classifiers: A loss framework for language modeling. In Proceedings of 5th International Conference for Learning Representations. find
  17. Radim Řehůřek and Petr Sojka. 2010. Software Framework for Topic Modelling with Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, pages 45–50, Valletta, Malta. ELRA. find
  18. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of 3rd International Conference for Learning Representations, San Diego. find
  19. Julian Kupiec, Jan Pedersen, and Francine Chen. 1995. A trainable document summarizer. In Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval, pages 68–73. ACM. find
  20. Pengjie Ren, Zhumin Chen, Zhaochun Ren, Furu Wei, Jun Ma, and Maarten de Rijke. 2017. Leveraging contextual sentence relations for extractive summarization using a neural attention model. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retriev… find
  21. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out: Proceedings of the ACL-04 workshop, volume 8. Barcelona, Spain. find
  22. Pengjie Ren, Furu Wei, CHEN Zhumin, MA Jun, and Ming Zhou. 2016. A redundancy-aware sentence regression framework for extractive summarization. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 33–43. find
  23. Hui Lin and Jeff Bilmes. 2011. A class of submodular functions for document summarization. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 510–520. Association for Computational Linguistics. find our card
  24. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073– 1083, Vancouver, Canada. Association for Compu… find
  25. Hans Peter Luhn. 1958. The automatic creation of literature abstracts. IBM Journal of research and development, 2(2):159–165. find
  26. Ryan McDonald. 2007. A study of global inference algorithms in multi-document summarization. In European Conference on Information Retrieval, pages 557–564. Springer. find
  27. Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958. find
  28. Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing. find
  29. Xiaojun Wan and Jianwu Yang. 2006. Improved affinity graph based multi-document summarization. In Proceedings of the human language technology conference of the NAACL, Companion volume: Short papers, pages 181–184. Association for Computational Linguistics. find
  30. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In AAAI, pages 3075–3081. find our card
  31. Ramesh Nallapati, Bowen Zhou, Ça glar Gulçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning. find
  32. Ani Nenkova and Kathleen McKeown. 2011. Automatic summarization. Foundations and Trends R in Information Retrieval, 5(2–3):103–233. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2018-neural-document-summarization-jointly-learning/