Elamur Research⌕
☰
home / papers / 2018 / Ranking Sentences for Extractive Summarization with Reinforcement Learning

Ranking Sentences for Extractive Summarization with Reinforcement Learning

Summary

The paper addresses single-document extractive summarization. The authors argue that standard training, which uses cross-entropy against sentence-level labels, fits the task poorly. Those labels are derived heuristically from abstractive reference summaries. The objective also never sees ROUGE, which scores the summary as a whole. As a result, such models tend to produce long, redundant extracts. The authors instead treat extraction as sentence ranking. Their model, REFRESH, has three parts: a convolutional sentence encoder, an LSTM document encoder, and an LSTM sentence extractor that scores each sentence's suitability for the summary. During training, the top-scored sentences form a candidate summary, which is compared with the gold highlights. The resulting ROUGE reward updates the network through REINFORCE, combined with the maximum-likelihood signal. Sentences that often appear in high-scoring extracts therefore get pushed up the ranking. On the CNN and DailyMail highlights datasets, REFRESH beats strong extractive and abstractive baselines on ROUGE and in two human studies. This shows that a globally trained extractor can outperform contemporary abstractive models.

Key points

  • Data: news articles with highlight summaries from CNN and DailyMail, as introduced by Hermann et al. (2015). Code and data are released at github.com/shashiongithub/Refresh.
  • Architecture:
    • Sentence encoder: temporal narrow convolution with several kernel widths, followed by max-pooling over time.
    • Document encoder: LSTM that reads sentences in reverse order, so the lead sentences are emphasized.
    • Extractor: LSTM with a softmax layer; its confidence scores rank the sentences.
  • Labeling: the authors contrast two ways of deriving labels.
    • Individual oracle: each sentence is labeled separately by its match to the gold summary, which tends to yield too many positives.
    • Collective oracle: the sentence subset with the highest joint ROUGE is labeled positive.
  • Question-answering evaluation: readers answered 66.34% of questions correctly from REFRESH summaries, versus 36.33% for LEAD and 28.73% for See et al. (2017).
  • Ranking evaluation: REFRESH was ranked second, and its difference from GOLD was not statistically significant (one-way ANOVA with post-hoc Tukey HSD, p < 0.01).

Interesting ideas

  • I1. Reward the assembled output, not each label. The model is trained by scoring the summary actually built from its top-ranked sentences against the metric used at test time. This transfers to any top-k selection task, such as retrieval or keyphrase picking, whose evaluation measure is non-differentiable and set-level.
  • I2. Measure informativeness with questions. Instead of relying only on n-gram overlap, the authors check how many questions about the article readers can answer from a summary. This makes differences between systems far sharper and better reflects what users actually need.
  • I3. Separate preference from content. Human rankings placed See et al. above LEAD, yet LEAD carried more answerable content. The authors attribute this to readers favoring shorter texts (61.6 vs 105.7 words on average). It is a useful warning to control for length, or measure it, in any human evaluation.
  • I4. Build oracle labels for sets, not single items. Selecting the jointly best-scoring subset gives cleaner supervision than labeling sentences one by one. The same idea applies whenever silver labels must be extracted from abstractive references.
  • I5. Feed the document in reverse order. Reading sentences backwards into the encoder keeps the opening of the document salient in its representation. This is a cheap trick for any genre where early content matters most.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

180 papers of our library cite it (of the 140884 whose bibliographies we read); 15 of them have a page here, 165 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n18-1158

@inproceedings{narayan2018ranking,
  title = {{Ranking Sentences for Extractive Summarization with Reinforcement Learning}},
  author = {Narayan, Shashi and Cohen, Shay B. and Lapata, Mirella},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-1158},
  url = {https://doi.org/10.18653/v1/n18-1158}
}

References

71 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martı́n Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenbe… find
  2. Miguel B. Almeida and André F. T. Martins. 2013. Fast and robust compressive summarization with dual decomposition and multi-task learning. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics. Sofia, Bulgaria, pages 196–206. find
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the 3rd International Conference on Learning Representations. San Diego, California, USA. find our card
  4. Regina Barzilay and Michael Elhadad. 1997. Using lexical chains for text summarization. In Proceedings of the ACL Workshop on Intelligent Scalable Text Summarization. Madrid, Spain, pages 10–17. find our card
  5. Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems 28. pages 1171–1179. find
  6. Taylor Berg-Kirkpatrick, Dan Gillick, and Dan Klein. 2011. Jointly learning to extract and compress. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Portland, Oregon, USA, pages 481–490. find
  7. Ziqiang Cao, Chengyao Chen, Wenjie Li, Sujian Li, Furu Wei, and Ming Zhou. 2016. TGSum: Build tweet guided multi-document summarization dataset. In Proceedings of the 30th AAAI Conference on Artificial Intelligence. Phoenix, Arizona USA, pages 2906–2912. find
  8. Jaime Carbonell and Jade Goldstein. 1998. The use of MMR, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. Melbourne, Australia, pages 335– 336. find
  9. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2013. One billion word benchmark for measuring progress in statistical language modeling. Technical report, Google. find our card
  10. Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, and Hui Jiang. 2016. Distraction-based neural networks for modeling documents. In Proceedings of the 25th International Joint Conference on Artificial Intelligence. New York, USA, pages 2754–2760. find
  11. Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by extracting sentences and words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Berlin, Germany, pages 484–494. find our card
  12. James Clarke and Mirella Lapata. 2010. Discourse constraints for document compression. Computational Linguistics 36(3):411–441. find
  13. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research 12:2493–2537. find our card
  14. Günes Erkan and Dragomir R. Radev. 2004. LexRank: Graph-based lexical centrality as salience in text summarization. Journal of Artificial Intelligence Research 22(1):457–479. find
  15. Elena Filatova and Vasileios Hatzivassiloglou. 2004. Event-based extractive summarization. In Proceedings of ACL Workshop on Text Summarization Branches Out. Barcelona, Spain, pages 104–111. find
  16. Katja Filippova, Enrique Alfonseca, Carlos A. Colmenares, Lukasz Kaiser, and Oriol Vinyals. 2015. Sentence compression by deletion with LSTMs. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Lisbon, Portugal, pages 360–368. find our card
  17. Sebastian Henß, Margot Mieskes, and Iryna Gurevych. 2015. A reinforcement learning approach for adaptive single- and multi-document summarization. In Proceedings of International Conference of the German Society for Computational Linguistics and Language Technology. Duisburg-Essen, Germany, pages 3–… find
  18. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems 28. pages 1693– 1701. find our card
  19. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation 9(8):1735–1780. find
  20. Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics. Baltimore, Maryland, pages 655– 665. find our card
  21. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Doha, Qatar, pages 1746–1751. find our card
  22. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M. Rush. 2016. Character-aware neural language models. In Proceedings of the 30th AAAI Conference on Artificial Intelligence. Phoenix, Arizona USA, pages 2741–2749. find our card
  23. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations. San Diego, California, USA. find
  24. Mikael Kågebäck, Olof Mogren, Nina Tahmasebi, and Devdatt Dubhashi. 2014. Extractive summarization using continuous vector space models. In Proceedings of the Workshop on Continuous Vector Space Models and their Compositionality. Gothenburg, Sweden, pages 31–39. find
  25. Julian Kupiec, Jan Pedersen, and Francine Chen. 1995. A trainable document summarizer. In Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. Seattle, Washington, USA, pages 406–407. find
  26. Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2015. Molding CNNs for text: Non-linear, non-consecutive convolutions. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Lisbon, Portugal, pages 1565–1575. find
  27. Jiwei Li, Thang Luong, and Dan Jurafsky. 2015. A hierarchical neural autoencoder for paragraphs and documents. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing. Beijing, China, pages 1… find
  28. Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016. Deep reinforcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Austin, Texas, pages 1192–1202. find our card
  29. Liangda Li, Ke Zhou, Gui-Rong Xue, Hongyuan Zha, and Yong Yu. 2009. Enhancing diversity, coverage and balance for summarization through structure learning. In Proceedings of the 18th international Conference on World Wide Web. Madrid, Spain, pages 71–80. find
  30. Chin-Yew Lin and Eduard Hovy. 2003. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics. Edmonton, Canada, pages 71–78. find our card
  31. Inderjeet Mani. 2001. Automatic Summarization. Natural language processing. John Benjamins Publishing Company. find
  32. Inderjeet Mani, Gary Klein, David House, Lynette Hirschman, Therese Firmin, and Beth Sundheim. 2002. SUMMAC: A text summarization evaluation. Natural Language Engineering 8(1):4368. find
  33. William C. Mann and Sandra A. Thompson. 1988. Rhetorical Structure Theory: Toward a functional theory of text organization. Text 8(3):243–281. find our card
  34. Rada Mihalcea and Paul Tarau. 2004. TextRank: Bringing order into texts. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing. Barcelona, Spain, pages 404– 411. find
  35. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26. pages 3111–3119. find
  36. Diego Mollá-Aliod. 2017. Towards the use of deep reinforcement learning with global policy for query-based extractive summarisation. In Proceedings of the Australasian Language Technology Association Workshop 2017. Brisbane, Australia, pages 103– 107. find
  37. Andrew H. Morris, George M. Kasper, and Dennis A. Adams. 1992. The effects and limitations of automated text condensing on reading comprehension performance. Information Systems Research 3(1):17–35. find
  38. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. SummaRuNNer: A recurrent neural network based sequence model for extractive summarization of documents. In Proceedings of the 31st AAAI Conference on Artificial Intelligence. San Francisco, California USA, pages 3075–3081. find our card
  39. Ramesh Nallapati, Bowen Zhou, Cı́cero Nogueira dos Santos, Çaglar Gülçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning. Berlin, Germany, pages 280– 290. find
  40. Shashi Narayan, Nikos Papasarantopoulos, Shay B. Cohen, and Mirella Lapata. 2017. Neural extractive summarization with side information. CoRR abs/1704.04530. — [arXiv:1704.04530](https://arxiv.org/abs/1704.04530) find
  41. Ani Nenkova and Kathleen McKeown. 2011. Automatic summarization. Foundations and Trends in Information Retrieval 5(2–3):103–233. find
  42. Ani Nenkova, Lucy Vanderwende, and Kathleen McKeown. 2006. A compositional context sensitive multi-document summarizer: Exploring the factors that influence summarization. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. pages… find
  43. Rodrigo Nogueira and Kyunghyun Cho. 2017. Task-oriented query reformulation with reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Copenhagen, Denmark, pages 585–594. find
  44. Daraksha Parveen, Hans-Martin Ramsl, and Michael Strube. 2015. Topical coherence for graph-based extractive summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Lisbon, Portugal, pages 1949– 1954. find
  45. Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. CoRR abs/1705.04304. find our card
  46. Dragomir Radev, Timothy Allison, Sasha Blair-Goldensohn, John Blitzer, Arda Çelebi, Stanko Dimitrov, Elliott Drabek, Ali Hakim, Wai Lam, Danyu Liu, Jahna Otterbacher, Hong Qi, Horacio Saggion, Simone Teufel, Michael Topper, Adam Winkel, and Zhu Zhang. 2004. MEAD — A platform for multidocument multi… find our card
  47. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2015. Sequence level training with recurrent neural networks. CoRR abs/1511.06732. find our card
  48. Cody Rioux, Sadid A. Hasan, and Yllias Chali. 2014. Fear the REAPER: A system for automatic multi-document summarization with reinforcement learning. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Doha, Qatar, pages 681–690. find
  49. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Lisbon, Portugal, pages 379–389. find
  50. Seonggi Ryang and Takeshi Abekawa. 2012. Framework of automatic text summarization using reinforcement learning. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. Jeju Island, Korea, pages 256–265. find
  51. Frank Schilder and Ravikumar Kondadadi. 2008. FastSum: Fast and accurate query-based multi-document summarization. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics and HLT: Short Papers. Columbus, Ohio, USA, pages 205–208. find
  52. Natalie Schluter. 2017. The limits of automatic summarisation according to rouge. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Short Papers. Valencia, Spain, pages 41–45. find
  53. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. Vancouver, Canada, pages 1073–1083. find
  54. Dou Shen, Jian-Tao Sun, Hua Li, Qiang Yang, and Zheng Chen. 2007. Document summarization using conditional random fields. In Proceedings of the 20th International Joint Conference on Artifical intelligence. Hyderabad, India, pages 2862–2867. find
  55. Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum risk training for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Berlin, Germany, pages 1683–1692. find our card
  56. Karen Spärck Jones. 2007. Automatic summarising: The state of the art. Information Processing and Management 43(6):1449–1481. find
  57. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27. pages 3104–3112. find
  58. Richard S. Sutton and Andrew G. Barto. 1998. Reinforcement Learning : An Introduction. MIT Press. find
  59. Krysta Marie Svore, Lucy Vanderwende, and Christopher J. C. Burges. 2007. Enhancing single-document summarization by combining ranknet and third-party sources. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning.… find
  60. Jiwei Tan and Xiaojun Wan. 2017. Abstractive document summarization with a graph-based attentional neural model. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. Vancouver, Canada, pages 1171–1181. find our card
  61. Simone Teufel and Marc Moens. 1997. Sentence extraction as a classification task. In Proceedings of ACL Workshop on Intelligent and Scalable Text Summarization. Madrid, Spain, pages 58–65. find
  62. Xiaojun Wan. 2010. Towards a unified approach to simultaneous single-document and multi-document summarizations. In Proceedings of the 23rd International Conference on Computational Linguistics. Beijing, China, pages 1137–1145. find
  63. Ronald J. Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning 8(3-4):229–256. find
  64. Kristian Woodsend and Mirella Lapata. 2010. Automatic generation of story highlights. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. Uppsala, Sweden, pages 565–574. find
  65. Kristian Woodsend and Mirella Lapata. 2012. Multiple aspect summarization using integer linear programming. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. Jeju Island, Korea, pages 233–243. find
  66. Kelvin Xu, Jimmy Lei Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the 32nd International Conference on Machine Learning. Lille, France,… find our card
  67. Michihiro Yasunaga, Rui Zhang, Kshitijh Meelu, Ayush Pareek, Krishnan Srinivasan, and Dragomir Radev. 2017. Graph-based neural multi-document summarization. In Proceedings of the 21st Conference on Computational Natural Language Learning. Vancouver, Canada, pages 452–462. find our card
  68. Wen-tau Yih, Joshua Goodman, Lucy Vanderwende, and Hisami Suzuki. 2007. Multi-document summarization by maximizing informative content-words. In Proceedings of the 20th International Joint Conference on Artifical intelligence. Hyderabad, India, pages 1776–1782. find
  69. Wenpeng Yin and Yulong Pei. 2015. Optimizing sentence modeling and selection for document summarization. In Proceedings of the 24th International Joint Conference on Artificial Intelligence. Buenos Aires, Argentina, pages 1383–1389. find
  70. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems 28. pages 649–657. find our card
  71. Xingxing Zhang and Mirella Lapata. 2017. Sentence simplification with deep reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Copenhagen, Denmark, pages 595–605. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/narayan-2018-ranking-sentences-extractive-summarization-reinforcement/