Elamur Research⌕
☰
home / papers / 2016 / Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond

Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond

Paper · 2016 · Ramesh Nallapati, Bowen Zhou, Cicero dos Santos et al.

Summary

The paper addresses abstractive summarization. The model must itself formulate a headline or several sentences conveying the gist of an article, rather than simply selecting ready-made fragments from it. As a basis, the authors took an attentional neural machine translation model: a bidirectional GRU encoder and a unidirectional GRU decoder. To speed up training, they used the LVT technique. It restricts the decoder vocabulary to the words in the documents of the current batch, supplemented with the most frequent words. Even this baseline system outperformed the best results of the time on two English-language corpora. The authors then proposed extensions addressing weaknesses of the baseline model:

  • an encoder with additional POS, NER, TF and IDF features, to better identify keywords;
  • a switch between generating a word and pointing to a position in the source, for rare and unseen words;
  • hierarchical attention at the word and sentence levels, for long documents.

Most of these extensions improved the results slightly further. A separate contribution is the CNN/Daily Mail dataset with multi-sentence summaries, together with baseline results on it for future research.

Key points

  • Data and evaluation: experiments were conducted on three datasets: Gigaword, DUC-2004 and the new CNN/Daily Mail. Quality was measured with the Rouge metric. The strongest prior approach on DUC-2004 was the TOPIARY system.
  • Switching generator-pointer: at each step, a sigmoid switch decides whether to generate a word or copy it from the source. During training, explicit pointers are provided for out-of-vocabulary words; if a word occurs several times, the first occurrence is used. The model accurately copies named entities and multi-word phrases, but the overall quality gain is marginal.
  • Setup for CNN/Daily Mail: documents are limited to 800 words and summaries to 100. 100-dimensional word2vec vectors trained on the same dataset were used, with a hidden state of size 200. Training ran on a single Tesla K-40; the flat models (words-lvt2k, words-lvt2k-ptr) took less than 5 hours per epoch.
  • Qualitative analysis: one of the best models is feats-lvt2k-2sent. Even when its summaries differ from the reference ones, they are mostly meaningful, but the model sometimes misinterprets the content. On CNN/Daily Mail it often repeats the same sentences; the authors name intra-attention as a possible solution.
  • Difference from translation: in summarization, the output is short and nearly independent of the document length. Compression is lossy, and the alignment between source and output words is much weaker than in machine translation.

The provided text lacks the middle part of the paper, so the summary contains no specific Rouge values.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢IBMArmonk · United States139 works · 70,691 citations🎓Université de MontréalMontreal · Canada68 works · 79,382 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Summarization · 232 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Text summarization · 49 works

Cite

Cite as Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gu\ensuremath\dotlçehre and Bing Xiang. 2016. Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond. Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/k16-1028 · arXiv 1602.06023

@inproceedings{nallapati2016abstractive,
  title = {{Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond}},
  author = {Ramesh Nallapati, Bowen Zhou, Cicero {dos Santos}, Çağlar Guensuremathdotlçehre and Bing Xiang},
  booktitle = {Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning},
  year = {2016},
  doi = {10.18653/v1/k16-1028},
  eprint = {1602.06023},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/k16-1028}
}

References

31 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Bengio. 2016. Pointing the unknown words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics.
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473.
  3. Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. CoRR, abs/1506.03340.
  4. Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Philemon Brakel, and Yoshua Bengio. 2015. End-to-end attention-based large vocabulary speech recognition. CoRR, abs/1508.04395. find
  5. Michele Banko, Vibhu O. Mittal, and Michael J Witbrock. 2000. Headline generation based on statistical translation. In Proceedings of the 38th Annual Meeting on Association for Computational Linguistics, 22:318–325. find
  6. Baotian Hu, Qingcai Chen, and Fangze Zhu. 2015. Lc-sts: A large scale chinese short text summarization dataset. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1967–1972, Lisbon, Portugal, September. Association for Computational Linguistics. find
  7. Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by extracting sentences and words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics.
  8. Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2014. On using very large target vocabulary for neural machine translation. CoRR, abs/1412.2007.
  9. Jianpeng Cheng, Li Dong, and Mirella Lapata. 2016. Long short-term memory-networks for machine reading. CoRR, abs/1601.06733.
  10. B. Favre K. Riedhammer and D. Hakkani-Tur. 2010. Long story short âĂŞ global unsupervised models for keyphrase based meeting summarization. In Speech Communication, pages 801–815. find
  11. Sumit Chopra, Michael Auli, and Alexander M. Rush. 2016. Abstractive sentence summarization with attentive recurrent neural networks. In HLT-NAACL.
  12. Jiwei Li, Minh-Thang Luong, and Dan Jurafsky. 2015. A hierarchical neural autoencoder for paragraphs and documents. CoRR, abs/1506.01057. find
  13. Junyoung Chung, Çaglar Gülçehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR, abs/1412.3555. find
  14. M. Litvak and M. Last. 2008. Graph-based keyword extraction for single-document summarization. In Coling 2008, pages 17–24. find
  15. Trevor Cohn and Mirella Lapata. 2008. Sentence compression beyond word deletion. In Proceedings of the 22Nd International Conference on Computational Linguistics - Volume 1, pages 137–144.
  16. Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. 2015. Addressing the rare word problem in neural machine translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Languag…
  17. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel P. Kuksa. 2011. Natural language processing (almost) from scratch. CoRR, abs/1103.0398.
  18. Carlos A. Colmenares, Marina Litvak, Amin Mantrach, and Fabrizio Silvestri. 2015. Heads: Headline generation as sequence prediction using an abstract feature-rich space. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua… find
  19. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. CoRR, abs/1310.4546.
  20. Joel Larocca Neto, Alex Alves Freitas, and Celso A. A. Kaestner. 2002. Automatic text summarization using a machine learning approach. In Proceedings of the 16th Brazilian Symposium on Artificial Intelligence: Advances in Artificial Intelligence, pages 205–215. find
  21. G. Erkan and D. R. Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of Artificial Intelligence Research, 22:457–479. find
  22. David Martins de Matos JoÃčo P. Neto Anatole Gershman Jaime Carbonell Ricardo Ribeiro, Luà s Marujo. 2013. Self reinforcement for important passage retrieval. In 36th international ACM SIGIR conference on Research and development in information retrieval, pages 845–848. find
  23. Katja Filippova and Yasemin Altun. 2013. Overcoming the lack of parallel data in sentence compression. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1481–1491.
  24. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. CoRR, abs/1509.00685.
  25. Subhashini Venugopalan, Marcus Rohrbach, Jeff Donahue, Raymond J. Mooney, Trevor Darrell, and Kate Saenko. 2015. Sequence to sequence - video to text. CoRR, abs/1505.00487. find
  26. O. Vinyals, M. Fortunato, and N. Jaitly. 2015. Pointer Networks. ArXiv e-prints, June. find
  27. Kam-Fai Wong, Mingli Wu, and Wenjie Li. 2008a. Extractive summarization using supervised and semi-supervised learning. In Proceedings of the 22Nd International Conference on Computational Linguistics - Volume 1, pages 985–992. find
  28. Kam-Fai Wong, Mingli Wu, and Wenjie Li. 2008b. Extractive summarization using supervised and semi-supervised learning. In Proceedings of the 22nd Annual Meeting of the Association for Computational Linguistics, pages 985–992. find
  29. Kristian Woodsend, Yansong Feng, and Mirella Lapata. 2010. Title generation with quasi-synchronous grammar. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, EMNLP ’10, pages 513–523, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  30. David Zajic, Bonnie J. Dorr, and Richard Schwartz. 2004. Bbn/umd at duc-2004: Topiary. In Proceedings of the North American Chapter of the Association for Computational Linguistics Workshop on Document Understanding, pages 112–119. find
  31. Matthew D. Zeiler. 2012. ADADELTA: an adaptive learning rate method. CoRR, abs/1212.5701.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nallapati-2016-abstractive-text-summarization-sequence-sequence/