Elamur Research
☰
home / papers / 2016 / Abstractive Sentence Summarization with Attentive Recurrent Neural Networks

Abstractive Sentence Summarization with Attentive Recurrent Neural Networks

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N16-1012.pdf

Summary

The paper addresses abstractive sentence summarization: producing a shorter rewrite of one sentence that keeps its meaning, rather than cutting and stitching fragments of the source. The authors propose RAS (Recurrent Attentive Summarizer), which extends the feed-forward ABS model of Rush et al. (2015). In RAS, a recurrent decoder (an Elman RNN or an LSTM) generates the summary word by word. At each step it is conditioned on a context vector from a convolutional attention encoder. That encoder adds learned position embeddings to word embeddings and applies a convolution of width 5. The resulting vectors set the attention weights over the input words. The model is trained end-to-end on first-sentence/headline pairs from annotated Gigaword and decoded with beam search. It uses no hand-crafted extractive features. RAS clearly beats ABS and ABS+ on the Gigaword test set and is competitive on DUC-2004. Its summaries are also more abstractive. The work matters because it shows that a light recurrent attention model can match much heavier NMT systems on this task.

Key points

  • Data and evaluation: RAS is trained on annotated Gigaword using the preprocessing and splits of Rush et al. (2015). It is tested on 2000 held-out Gigaword pairs (ROUGE F1) and 500 DUC-2004 pairs (recall ROUGE).
  • Gigaword results: with beam size 10, RAS-Elman scores 33.78 / 15.97 / 31.15 on ROUGE-1 / ROUGE-2 / ROUGE-L. ABS+ scores 29.76 / 11.88 / 26.96. RAS-Elman also slightly exceeds a re-implemented Luong-NMT (33.10 / 14.45 / 30.71).
  • Perplexity: validation perplexity is 18.9 for RAS-Elman and 20.3 for RAS-LSTM, versus 27.1 for ABS. The authors attribute the LSTM's weaker result to over-fitting.
  • DUC-2004 results: gains over ABS+ are small and do not hold on every metric. RAS-Elman reaches 28.97 ROUGE-1 and 24.06 ROUGE-L, against 28.18 and 23.81 for ABS+, but trails ABS+ on ROUGE-2 (8.26 vs. 8.49).
  • Abstractiveness and ablation: RAS copies 74% of its summary words from the input, compared with 92% for ABS+. Replacing the RAS encoder with the ABS encoder, which lacks position features, raises perplexity from 29 to 38 on a Gigaword subset. The code was released as facebook/namas (Torch).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

204 papers of our library cite it (of the 140673 whose bibliographies we read); 17 of them have a page here, 187 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n16-1012

@inproceedings{chopra2016abstractive,
  title = {{Abstractive Sentence Summarization with Attentive Recurrent Neural Networks}},
  author = {Chopra, Sumit and Auli, Michael and Rush, Alexander M.},
  booktitle = {Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2016},
  doi = {10.18653/v1/n16-1012},
  url = {https://doi.org/10.18653/v1/n16-1012}
}

References

25 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473. find our card
  2. Michele Banko, Vibhu O Mittal, and Michael J Witbrock. 2000. Headline generation based on statistical translation. In Proceedings of the 38th Annual Meeting on Association for Computational Linguistics, pages 318–325. Association for Computational Linguistics. Kyunghyun Cho, Bart van Merrienboer, Ç… find
  3. James Clarke and Mirella Lapata. 2008. Global inference for sentence compression: An integer linear programming approach. Journal of Artificial Intelligence Research, pages 399–429. find
  4. Trevor Cohn and Mirella Lapata. 2008. Sentence compression beyond word deletion. In Proceedings of the 22nd International Conference on Computational Linguistics-Volume 1, pages 137–144. Association for Computational Linguistics. find
  5. Trevor Cohn and Mirella Lapata. 2013. An abstractive approach to sentence compression. ACM Transactions on Intelligent Systems and Technology (TIST’13), 4,3(41). find
  6. Jeffrey L. Elman. 1990. Finding structure in time. Cognitive Science, 14(2):179–211. find
  7. Katja Filippova and Yasemin Altun. 2013. Overcoming the lack of parallel data in sentence compression. In EMNLP, pages 1481–1491. find
  8. Katja Filippova, Enrique Alfonseca, Carlos A Col-menares, Lukasz Kaiser, and Oriol Vinyals. 2015. Sentence compression by deletion with lstms. In EMNLP. find
  9. Dimitrios Galanis and Ion Androutsopoulos. 2010. An extractive supervised two-stage method for sentence compression. In Proceedings of NAACL-HLT 2010. find
  10. David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2003. English gigaword. find
  11. Linguistic Data Consortium, Philadelphia. S. Hochreiter and J. Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. find
  12. H Jing. 2000. Sentence reduction for automatic text summarization. In ANLP-00, pages 703–711. find
  13. Kevin Knight and Daniel Marcu. 2002. Summarization beyond sentence extraction: A probabilistic approach to sentence compression. Artificial Intelligence, 139(1):91–107. find
  14. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pages 74–81. find
  15. Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421, Lisbon, Portugal, September. Association for Computational Li… find
  16. R McDonald. 2006. Discriminative sentence compression with soft syntactic evidence. In EACL-06, pages 297–304. find
  17. Ramesh Nallapati, Bing Xiang, and Zhou Bowen. 2016. Sequence-to-sequence rnns for text summarization. In http://arxiv.org/abs/1602.06023. ↗
  18. Courtney Napoles, Chris Callison-Burch, Juri Ganitkevitch, and Benjamin Van Durme. 2011. Paraphratic sentence compression with a character-based metric: Tightening without deletion. In Proceedings of the Workshop on Monolingual Text-To-Text Generation (MTTG’11). find
  19. Courtney Napoles, Matthew Gormley, and Benjamin Van Durme. 2012. Annotated gigaword. In Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction, pages 95–100. Association for Computational Linguistics. find
  20. Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics-Volume 1, pages 160–167. Association for Computational Linguistics. find
  21. Paul Over, Hoa Dang, and Donna Harman. 2007. Duc in context. Information Processing & Management, 43(6):1506–1520. find
  22. Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In EMNLP. find
  23. Kristian Woodsend, Yansong Feng, and Mirella Lapata. 2010. Generation with quasi-synchronous grammar. In Proceedings of the 2010 conference on empirical methods in natural language processing, pages 513– 523. Association for Computational Linguistics. find
  24. Sander Wubben, Antal Van Den Bosch, and Emiel Krahmer. 2012. Sentence simplification by monolingual machine translation. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 1015–1024. Association for Computational Linguistics. find our card
  25. David Zajic, Bonnie Dorr, and Richard Schwartz. 2004. Bbn/umd at duc-2004: Topiary. In Proceedings of the HLT-NAACL 2004 Document Understanding Workshop, Boston, pages 112–119. find