Elamur Research⌕
☰
home / papers / 2015 / A Neural Attention Model for Abstractive Sentence Summarization

A Neural Attention Model for Abstractive Sentence Summarization

Paper · 2015 · Alexander M. Rush, Sumit Chopra and Jason Weston

Summary

The authors address abstractive sentence summarization. The task is to produce a short version that may contain new words, paraphrases, and generalizations, rather than only fragments cut from the original. Previous approaches to this task relied on linguistic rules or syntactic transformations. Instead, the authors build a fully data-driven model. It combines a feed-forward neural language model (NNLM) in the spirit of Bengio et al. with an encoder of the input sentence, and both components are trained jointly. The best encoder uses an attention mechanism similar to that of neural machine translation. At each step, it softly aligns the output generated so far with the relevant parts of the input. The system is called ABS. The model was trained to generate headlines on "first sentence of an article — headline" pairs from the Gigaword corpus. Beam search is used during generation. The extended version, ABS+, adds features that help copy words from the input. On DUC-2004, the model outperformed all strong baselines, including phrase-based statistical MT trained on the same data. The work showed that attention-based neural generation can be applied to summarization without hand-crafted linguistic constraints.

Key points

  • Approximately 4 million articles from Gigaword (Graff et al., 2003) were used for training. Evaluation was carried out on the DUC-2004 shared task, and development on DUC-2003.
  • The authors compared three encoders: bag-of-words, convolutional (TDNN), and attention-based. The language model sees a window of C=5 previous words.
  • On DUC-2003, the best variant (ABS+ with beam search in abstractive mode) achieved ROUGE-1 28.48, ROUGE-2 8.91, and ROUGE-L 23.97. For comparison: the BoW encoder reached only 22.15 ROUGE-1, greedy decoding 26.67, and the purely extractive mode 27.89.
  • The largest gains came from two factors: replacing BoW with attention and switching to beam search.
  • The limitations are as follows: the results are still far from human level, and the model sometimes reorders words ungrammatically or confuses the subject. As next steps, the authors name improved grammaticality and scaling up to summaries of entire paragraphs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MetaMenlo Park · United States183 works · 228,558 citations🎓Harvard UniversityCambridge · United States112 works · 42,023 citations🎓Imperial College LondonLondon · United Kingdom48 works · 12,181 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Summarization · 232 works

💡 Concepts 2Key ideas and methods explained on our concept pages.Natural language processing · 131 worksText summarization · 49 works

Cite

Cite as Alexander M. Rush, Sumit Chopra and Jason Weston. 2015. A Neural Attention Model for Abstractive Sentence Summarization. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d15-1044 · arXiv 1509.00685

@inproceedings{rush2015neural,
  title = {{A Neural Attention Model for Abstractive Sentence Summarization}},
  author = {Alexander M. Rush, Sumit Chopra and Jason Weston},
  booktitle = {Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1044},
  eprint = {1509.00685},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d15-1044}
}

References

27 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
  2. Michele Banko, Vibhu O Mittal, and Michael J Witbrock. 2000. Headline generation based on statistical translation. In Proceedings of the 38th Annual Meeting on Association for Computational Linguistics, pages 318–325. Association for Computational Linguistics. find
  3. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. The Journal of Machine Learning Research, 3:1137–1155.
  4. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078.
  5. James Clarke and Mirella Lapata. 2008. Global inference for sentence compression: An integer linear programming approach. Journal of Artificial Intelligence Research, pages 399–429. find
  6. Trevor Cohn and Mirella Lapata. 2008. Sentence compression beyond word deletion. In Proceedings of the 22nd International Conference on Computational Linguistics-Volume 1, pages 137–144. Association for Computational Linguistics.
  7. Hal Daumé III and Daniel Marcu. 2002. A noisy-channel model for document compression. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pages 449–456. Association for Computational Linguistics. find
  8. Bonnie Dorr, David Zajic, and Richard Schwartz. 2003. Hedge trimmer: A parse-and-trim approach to headline generation. In Proceedings of the HLT-NAACL 03 on Text summarization workshop-Volume 5, pages 1–8. Association for Computational Linguistics.
  9. Katja Filippova and Yasemin Altun. 2013. Overcoming the lack of parallel data in sentence compression. In EMNLP, pages 1481–1491.
  10. David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2003. English gigaword. Linguistic Data Consortium, Philadelphia. find
  11. Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580. find
  12. Hongyan Jing. 2002. Using hidden markov modeling to decompose human-written summaries. Computational linguistics, 28(4):527–543.
  13. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In EMNLP, pages 1700–1709.
  14. Kevin Knight and Daniel Marcu. 2002. Summarization beyond sentence extraction: A probabilistic approach to sentence compression. Artificial Intelligence, 139(1):91–107. find
  15. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the ACL on interacti… find
  16. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pages 74–81.
  17. Thang Luong, Ilya Sutskever, Quoc V Le, Oriol Vinyals, and Wojciech Zaremba. 2014. Addressing the rare word problem in neural machine translation. arXiv preprint arXiv:1410.8206.
  18. Christopher D Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to information retrieval, volume 1. Cambridge university press Cambridge. find
  19. Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J Bethard, and David Mc-Closky. 2014. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60.
  20. Courtney Napoles, Matthew Gormley, and Benjamin Van Durme. 2012. Annotated gigaword. In Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction, pages 95–100. Association for Computational Linguistics.
  21. Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics-Volume 1, pages 160–167. Association for Computational Linguistics. find
  22. Paul Over, Hoa Dang, and Donna Harman. 2007. Duc in context. Information Processing & Management, 43(6):1506–1520. find
  23. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pages 3104–3112.
  24. Kristian Woodsend, Yansong Feng, and Mirella Lapata. 2010. Generation with quasi-synchronous grammar. In Proceedings of the 2010 conference on empirical methods in natural language processing, pages 513– 523. Association for Computational Linguistics. find
  25. Sander Wubben, Antal Van Den Bosch, and Emiel Krahmer. 2012. Sentence simplification by monolingual machine translation. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 1015–1024. Association for Computational Linguistics.
  26. Omar Zaidan. 2009. Z-mert: A fully configurable open source tool for minimum error rate training of machine translation systems. The Prague Bulletin of Mathematical Linguistics, 91:79–88. find
  27. David Zajic, Bonnie Dorr, and Richard Schwartz. 2004. Bbn/umd at duc-2004: Topiary. In Proceedings of the HLT-NAACL 2004 Document Understanding Workshop, Boston, pages 112–119. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rush-2015-neural-attention-model-abstractive-sentence/