Elamur Research
☰
home / papers / 2018 / A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents

A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1804.05685

Summary

Most neural abstractive summarizers have been built and tested on short news stories of a few hundred words. They do poorly on much longer texts, because at every decoding step the attention has to pick useful content from thousands of tokens. Cohan et al. study this problem using scientific articles, which are long and follow a standard section structure. They make two contributions. The first is a pair of large corpora built from arXiv and PubMed, where the paper's abstract is the target summary. The second is a sequence-to-sequence model whose hierarchical encoder first encodes each section with a word-level bidirectional LSTM and then encodes the whole document with a section-level LSTM. The decoder weights its word-level attention by attention over sections, and it also uses a coverage mechanism and a pointer-generator copy mechanism. On both corpora, the model beats strong abstractive baselines on ROUGE by a statistically significant margin and is competitive with extractive systems. This work set an early benchmark and released datasets for summarizing long, structured documents.

Key points

  • New corpora: arXiv has 215K papers (about 4,938 words on average, with ~220-word abstracts), and PubMed has 133K papers (about 3,016 words, with ~203-word abstracts). For comparison, CNN/Daily Mail articles average roughly 650 words. Data and code are at github.com/acohan/long-summarization.
  • Architecture: a single-layer bidirectional LSTM at both the section level and the document level. Section-level attention scales the word-level additive attention scores. The model also tracks coverage to reduce repetition and uses a joint pointer-generator to copy words from the source.
  • On arXiv, the model scores ROUGE-1/2/L of 35.80 / 11.05 / 31.80, compared with 32.06 / 9.04 / 25.16 for Pntr-Gen-Seq2Seq.
  • On PubMed, it scores 38.93 / 15.37 / 35.21. LexRank's ROUGE-1 (39.19) is slightly higher, but the model beats all baselines on the other ROUGE metrics.
  • The authors acknowledge that ROUGE does not capture coherence or coverage. They leave expert human evaluation to future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Discourse analysis

Cited by

388 papers of our library cite it (of the 140673 whose bibliographies we read); 16 of them have a page here, 372 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1804.05685 · arXiv 1804.05685

@misc{cohan2018discourse,
  title = {{A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents}},
  author = {Cohan, Arman and Dernoncourt, Franck and Kim, Doo Soon and Bui, Trung and Kim, Seokhwan and Chang, Walter and Goharian, Nazli},
  year = {2018},
  doi = {10.48550/arxiv.1804.05685},
  eprint = {1804.05685},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1804.05685}
}

References

30 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 . find our card
  2. Sumit Chopra, Michael Auli, Alexander M Rush, and SEAS Harvard. 2016. Abstractive sentence summarization with attentive recurrent neural networks. In HLT-NAACL. pages 93–98. find our card
  3. Arman Cohan and Nazli Goharian. 2015. Scientific article summarization using citation-context and article’s discourse structure. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Lisbon, Portugal, pages 390–400. http… find
  4. Arman Cohan and Nazli Goharian. 2017a. Contextualizing citations for scientific summarization using word embeddings and domain knowledge. arXiv preprint arXiv:1705.08063 . — [arXiv:1705.08063](https://arxiv.org/abs/1705.08063) find
  5. Arman Cohan and Nazli Goharian. 2017b. Scientific document summarization via citation contextualization and scientific discourse. International Journal on Digital Libraries pages 1–17. https://doi. org/10.1007/s00799-017-0216-8. — [doi:10.1007/s00799-017-0216-8](https://doi.org/10.1007/s00799-017-0216-8) find
  6. Günes Erkan and Dragomir R Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of Artificial Intelligence Research 22:457–479. find
  7. Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013. Speech recognition with deep recurrent neural networks. In Acoustics, speech and signal processing (icassp), 2013 ieee international conference on. IEEE, pages 6645–6649. find
  8. Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany… find our card
  9. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems. pages 1693– 1701. find our card
  10. Matthew Honnibal and Mark Johnson. 2015. An improved non-monotonic transition system for dependency parsing. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Lisbon, Portugal, pages 1373– 1378. https://aclweb.org/an… ↗ our card
  11. Hongyan Jing. 2002. Using hidden markov modeling to decompose human-written summaries. Computational linguistics 28(4):527–543. find
  12. Piji Li, Wai Lam, Lidong Bing, Weiwei Guo, and Hang Li. 2017. Cascaded attention based unsupervised information distillation for compressive summarization. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. pages 2071–2080. find
  13. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out: Proceedings of the ACL-04 workshop. Barcelona, Spain, volume 8. find
  14. Jeffrey Ling and Alexander Rush. 2017. Coarse-to-fine attention models for document summarization. In Proceedings of the Workshop on New Frontiers in Summarization. Association for Computational Linguistics, Copenhagen, Denmark, pages 33– 42. http://www.aclweb.org/anthology/W17-4505. ↗
  15. Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summarizing long sequences. In International Conference on Learning Representations. https://openreview.net/forum?id=Hyg0vbWC-. ↗ our card
  16. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025 . find
  17. Yishu Miao and Phil Blunsom. 2016. Language as a latent variable: Discrete generative models for sentence compression. arXiv preprint arXiv:1609.07317 . find
  18. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. AAAI 1:1. find our card
  19. Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. arXiv preprint arXiv:1602.06023 . find
  20. Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304 . find our card
  21. Vahed Qazvinian, Dragomir R Radev, Saif M Mohammad, Bonnie Dorr, David Zajic, Michael Whidby, and Taesun Moon. 2013. Generating extractive summaries of scientific paradigms. Journal of Artificial Intelligence Research 46:165–201. find
  22. Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. arXiv preprint arXiv:1509.00685 . find
  23. Abigail See, Christopher Manning, and Peter Liu. 2017. Get to the point: Summarization with pointer-generator networks. In Association for Computational Linguistics. https://arxiv.org/abs/1704.04368. ↗
  24. Yuanlong Shao, Stephan Gouws, Denny Britz, Anna Goldie, Brian Strope, and Ray Kurzweil. 2017. Generating high-quality and informative conversation responses with sequence-to-sequence models. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. pages 2210–2219. find
  25. Josef Steinberger and Karel Jezek. 2004. Using latent semantic analysis in text summarization and summary evaluation. In Proc. ISIM04. pages 93–100. find
  26. Frederick Suppe. 1998. The structure of a scientific paper. Philosophy of Science 65(3):381–405. find
  27. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems. pages 3104–3112. find
  28. Lucy Vanderwende, Hisami Suzuki, Chris Brockett, and Ani Nenkova. 2007. Beyond sumbasic: Task-focused summarization with sentence simplification and lexical expansion. Information Processing & Management 43(6):1606–1618. find
  29. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Proces… find
  30. Sam Wiseman, Stuart M Shieber, and Alexander M Rush. 2017. Challenges in data-to-document generation. arXiv preprint arXiv:1707.08052 . find our card