Elamur Research⌕
☰
home / papers / 2018 / Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization

Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization

Paper · 2018 · Shashi Narayan, Shay B. Cohen and Mirella Lapata

Summary

The authors propose a new task: "extreme" single-document summarization. Given a news article, a model must generate a single short sentence explaining what the article is about. Such a summary can hardly be obtained by simply selecting an existing sentence from the text, so an abstractive approach is required. For this task, they built the XSum corpus of BBC articles from 2010–2017. Its reference summary is the introductory sentence written by the journalist. They also propose T-ConvS2S, a sequence-to-sequence architecture built entirely on convolutional networks. Its encoder and decoder additionally receive topic vectors for individual words and for the whole document. Convolutions capture long-range dependencies in text better than RNNs. In terms of ROUGE, the model outperformed the extractive oracle and strong RNN systems, including PtGen. Human evaluators also judged its summaries to be more informative and complete. The work matters for two reasons. First, it showed that the popular CNN, DailyMail and NY Times corpora favor extractive methods. Second, it provided an open resource for developing truly abstractive systems.

Key points

  • XSum: 226,711 BBC articles, split into train/val/test as 204,045 / 11,332 / 11,334. A document averages about 431 words and a summary about 23 words (exactly one sentence). The data, code and demo are available on GitHub (shashiongithub/XSum).
  • Abstractiveness of the corpus: about 36% of unigrams in the XSum reference summaries do not appear in the article text. For CNN and DailyMail this figure is about 17%, and for NY Times it is 23%. About 83% of bigrams in XSum are novel, compared with roughly half on the other corpora.
  • Extractive baselines are weak on XSum: LEAD achieves a ROUGE-1 of 16.30 and EXT-ORACLE 29.79. On the other corpora, the oracle exceeds 50.
  • Human evaluation using Best-Worst Scaling (Mechanical Turk, 50 documents): GOLD 0.431, T-ConvS2S 0.037 (second place), EXT-ORACLE −0.121, ConvS2S −0.130, PtGen −0.218.
  • QA-based evaluation (100 factual questions): using T-ConvS2S summaries, participants correctly answered 46.05% of the questions. The figure is 30.90% for ConvS2S, 21.40% for PtGen, 15.70% for EXT-ORACLE, and 97.23% for the reference summaries.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom381 works · 80,747 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Summarization · 243 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Text summarization · 49 works

Cite

Cite as Shashi Narayan, Shay B. Cohen and Mirella Lapata. 2018. Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1206 · arXiv 1808.08745

@inproceedings{narayan2018don,
  title = {{Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization}},
  author = {Shashi Narayan, Shay B. Cohen and Mirella Lapata},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1206},
  eprint = {1808.08745},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d18-1206}
}

References

41 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the 3rd International Conference on Learning Representations, San Diego, California, USA.
  2. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. The Journal of Machine Learning Research, 3:993–1022.
  3. Asli Celikyilmaz, Antoine Bosselut, Xiaodong He, and Yejin Choi. 2018. Deep communicating agents for abstractive summarization. In Proceedings of the 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, USA.
  4. Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, and Hui Jiang. 2016. Distraction-based neural networks for modeling documents. In Proceedings of the 25th International Joint Conference on Artificial Intelligence, pages 2754–2760, New York, USA. find
  5. Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by extracting sentences and words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 484– 494, Berlin, Germany. of the 56th Annual Meeting of the Association for Computational Linguistics, M…
  6. Jonas Gehring, Michael Auli, David Grangier, and Yann Dauphin. 2017a. A convolutional encoder model for neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 123–135, Vancouver, Canada.
  7. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017b. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 1243–1252, Sydney, Australia.
  8. Shalini Ghosh, Oriol Vinyals, Brian Strope, Scott Roy, Tom Dean, and Larry Heck. 2016. Contextual LSTM (CLSTM) models for large scale NLP tasks. CoRR, abs/1602.06291. find
  9. Max Grusky, Mor Naaman, and Yoav Artzi. 2018. NEWSROOM: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, USA.
  10. James Clarke and Mirella Lapata. 2010. Discourse constraints for document compression. Computational Linguistics, 36(3):411–441.
  11. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, Las Vegas, USA.
  12. Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning, pages 933–941, Sydney, Australia.
  13. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems 28, pages 1693– 1701. Morgan, Kaufmann.
  14. Adji B. Dieng, Chong Wang, Jianfeng Gao, and John Paisley. 2017. Topicrnn: A recurrent neural network with long-range semantic dependency. In Proceedings of the 5th International Conference on Learning Representations, Toulon, France. find
  15. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121–2159. find
  16. Greg Durrett, Taylor Berg-Kirkpatrick, and Dan Klein. 2016. Learning-based single-document summarization with compression and anaphoricity constraints. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1998–2008, Berlin, Germany.
  17. Angela Fan, David Grangier, and Michael Auli. 2017. Controllable abstractive summarization. CoRR, abs/1711.05217.
  18. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings Svetlana Kiritchenko and Saif Mohammad. 2017. Best-worst scaling more reliable than rating scales: A case study on sentiment intensity annotation. In Proceedings of the 55th Annual Meeting of the Ass…
  19. Chin-Yew Lin and Eduard Hovy. 2003. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pages 71–78, Edmonton, Canada.
  20. Jordan J Louviere, Terry N Flynn, and Anthony Alfred John Marley. 2015. Best-worst scaling: Theory, methods and applications. Cambridge University Press. find
  21. Jordan J Louviere and George G Woodworth. 1991. Best-worst scaling: A model for the largest difference judgments. University of Alberta: Working Paper. entailment. In Proceedings of the 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu… find
  22. Inderjeet Mani. 2001. Automatic Summarization. Natural language processing. John Benjamins Publishing Company. find
  23. Tomas Mikolov and Geoffrey Zweig. 2012. Context dependent recurrent neural network language model. In Proceedings of the Spoken Language Technology Workshop, pages 234–239. IEEE. find
  24. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. SummaRuNNer: A recurrent neural network based sequence model for extractive summarization of documents. In Proceedings of the 31st AAAI Conference on Artificial Intelligence, pages 3075–3081, San Francisco, California USA.
  25. Ramesh Nallapati, Bowen Zhou, Cı́cero Nogueira dos Santos, Çaglar Gülçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pages 280–290, Berlin, Germany.
  26. Shashi Narayan, Ronald Cardenas, Nikos Papasarantopoulos, Shay B. Cohen, Mirella Lapata, Jiang-sheng Yu, and Yi Chang. 2018a. Document modeling with external attention for sentence extraction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Aust… find
  27. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018b. Ranking sentences for extractive summarization with reinforcement learning. In Proceedings of the 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans,…
  28. Shashi Narayan, Nikos Papasarantopoulos, Shay B. Cohen, and Mirella Lapata. 2017. Neural extractive summarization with side information. CoRR, abs/1704.04530. — [arXiv:1704.04530](https://arxiv.org/abs/1704.04530) find
  29. Ani Nenkova. 2005. Automatic text summarization of newswire: Lessons learned from the Document Understanding Conference. In Proceedings of the 29th National Conference on Artificial Intelligence, pages 1436–1441, Pittsburgh, Pennsylvania, USA. find
  30. Ani Nenkova and Kathleen McKeown. 2011. Automatic summarization. Foundations and Trends in Information Retrieval, 5(2–3):103–233. find
  31. Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In Proceedings of the 6th International Conference on Learning Representations, Vancouver, BC, Canada.
  32. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379–389, Lisbon, Portugal.
  33. Evan Sandhaus. 2008. The New York Times Annotated Corpus. Linguistic Data Consortium, Philadelphia, 6(12). find
  34. Natalie Schluter. 2017. The limits of automatic summarisation according to rouge. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Short Papers, pages 41–45, Valencia, Spain.
  35. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1073–1083, Vancouver, Canada. find
  36. Xing Shi, Kevin Knight, and Deniz Yuret. 2016. Why neural translations are the right length. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2278–2282, Austin, Texas. find
  37. Sainbayar Sukhbaatar, arthur szlam, Jason Weston, and Rob Fergus. 2015. End-to-end memory networks. In Advances in Neural Information Processing Systems 28, pages 2440–2448. Morgan, Kaufmann. find
  38. Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. 2013. On the importance of initialization and momentum in deep learning. In Proceedings of the 30th International Conference on International Conference on Machine Learning, pages 1139–1147, Atlanta, GA, USA. find
  39. Jiwei Tan and Xiaojun Wan. 2017. Abstractive document summarization with a graph-based attentional neural model. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1171–1181, Vancouver, Canada.
  40. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on International Conference on Machine Learning, pages 1310–1318, Atlanta, GA, USA. find
  41. Ramakanth Pasunuru and Mohit Bansal. 2018. Multi-reward reinforced summarization with saliency and find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/narayan-2018-don-t-give-me-details/