Elamur Research⌕
☰
home / papers / 2018 / Content Selection in Deep Learning Models of Summarization

Content Selection in Deep Learning Models of Summarization

Paper · 2018 · Chris Kedzie, Kathleen McKeown and Hal Daumé III

Summary

Neural extractive summarizers are widely used, but it is unclear which signals they actually rely on when choosing summary sentences. Kedzie, McKeown and Daumé III study this empirically on six datasets from four domains: news (CNN-DailyMail, NYT, DUC), personal stories from Reddit, AMI meeting transcripts, and medical articles from PubMed, a corpus the authors assembled themselves. They frame extraction as binary sentence tagging under a word budget. Three sentence encoders (word-embedding averaging, bidirectional GRU, Kim-style CNN) are paired with four extractors. Two of these are the auto-regressive Cheng & Lapata and SummaRunner designs. The other two are newly proposed RNN and Seq2Seq extractors that label each sentence independently. The elaborate components add little. Simple averaging matches or beats recurrent and convolutional encoders, and independent extraction is at least as good as conditioning on earlier decisions. In news, models mainly learn where an article starts. This suggests that adapting summarizers to new domains may be easier than assumed. It also suggests that summarization needs better sentence representations or external knowledge.

Key points

  • Dataset training-set sizes: CNN/DM 287,113, NYT 44,382, DUC 516, Reddit 404, AMI 98, PubMed 21,250 documents. Code and preprocessing are released in the kedz/nnsum repository (branch emnlp18-release).
  • Encoders and embeddings: Averaging word embeddings performed as well as or better than RNN and CNN encoders in every domain. Pre-trained embeddings matched or beat learned ones on five of the six datasets.
  • Shuffling experiment: Randomly reordering sentences during training caused significant drops on the news sets and PubMed. It made no significant difference on Reddit and slightly improved AMI, showing that position is learned implicitly even without explicit location features.
  • Lead bias in news: On CNN/DM, the Cheng & Lapata and Seq2Seq extractors (with averaging) shared 87.8% of their selected sentences. Also, 58% of Seq2Seq picks appeared in the lead baseline. Shuffling lowered this overlap to 35.2%, but at a clear cost in quality.
  • Manual analysis: News models learned to skip quotations near the top of articles. However, they often still selected the separate quote-attribution sentences, which carry little information.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Columbia UniversityNew York · United States149 works · 48,852 citations🎓University of Maryland, College ParkCollege Park · United States83 works · 23,784 citations

Also on the byline: Microsoft Research New York City (United States).

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Summarization · 159 works

Cite

Cite as Chris Kedzie, Kathleen McKeown and Hal Daumé III. 2018. Content Selection in Deep Learning Models of Summarization. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1208

@inproceedings{kedzie2018content,
  title = {{Content Selection in Deep Learning Models of Summarization}},
  author = {Chris Kedzie, Kathleen McKeown and Hal Daumé III},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1208},
  url = {https://doi.org/10.18653/v1/d18-1208}
}

References

37 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2016. A simple but tough-to-beat baseline for sentence embeddings. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
  3. Taylor Berg-Kirkpatrick, Dan Gillick, and Dan Klein. 2011. Jointly learning to extract and compress. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 481–490. Association for Computational Linguistics.
  4. Ronald Brandow, Karl Mitze, and Lisa Rau. 1999. Automatic condensation of electronic publications by sentence selection. In Jan Fagerberg, David C. Mowery, and Richard R. Nelson, editors, Advances in Automatic Text Summarization, chapter 19, pages 293–303. MIT Press, Oxford. find
  5. Jean Carletta, Simone Ashby, Sebastien Bourban, Mike Flynn, Mael Guillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaiskos, Wessel Kraaij, Melissa Kronenthal, et al. 2005. The ami meeting corpus: A pre-announcement. In International Workshop on Machine Learning for Multimodal Interaction, pages 28–… find
  6. Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by extracting sentences and words. arXiv preprint arXiv:1603.07252.
  7. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555. find
  8. John M Conroy and Dianne P O’Leary. 2001. Text summarization via hidden markov models. In Proceedings of the 24th annual international ACM SI-GIR conference on Research and development in information retrieval, pages 406–407. ACM. find
  9. Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the EACL 2014 Workshop on Statistical Machine Translation.
  10. Greg Durrett, Taylor Berg-Kirkpatrick, and Dan Klein. 2016. Learning-based single-document summarization with compression and anaphoricity constraints. arXiv preprint arXiv:1603.08887.
  11. Noemie Elhadad, M-Y Kan, Judith L Klavans, and KR McKeown. 2005. Customization in a unified framework for summarizing medical literature. Artificial intelligence in medicine, 33(2):179–198. find
  12. Michel Galley. 2006. A skip-chain conditional random field for ranking meeting utterances by importance. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 364–372. Association for Computational Linguistics. find
  13. Albert Gatt and Emiel Krahmer. 2018. Survey of the state of the art in natural language generation: Core tasks, applications and evaluation. Journal of Artificial Intelligence Research, 61:65–170. find
  14. Dan Gillick, Korbinian Riedhammer, Benoit Favre, and Dilek Hakkani-Tur. 2009. A global optimization framework for meeting summarization. In Acoustics, Speech and Signal Processing, 2009. find
  15. ICASSP 2009. IEEE International Conference on, pages 4769–4772. IEEE. find
  16. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256. find
  17. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems (NIPS).
  18. Kai Hong and Ani Nenkova. 2014. Improving the estimation of word importance for news multi-document summarization. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 712– 721. find
  19. Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015. Deep unordered composition rivals syntactic methods for text classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural L…
  20. Julian Kupiec, Jan Pedersen, and Francine Chen. 1995. A trainable document summarizer. In Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval, pages 68–73. ACM. find
  21. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out.
  22. Rashmi Mishra, Jiantao Bian, Marcelo Fiszman, Charlene R Weir, Siddhartha Jonnalagadda, Javed Mostafa, and Guilherme Del Fiol. 2014. Text summarization in the biomedical domain: a systematic review of recent research. Journal of biomedical informatics, 52:457–467. find
  23. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In AAAI, pages 3075–3081.
  24. Ani Nenkova. 2005. Automatic text summarization of newswire: Lessons learned from the document understanding conference. In AAAI, volume 5, pages 1436–1441. find
  25. Jessica Ouyang, Serina Chang, and Kathy McKeown. 2017. Crowd-sourced iterative annotation for narrative summarization corpora. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, volume 2, pages 46–51. find
  26. Paul Over and Walter Liggett. 2002. Introduction to duc: An intrinsic evaluation of generic news text summarization systems. Proc. DUC. http://wwwnlpir.nist. gov/projects/duc/guidelines/2002. html. ↗
  27. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543.
  28. Stefan Riezler and John T Maxwell. 2005. On some pitfalls in automatic evaluation and significance testing for mt. In Proceedings of the ACL workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 57– 64.
  29. Kokil Jaidka, Muthu Kumar Chandrasekaran, Sajal Rustagi, and Min-Yen Kan. 2017. Insights from cl-scisumm 2016: the faceted scientific document summarization shared task. International Journal on Digital Libraries, pages 1–9. find
  30. Evan Sandhaus. 2008. The new york times annotated corpus. Linguistic Data Consortium, Philadelphia, 6(12):e26752. find
  31. Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882.
  32. Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368. find
  33. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  34. Ruben Sipos, Pannaga Shivaswamy, and Thorsten Joachims. 2012. Large-margin learning of submodular summarization models. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 224– 233. Association for Computational Linguistics. find
  35. John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2015. Towards universal paraphrastic sentence embeddings. arXiv preprint arXiv:1511.08198.
  36. John Wieting and Kevin Gimpel. 2017. Revisiting recurrent networks for paraphrastic sentence embeddings. arXiv preprint arXiv:1705.00364. find
  37. Michihiro Yasunaga, Rui Zhang, Kshitijh Meelu, Ayush Pareek, Krishnan Srinivasan, and Dragomir Radev. 2017. Graph-based neural multi-document summarization. arXiv preprint arXiv:1706.06681.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kedzie-2018-content-selection-deep-learning-models/