Elamur Research
☰
home / papers / 2018 / Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies

Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N18-1065.pdf

Summary

Automatic summarization lacks large, high-quality data for training and evaluation. Earlier resources use stand-ins such as headlines, concatenated bullet points, or librarian-written abstracts, and each reflects a narrow style. The authors built NEWSROOM, which pairs 1.3 million news articles with summaries that newsroom authors and editors wrote for search engines and social media. They crawled over 100 million archived pages via Archive.org from 38 major English-language publishers, covering 1998–2017. The dataset is distributed as a list of immutable archive URLs.

The authors also define measures of how extractive a summary is. They use these measures to split the corpus into extractive, mixed, and abstractive subsets and to compare it with DUC, Gigaword, the New York Times Corpus, and CNN / Daily Mail. Baselines were tested with ROUGE and with a new crowdsourced human evaluation protocol. The simple Lede-3 baseline proved very hard to beat, so the dataset remains an open challenge. Training a pointer-generator on NEWSROOM also improved results on out-of-domain data.

Key points

  • NEWSROOM is described as the largest summarization dataset to date: 1.3M article–summary pairs from 38 publishers, available at summari.es.
  • The summaries come from HTML metadata and mix abstractive and extractive strategies to varying degrees. Proposed measures (including coverage and compression) quantify this diversity.
  • The evaluated systems were Lede-3, an extractive oracle (Fragments), TextRank, Abs-N, and the pointer-generator variants Pointer-C, Pointer-S and Pointer-N. Pointer-N, trained on NEWSROOM, scored best among models on every dataset except CNN / Daily Mail.
  • ROUGE-1, ROUGE-2 and ROUGE-L F1 were computed with ROUGE v1.5.5, using text tokenized by the Stanford CoreNLP tokenizer.
  • In the human evaluation, Amazon Mechanical Turk raters judged 60 summaries (20 per subset, 3 raters each) on informativeness, relevance, fluency and coherence. Lede-3 averaged 4.08, TextRank 3.81 and Pointer-S 3.73. The Fragments oracle scored only 3.08.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

299 papers of our library cite it (of the 140673 whose bibliographies we read); 16 of them have a page here, 283 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n18-1065 · arXiv 1804.11283

@inproceedings{grusky2018newsroom,
  title = {{Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies}},
  author = {Grusky, Max and Naaman, Mor and Artzi, Yoav},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-1065},
  eprint = {1804.11283},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/n18-1065}
}

References

34 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR abs/1409.0473. http://arxiv.org/abs/1409.0473. ↗ our card
  2. Federico Barrios, Federico López, Luis Argerich, and Rosa Wachenchauzer. 2016. Variations of the similarity function of textrank for automated summarization. CoRR abs/1602.03606. http://arxiv.org/abs/ 1602.03606. — [arXiv:1602.03606](https://arxiv.org/abs/1602.03606) ↗
  3. Katja Filippova and Yasemin Altun. 2013. Overcoming the lack of parallel data in sentence compression. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Spec… find
  4. Danqi Chen, Jason Bolton, and Christopher D. Manning. 2016. A thorough examination of the cnn/daily mail reading comprehension task. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. http:/… find our card
  5. Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by extracting sentences and words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. http://aclweb.org/anthology/P/P16/ P16-104… ↗ our card
  6. Çaglar Gülçehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Bengio. 2016. Pointing the unknown words http://aclweb.org/anthology/P/ P16/P16-1014.pdf. ↗
  7. Donna Harman and Paul Over. 2004. The effects of human variation in duc summarization evaluation. http://www.aclweb.org/anthology/W04-1003. ↗
  8. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Langua… find
  9. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1. MIT Press, Cambridge, MA… find our card
  10. Sumit Chopra, Michael Auli, and Alexander M. Rush. 2016. Abstractive sentence summarization with attentive recurrent neural networks. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego Californ… find our card
  11. Kai Hong and Ani Nenkova. 2014. Improving the estimation of word importance for news multi-document summarization. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2014, April 26-30, 2014, Gothenburg, Sweden. pages 712–721. http://a… find
  12. Duc 2005: Evaluation of question-focused summarization systems. In Proceedings of the Workshop on Task-Focused Summarization and Question Answering. Association for Computational Linguistics, Stroudsburg, PA, USA, SumQA ’06, pages 48–55. http://dl.acm.org/ citation.cfm?id=1654679.1654689. ↗
  13. C. Y. Lin. 2004a. Looking for a few good metrics: Automatic summarization evaluation - how many samples are enough? In Proceedings of the NTCIR Workshop 4. find
  14. Chin-Yew Lin. 2004b. Rouge: A package for automatic evaluation of summaries. In Proc. ACL workshop on Text Summarization Branches Out. page 10. http://research.microsoft.com/ ∼ cyl/ download/papers/WAS2004.pdf. ↗
  15. Bonnie Dorr, David Zajic, and Richard Schwartz. 2003. Hedge trimmer: A parse-and-trim approach to headline generation. In Proceedings of the HLT-NAACL 03 on Text Summarization Workshop - Volume 5. Association for Computational Linguistics, Stroudsburg, PA, USA, HLT-NAACL-DUC ’03, pages 1–8. https://… find
  16. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations. pages 55–60. http://www.aclweb.org/anthology/P/ P14/P14-… ↗ our card
  17. Greg Durrett, Taylor Berg-Kirkpatrick, and Dan Klein. 2016. Learning-based single-document summarization with compression and anaphoricity constraints. The Association for Computer Linguistics. http: //www.aclweb.org/anthology/P16-1188. find
  18. R. Mihalcea and P. Tarau. 2004. TextRank: Bringing order into texts. In Proceedings of EMNLP-04and the 2004 Conference on Empirical Methods in Natural Language Processing. http://www.aclweb.org/ anthology/W04-3252. ↗
  19. Katja Filippova, Enrique Alfonseca, Carlos A. Col-menares, Lukasz Kaiser, and Oriol Vinyals. 2015. Sentence compression by deletion with lstms. In Llus Mrquez, Chris Callison-Burch, Jian Su, Daniele Pighin, and Yuval Marton, editors, EMNLP. The Association for Computational Linguistics, pages 360–36… find
  20. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In Proceedings of the Thirty-First AAAI Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks… find our card
  21. Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA.. pages 3075–3081. http://aaai.org/ocs/index.php/AAAI/ AAAI17/paper/view/14636. ↗
  22. Jiwei Tan, Xiaojun Wan, and Jianguo Xiao. 2017. Abstractive document summarization with a graph-based attentional neural model. In ACL. http: //www.aclweb.org/anthology/P17-1108. find
  23. Ramesh Nallapati, Bowen Zhou, Cı́cero Nogueira dos Santos, Çaglar Gülçehre, and Bing Xiang. 2016a. Abstractive text summarization using sequence-to-sequence rnns and beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, CoNLL 2016, Berlin, Germany, Augus… find
  24. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in opus. In LREC. volume 2012, pages 2214– 2218. find
  25. Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Modeling coverage for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, pages 76–85. https://d… find our card
  26. Ramesh Nallapati, Bowen Zhou, and Mingbo Ma. 2016b. Classify or select: Neural architectures for extractive document summarization. CoRR abs/1611.04244. http://arxiv.org/abs/1611.04244. — [arXiv:1611.04244](https://arxiv.org/abs/1611.04244) ↗
  27. Courtney Napoles, Matthew Gormley, and Benjamin Van Durme. 2012. Annotated gigaword. In Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction. Association for Computational Linguistics, Stroudsburg, PA, USA, AKBC-WEKEX ’12, pages 95–100. http:/… find
  28. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28. Curran Associates, Inc., pages 2692–2700. http://papers.nips.cc/paper/ 5866-pointer-network… ↗
  29. Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1999. The pagerank citation ranking: Bringing order to the web. Technical Report 1999-66, Stanford InfoLab. Previous number = SIDL-WP-1999-0120. http://ilpubs.stanford.edu: 8090/422/. ↗ our card
  30. Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. CoRR abs/1705.04304. http://arxiv.org/ abs/1705.04304. ↗ our card
  31. Matt Peters. 2015. Benchmarking python content extraction algorithms: Dragnet, readability, goose, and eatiht. https://moz.com/ devblog/benchmarking-python-content-extraction-/ algorithms-dragnet-readability-goose-and-eatiht/. ↗
  32. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015. pages 379–389. http://aclweb.org/a… ↗
  33. E. Sandhaus. 2008. The New York Times Annotated Corpus. Linguistic Data Consortium, Philadelphia 6(12). find
  34. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers. pages 1073… find