Elamur Research
☰
home / papers / 2019 / Multi-News: A Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model

Multi-News: A Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P19-1102.pdf

Summary

Neural summarization has advanced mainly for single documents, because large training corpora exist only there. News multi-document summarization (MDS) benchmarks such as DUC 2004 and TAC 2011 contain only a few hundred examples or fewer. To close this gap, the authors built Multi-News from newser.com, where professional editors write summaries that cite and link to several source articles. The dataset has 56,216 summary–cluster pairs drawn from a wide range of outlets. The authors also propose Hi-MAP, a hierarchical model that adds a Maximal Marginal Relevance (MMR) module to a pointer-generator network. Sentence-level relevance and redundancy scores reweight the word-level attention, and the similarity parameters are learned jointly with the rest of the network rather than fixed in advance. On Multi-News, Hi-MAP achieves the best R-2 and R-SU scores and comes second to a CopyTransformer on R-1. In human evaluation it produces much less redundant output than PG-MMR. Multi-News makes it possible to train data-hungry abstractive models for the multi-document setting.

Key points

  • Data: Multi-News has a 44,972 / 5,622 / 5,622 train/validation/test split. Each cluster has 2–10 sources, and more than 1,500 news sites appear at least 5 times. Summaries average about 264 words, much longer than in DUC or CNNDM.
  • Abstractiveness: 17.76% of summary unigrams and 82.30% of 4-grams do not appear in the sources. This is close to the single-document CNNDM dataset, while the small MDS sets are more abstractive.
  • Results on Multi-News: Hi-MAP scores R-1 43.47, R-2 14.89 and R-SU 17.41. CopyTransformer reaches R-1 43.57 and R-2 14.03. Both beat PG-MMR and extractive baselines such as LexRank and TextRank.
  • Human evaluation: Annotators used Best-Worst Scaling on 50 test clusters. In pairwise choices for non-redundancy, Hi-MAP was picked as best 100 times versus 45 for PG-MMR. Human-written summaries still clearly won overall.
  • Resources: Data and code are at github.com/Alex-Fabbri/Multi-News, with Wayback-archived links and reconstruction scripts.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

306 papers of our library cite it (of the 140673 whose bibliographies we read); 14 of them have a page here, 292 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1102 · arXiv 1906.01749

@inproceedings{fabbri2019multi,
  title = {{Multi-News: A Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model}},
  author = {Fabbri, Alexander and Li, Irene and She, Tianwei and Li, Suyi and Radev, Dragomir},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1102},
  eprint = {1906.01749},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p19-1102}
}

References

46 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 2008. The New York Times Annotated Corpus. find
  2. Stefanos Angelidis and Mirella Lapata. 2018. Summarizing Opinions: Aspect Extraction Meets Sentiment Prediction and They Are Both Weakly Supervised. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 3675… find
  3. Alexei Baevski and Michael Auli. 2019. Adaptive input representations for neural language modeling. In International Conference on Learning Representations. find
  4. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. arXiv preprint arXiv:1409.0473. find our card
  5. Regina Barzilay, Kathleen R. McKeown, and Michael Elhadad. 1999. Information Fusion in the Context of Multi-Document Summarization. In 27th Annual Meeting of the Association for Computational Linguistics, University of Maryland, College Park, Maryland, USA, 20-26 June 1999. find
  6. Tal Baumel, Matan Eyal, and Michael Elhadad. 2018. Query Focused Abstractive Summarization: Incorporating Query Relevance, Multi-Document Coverage, and Summary Length Constraints into seq2seq Models. CoRR, abs/1801.07704. — [arXiv:1801.07704](https://arxiv.org/abs/1801.07704) find
  7. Asli Çelikyilmaz, Antoine Bosselut, Xiaodong He, and Yejin Choi. 2018. Deep Communicating Agents for Abstractive Summarization. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orle… find
  8. Jianpeng Cheng and Mirella Lapata. 2016. Neural Summarization by Extracting Sentences and Words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. find our card
  9. Sumit Chopra, Michael Auli, and Alexander M. Rush. 2016. Abstractive Sentence Summarization with Attentive Recurrent Neural Networks. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego Californ… find our card
  10. Eric Chu and Peter J. Liu. 2018. Unsupervised Neural Multi-Document Abstractive Summarization. CoRR, abs/1810.05739. — [arXiv:1810.05739](https://arxiv.org/abs/1810.05739) find
  11. Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational… find our card
  12. Zihang Dai, Zhilin Yang, Yiming Yang, William W. Cohen, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Language modeling with longer-term dependency. find
  13. Günes Erkan and Dragomir R Radev. 2004. Lexrank: Graph-Based Lexical Centrality as Salience in Text Summarization. Journal of artificial intelligence research, 22:457–479. find
  14. Ziqiang Cao, Wenjie Li, Sujian Li, and Furu Wei. 2017. Improving Multi-Document Summarization via Text Classification. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA., pages 3053–3059. find
  15. Kavita Ganesan, ChengXiang Zhai, and Jiawei Han. 2010. Opinosis: A Graph Based Approach to Abstractive Summarization of Highly Redundant Opinions. In COLING 2010, 23rd International Conference on Computational Linguistics, Proceedings of the Conference, 23-27 August 2010, Beijing, China, pages 340–3… find
  16. Jaime Carbonell and Jade Goldstein. 1998. The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries. In Proceedings of the 21st annual international ACM SI-GIR conference on Research and development in information retrieval, pages 335–336. ACM. find
  17. Sebastian Gehrmann, Yuntian Deng, and Alexander M. Rush. 2018. Bottom-Up Abstractive Summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 4098–4109. find our card
  18. Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies. CoRR, abs/1804.11283. find our card
  19. Aria Haghighi and Lucy Vanderwende. 2009. Exploring Content Models for Multi-Document Summarization. In Human Language Technologies: Conference of the North American Chapter of the Association of Computational Linguistics, Proceedings, May 31 - June 5, 2009, Boulder, Colorado, USA, pages 362–370. find
  20. Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching Machines to Read and Comprehend. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December… find our card
  21. Proceedings, ACM Conference on Research and Development in Information Retrieval SIGIR’95, pages 74–82, Seattle, Washington. find
  22. Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing Order into Text. In Proceedings of the 2004 conference on empirical methods in natural language processing. find
  23. Ramesh Nallapati, Bowen Zhou, and Mingbo Ma. 2016a. Classify or Select: Neural Architectures for Extractive Document Summarization. CoRR, abs/1611.04244. — [arXiv:1611.04244](https://arxiv.org/abs/1611.04244) find
  24. Ramesh Nallapati, Bowen Zhou, Cı́cero Nogueira dos Santos, Çaglar Gülçehre, and Bing Xiang. 2016b. Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, CoNLL 2016, Berlin, Germany, Augus… find
  25. Kai Hong, John M. Conroy, Benoı̂t Favre, Alex Kulesza, Hui Lin, and Ani Nenkova. 2014. A repository of state of the art and competitive baseline summaries for generic news summarization. In Proceedings of the Ninth International Conference on Language Resources and Evaluation, LREC 2014, Reykjavik,… find
  26. Courtney Napoles, Matthew R. Gormley, and Benjamin Van Durme. 2012. Annotated Gigaword. In Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction, AKBC-WEKEX@NAACL-HLT 2012, Montrèal, Canada, June 7-8, 2012, pages 95–100. find
  27. Svetlana Kiritchenko and Saif Mohammad. 2017. Best-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 2: Sh… find
  28. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018a. Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807. Association for Comp… find
  29. Logan Lebanoff, Kaiqiang Song, and Fei Liu. 2018. Adapting the Neural Encoder-Decoder Framework from Single to Multi-Document summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 4131–4141. find
  30. Chin-Yew Lin. 2004. Rouge: A Package for Automatic Evaluation of Summaries. Text Summarization Branches Out. find
  31. Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating Wikipedia by Summarizing Long Sequences. CoRR, abs/1801.10198. find our card
  32. Jordan Louviere, Terry Flynn, and A. A. J. Marley. 2015. Best-Worst Scaling: Theory, Methods and Applications. find
  33. Jordan J Louviere and George G Woodworth. 1991. Best-Worst Scaling: A Model for the Largest Difference Judgments. find
  34. Kathleen R. McKeown and Dragomir R. Radev. 1995. Generating summaries of multiple news articles. In Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018b. Ranking Sentences for Extractive Summarization with Reinforcement Learning. In Proceedings of the 2018 Conference of the North American Chapte… find
  35. Karolina Owczarzak and Hoa Trang Dang. 2011. Overview of the TAC 2011 Summarization Track: Guided Task and AESOP Task. find
  36. Over Paul and Yen James. 2004. An Introduction to DUC-2004. In Proceedings of the 4th Document Understanding Conference (DUC 2004). find
  37. Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A Deep Reinforced Model for Abstractive Summarization. CoRR, abs/1705.04304. find our card
  38. Dragomir R. Radev, Hongyan Jing, and Malgorzata Budzikowska. 2000. Centroid-Based Summarization of Multiple Documents: Sentence Extraction utility-based evaluation, and user studies. CoRR, cs.CL/0005020. find
  39. Dragomir R. Radev and Kathleen R. McKeown. 1998. Generating Natural Language Summaries from Multiple On-Line Sources. Computational Linguistics, 24(3):469–500. find
  40. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A Neural Attention Model for Abstractive Sentence Summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 379–389. find
  41. Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get To The Point: Summarization with Pointer-Generator Networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1073–1083. find
  42. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008. find
  43. Michihiro Yasunaga, Rui Zhang, Kshitijh Meelu, Ayush Pareek, Krishnan Srinivasan, and Dragomir R. Radev. 2017. Graph-Based Neural Multi-Document Summarization. In Proceedings of CoNLL-2017. Association for Computational Linguistics. find
  44. Jianmin Zhang, Jiwei Tan, and Xiaojun Wan. 2018a. Adapting Neural Single-Document Summarization Model for Abstractive Multi-Document Summarization: A Pilot Study. In Proceedings of the 11th International Conference on Natural Language Generation, Tilburg University, The Netherlands, November 5-8, 20… find
  45. Jianmin Zhang, Jiwei Tan, and Xiaojun Wan. 2018b. Towards a Neural Network Approach to Abstractive Multi-Document Summarization. CoRR, abs/1804.09010. — [arXiv:1804.09010](https://arxiv.org/abs/1804.09010) find
  46. Markus Zopf. 2018. Auto-hmds: Automatic Construction of a Large Heterogeneous Multilingual Multi-Document Summarization Corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018. find