Elamur Research⌕
☰
home / papers / 2020 / Multi-XScience: A Large-scale Dataset for Extreme Multi-document Summarization of Scientific Articles

Multi-XScience: A Large-scale Dataset for Extreme Multi-document Summarization of Scientific Articles

Paper · 2020 · Yao Lu, Yue Dong and Laurent Charlin

Summary

Progress in multi-document summarization (MDS) has been slowed by a lack of large training corpora. Only Multi-News and WikiSum were available, and both allow models to score well by copying source sentences. The authors introduce Multi-XScience, built from arXiv papers linked to the Microsoft Academic Graph (MAG). Each example asks a model to produce one paragraph of a paper's related-work section. The input is the paper's own abstract plus the abstracts of the works cited in that paragraph. The design follows the "extreme summarization" idea behind XSum, so the targets reward abstraction over extraction.

Statistics show that the dataset's targets contain many n-grams absent from the sources. Lead and extractive-oracle baselines score relatively low, which indicates weak positional and copying shortcuts. Several state-of-the-art systems were benchmarked. Abstractive models clearly beat unsupervised extractive ones, and human judges preferred them to the extractive oracle. The resource gives the field a demanding, domain-specific MDS benchmark. It also opens directions such as citation-graph-aware summarization.

Key points

  • Construction: The team processed the LaTeX sources of about 1.3M arXiv papers and aligned them with MAG (October 2019 dumps). The records then went through five cleaning rounds combined with human checks. The final split is 30,369 / 5,066 / 5,093 train/val/test pairs.
  • Size and inputs: Inputs average 778 words and targets about 116 words. Each example has 4.42 references on average, roughly 60% more than Multi-News (2.79).
  • Abstractiveness: 42.33% of target unigrams are novel, the highest among compared datasets. Novel bigrams reach 81.75%, second only to XSum. The LEAD baseline reaches R-1 27.46 (vs. 43.08 on Multi-News), and the EXT-ORACLE reaches R-1 38.45 (vs. 49.06).
  • Models tested: LexRank, TextRank, Pointer-Generator, HiMAP, HierSumm, BART, BertAbs, and SciBertAbs.
  • Human evaluation:
    • Source–target fact coverage averaged 2.82 ± 0.4 on a 1–4 scale (25 pairs, two judges).
    • On a 1–3 system ranking, HiMAP scored 2.28, Pointer-Generator 2.18, and EXT-ORACLE 1.54.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University College LondonLondon · United Kingdom59 works · 16,463 citations🎓University of WaterlooWaterloo · Canada44 works · 12,783 citations🎓McGill UniversityMontreal · Canada51 works · 10,476 citations🔬Mila - Quebec Artificial Intelligence InstituteMontreal · Canada31 works · 13,061 citations🔬Canadian Institute for Advanced ResearchToronto · Canada17 works · 16,274 citations🎓HEC MontréalMontreal · Canada3 works · 9,680 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Computer Vision and Pattern Recognition · 1,282 works

🧭 Topics 1 Summarization · 159 works

Cite

Cite as Yao Lu, Yue Dong and Laurent Charlin. 2020. Multi-XScience: A Large-scale Dataset for Extreme Multi-document Summarization of Scientific Articles. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.648

@inproceedings{lu2020multi,
  title = {{Multi-XScience: A Large-scale Dataset for Extreme Multi-document Summarization of Scientific Articles}},
  author = {Yao Lu, Yue Dong and Laurent Charlin},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.648},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.648}
}

References

29 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed AbuRa’ed, Horacio Saggion, and Luis Chiruzzo. A multi-level annotated corpus of scientific papers for scientific document summarization and cross-document relation discovery. find
  2. Iz Beltagy, Arman Cohan, and Kyle Lo. 2019. Scibert: Pretrained contextualized embeddings for scientific text. arXiv preprint arXiv:1903.10676. our card
  3. Jaime Carbonell and Jade Goldstein. 1998. The use of mmr, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, pages 335–336. find
  4. Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational… our card
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… our card
  6. Günes Erkan and Dragomir R Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of artificial intelligence research, 22:457–479. find
  7. Alexander Richard Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074–1084. our card
  8. Matt Grenander, Yue Dong, Jackie Chi Kit Cheung, and Annie Louis. 2019. Countering the effects of lead bias in news summarization via multi-stage training and auxiliary losses. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint C… find
  9. Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pa… our card
  10. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in neural information processing systems, pages 1693–1701. our card
  11. Cong Duy Vu Hoang and Min-Yen Kan. 2010. Towards automated related work summarization. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters, pages 427–435. find
  12. Yue Hu and Xiaojun Wan. 2014. Automatic generation of related work sections in scientific papers: an optimization approach. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1624–1633. find
  13. Kokil Jaidka, Muthu Kumar Chandrasekaran, Sajal Rustagi, and Min-Yen Kan. 2016. Overview of the cl-scisumm 2016 shared task. In Proceedings of the joint workshop on bibliometric-enhanced information retrieval and natural language processing for digital libraries (BIRNDL), pages 93–102. find
  14. Kokil Jaidka, Christopher Khoo, and Jin-Cheon Na. 2013. Deconstructing human literature reviews–a framework for multi-document summarization. In proceedings of the 14th European workshop on natural language generation, pages 125–135. find
  15. Logan Lebanoff, Kaiqiang Song, and Fei Liu. 2018. Adapting the neural encoder-decoder framework from single to multi-document summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4131–4141. find
  16. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. our card
  17. Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summarizing long sequences. In International Conference on Learning Representations. our card
  18. Yang Liu and Mirella Lapata. 2019a. Hierarchical transformers for multi-document summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5070– 5081. our card
  19. Yang Liu and Mirella Lapata. 2019b. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3721–3731. find
  20. Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing, pages 404–411. find
  21. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In Thirty-First AAAI Conference on Artificial Intelligence. our card
  22. Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018a. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807. find
  23. Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018b. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807. find
  24. Evan Sandhaus. 2008. The new york times annotated corpus. Linguistic Data Consortium, Philadelphia, 6(12):e26752. find
  25. Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1073–1083. find
  26. Eva Sharma, Chen Li, and Lu Wang. 2019. Bigpatent: A large-scale dataset for abstractive and coherent summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2204–2213. our card
  27. Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June Hsu, and Kuansan Wang. 2015. An overview of microsoft academic service (mas) and applications. In Proceedings of the 24th international conference on world wide web, pages 243– 246. find
  28. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. our card
  29. Michihiro Yasunaga, Jungo Kasai, Rui Zhang, Alexander R Fabbri, Irene Li, Dan Friedman, and Dragomir R Radev. 2019. Scisummnet: A large annotated corpus and content-impact models for scientific paper summarization with citation networks. In Proceedings of the AAAI Conference on Artificial Intelligen… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lu-2020-multi-xscience-large-scale-dataset/