Elamur Research⌕
☰
home / papers / 2020 / TLDR: Extreme Summarization of Scientific Documents

TLDR: Extreme Summarization of Scientific Documents

Summary

The paper introduces TLDR generation: producing a one-sentence summary of a scientific paper that captures its main contribution. This is a form of extreme summarization that demands heavy compression and domain expertise. To support the task, the authors build SciTLDR, a corpus of computer science papers that has several reference summaries per paper. Some references are TLDRs that authors submitted on OpenReview (TLDR-Auth). Others are TLDR-PR summaries, which trained annotators wrote by condensing the summary paragraphs found in peer reviews, so they never had to read the whole paper. The authors set up extractive and abstractive Transformer baselines. They then propose CATTS, a training strategy for BART that treats paper titles as an extra, freely available supervision signal. Control codes tell the model whether to output a title or a TLDR. CATTS beats strong baselines on both automatic metrics and human judgments of informativeness and correctness. The resource matters because scientific summarization data is scarce, and multiple references per paper allow more reliable evaluation.

Key points

  • SciTLDR has 5,411 TLDRs for 3,229 papers. Training uses 1,992 papers with one TLDR each. Dev has 619 papers with 1,452 TLDRs, and test has 618 papers with 1,967 TLDRs. Full texts were parsed with the S2ORC pipeline.
  • Documents average about 5K words and summaries about 21 words, giving a compression ratio of roughly 238, far above other scientific datasets. About 15% of summary words do not appear in the source.
  • TLDR-PR annotation used 28 University of Washington CS undergraduates. They saw only the first 128 words of a review and aimed for 15–25 word TLDRs.
  • CATTS stands for Controlled Abstraction for TLDRs with Title Scaffolding. It combines scaffold-style multitask learning with control codes and was evaluated with abstract-only and AIC (abstract, intro, conclusion) inputs.
  • An oracle input built from full-text sentences that maximize Rouge-1 improved all models. This suggests that better input selection beyond AIC is a promising direction.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Summarization

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.findings-emnlp.428

@inproceedings{cachola2020tldr,
  title = {{TLDR: Extreme Summarization of Scientific Documents}},
  author = {Cachola, Isabel and Lo, Kyle and Cohan, Arman and Weld, Daniel},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2020},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.428},
  url = {https://doi.org/10.18653/v1/2020.findings-emnlp.428}
}

References

58 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiulei Liu, Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2020. Unilmv2: Pseudo-masked language models for unified language model pre-training. ArXiv, abs/2002.12804. find our card
  2. Bin Bi, Chenliang Li, Chen Wu, Ming Yan, and Wei Wang. 2020. Palm: Pre-training an autoencoding and autoregressive language model for context-conditioned generation. ArXiv, abs/2004.07159. — [arXiv:2004.07159](https://arxiv.org/abs/2004.07159) find
  3. Muthu Kumar Chandrasekaran, Michihiro Yasunaga, Dragomir Radev, Dayne Freitag, and Min-Yen Kan. 2019. Overview and results: Cl-scisumm shared task 2019. In Workshop on Bibliometric-enhanced Information Retrieval and NLP for Digital Libraries (BIRNDL). find
  4. Arman Cohan, Waleed Ammar, Madeleine van Zuylen, and Field Cady. 2019. Structural scaffolds for citation intent classification in scientific publications. In NAACL-HLT. find our card
  5. Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In NAACL-HLT. find our card
  6. Arman Cohan and Nazli Goharian. 2015. Scientific article summarization using citation-context and article’s discourse structure. In EMNLP. find
  7. Arman Cohan and Nazli Goharian. 2016. Revisiting summarization evaluation for scientific articles. ArXiv, abs/1604.00400. find
  8. Arman Cohan and Nazli Goharian. 2017. Scientific document summarization via citation contextualization and scientific discourse. International Journal on Digital Libraries, 19:287–303. find
  9. Ed Collins, Isabelle Augenstein, and Sebastian Riedel. 2017. A supervised approach to extractive summarisation of scientific papers. CoNLL, abs/1706.03946. find
  10. John M. Conroy and Sashka Davis. 2017. Section mixture models for scientific document summarization. IJDL, 19:305–322. find
  11. John M Conroy, Judith D Schlesinger, and Dianne P O’Leary. 2011. Nouveau-rouge: A novelty metric for update summarization. Computational Linguistics, 37(1):1–8. find
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04805. find
  13. Hady ElSahar, Maximin Coavoux, Matthias Gallé, and Jos Rozen. 2020. Self-supervised and controlled multi-document opinion summarization. ArXiv, abs/2004.14754. find
  14. Alexander Fabbri, Irene Li, Prawat Trairatvorakul, Yijiao He, Weitai Ting, Robert Tung, Caitlin Westerfield, and Dragomir Radev. 2018. TutorialBank: A manually-collected corpus for prerequisite chains, survey extraction and resource recommendation. In ACL. find
  15. Max Grusky, Mor Naaman, and Yoav Artzi. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In NAACL-HLT. find our card
  16. Donna Harman and Paul Over. 2004. The effects of human variation in DUC summarization evaluation. In Text Summarization Branches Out, pages 10–17, Barcelona, Spain. Association for Computational Linguistics. find
  17. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in neural information processing systems, pages 1693–1701. find our card
  18. Sture Holm. 1979. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65–70. find
  19. Kokil Jaidka, Muthu Kumar Chandrasekaran, Sajal Rustagi, and Min-Yen Kan. 2018. Insights from cl-scisumm 2016: the faceted scientific document summarization shared task. IJDL, 19(2-3):163–171. find
  20. Rahul Jha, Reed Coke, and Dragomir R. Radev. 2015. Surveyor: A system for generating coherent survey articles for scientific topics. In AAAI. find
  21. Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019. CTRL: A Conditional Transformer Language Model for Controllable Generation. ArXiv, abs/1909.05858. find our card
  22. Guy Lev, Michal Shmueli-Scheuer, Jonathan Herzig, Achiya Jerbi, and David Konopnicki. 2019. Talk-summ: A dataset and scalable annotation method for scientific paper summarization based on conference talks. In ACL. find
  23. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. ACL. find our card
  24. Kun Li, Chengbo Chen, Xiaojun Quan, Qing Ling, and Yan Song. 2020. Conditional augmentation for aspect term extraction via masked sequence-to-sequence generation. ArXiv, abs/2004.14769. find
  25. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. find
  26. Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In EMNLP/IJCNLP. find
  27. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  28. Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel S. Weld. 2020. S2orc: The semantic scholar open research corpus. In Proceedings of ACL. find our card
  29. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3269– 3279, Hong Kong, China. Association for Computational Linguistics. find
  30. Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into texts. In EMNLP. find
  31. Saif M. Mohammad, Bonnie J. Dorr, Melissa Egan, Ahmed Hassan Awadallah, Pradeep Muthukrishnan, Vahed Qazvinian, Dragomir R. Radev, and David M. Zajic. 2009. Using citations to generate surveys of scientific paradigms. In HLT-NAACL. find
  32. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. pages 1797–1807. find
  33. Ani Nenkova and Rebecca J Passonneau. 2004. Evaluating content selection in summarization: The pyramid method. In NAACL. find
  34. Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. 2019. Scispacy: Fast and robust models for biomedical natural language processing. find our card
  35. Nikola I. Nikolov, Michael Pfeiffer, and Richard H. R. Hahnloser. 2018. Data-driven summarization of scientific articles. ArXiv, abs/1804.08875. — [arXiv:1804.08875](https://arxiv.org/abs/1804.08875) find
  36. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: a fast, extensible toolkit for sequence modeling. In NAACL-HLT, Demonstrations. find our card
  37. Paul Over. 2003. An introduction to duc 2003: Intrinsic evaluation of generic news text summarization systems. In Proceedings of Document Understanding Conference 2003. find
  38. Vahed Qazvinian, Dragomir R. Radev, Saif M. Mohammad, Bonnie J. Dorr, David M. Zajic, Michael Whidby, and Taesun Moon. 2013. Generating extractive summaries of scientific paradigms. J. Artif. Intell. Res., 46:165–201. find
  39. R Core Team. 2018. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. find
  40. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 21(140):1–67. find
  41. Evan Sandhaus. 2008. The new york times annotated corpus.(october 2008). ldc catalog no.: Ldc2008t19. find
  42. Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. 2019. Green ai. find our card
  43. Eva Sharma, Chen Li, and Lu Wang. 2019. Bigpatent: A large-scale dataset for abstractive and coherent summarization. In ACL. find
  44. Akhilesh Sudhakar, Bhargav Upadhyay, and Arjun Maheswaran. 2019. “transforming” delete, retrieve, generate approach for controlled text style transfer. find our card
  45. Swabha Swayamdipta, Sam Thomson, Kenton Lee, Luke Zettlemoyer, Chris Dyer, and Noah A. Smith. 2018a. Syntactic scaffolds for semantic structures. In EMNLP. find
  46. Swabha Swayamdipta, Sam Thomson, Kenton Lee, Luke Zettlemoyer, Chris Dyer, and Noah A. Smith. 2018b. Syntactic scaffolds for semantic structures. In EMNLP. find
  47. Justin Sybrandt and Ilya Safro. 2020. Cbag: Conditional biomedical abstract generation. ArXiv, abs/2002.05637. — [arXiv:2002.05637](https://arxiv.org/abs/2002.05637) find
  48. S. Teufel. 1999. Argumentative zoning information extraction from scientific text. find
  49. Richard Van Noorden. 2014. Global scientific output doubles every nine years. Nature news blog. find
  50. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, NeurIPS. find
  51. Jie Wang, Chengzhi Zhang, Mengying Zhang, and Sanhong Deng. 2018. Citationas: A tool of automatic survey generation based on citation content. Journal of Data and Information Science, 3(2):20–37. find
  52. Wen Xiao and Giuseppe Carenini. 2019. Extractive summarization of long documents by combining global and local context. In EMNLP/IJCNLP. find
  53. Michihiro Yasunaga, Jungo Kasai, Rui Zhang, Alexander Richard Fabbri, Irene Li, Dan Friedman, and Dragomir R. Radev. 2019. Scisummnet: A large annotated corpus and content-impact models for scientific paper summarization with citation networks. In AAAI. find
  54. Klaus Zechner. 1996. Fast generation of abstracts from general domain text corpora by extracting relevant sentences. In Proceedings of the 16th conference on Computational linguistics-Volume 2, pages 986– 989. find
  55. Chrysoula Zerva, Minh-Quoc Nghiem, Nhung T. H. Nguyen, and S. Ananiadou. 2020. Cited text span identification for scientific summarisation using pre-trained encoders. Scientometrics, pages 1 – 29. find
  56. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2019. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. ArXiv, abs/1912.08777. find our card
  57. Hao Zheng and Mirella Lapata. 2019. Sentence centrality revisited for unsupervised summarization. In ACL. find
  58. Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2020. Extractive summarization as text matching. ACL. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cachola-2020-tldr-extreme-summarization-scientific-documents/