Elamur Research⌕
☰
home / papers / 2019 / BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization

BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization

Paper · 2019 · Eva Sharma, Chen Li and Lu Wang

Summary

News-based summarization corpora have three weaknesses. Important content clusters near the start of each article, reference summaries copy long stretches of the source, and the summaries have a flat discourse structure where entities seldom reappear. These traits push models toward extraction and positional shortcuts. To address this, the authors built BIGPATENT, a corpus of about 1.3 million U.S. patents. Each patent's description is the input and its abstract is the reference summary. They compared it with CNN/DM, NYT, Newsroom, XSum, arXiv and PubMed and found three differences. Salient words are spread across the whole document, summaries reuse fewer and shorter source fragments, and entities recur across sentences more often. They then benchmarked extractive and abstractive systems. Taking the opening sentences works poorly. SentRewriting scored best, but neural generators often invented content and repeated entities far more than human writers do. The resource matters because it demands modeling of the entire input, understanding of entities, and discourse-level planning.

Key points

  • Corpus: 1,341,362 patent documents filed after 1971, covering nine technology areas.
    • Sourced from Google Patents Public Datasets through BigQuery.
    • Released under CC BY 4.0.
  • Size: inputs average roughly 3,573 words and summaries about 116.5 words (3.5 sentences).
  • Spread of salient content: covering a summary's salient words requires reading about 80% of a BIGPATENT input. The figures for news datasets are 63% (CNN/DM), 57% (XSum), 53% (NYT) and 29% (Newsroom).
  • Entity recurrence: about 24% of entities in BIGPATENT summaries appear more than once, versus about 4% for CNN/DM.
  • Benchmarks:
    • Nearly every system beats LEAD-3.
    • TextRank and LexRank outperform RNN-EXT RL.
    • SentRewriting achieves the top ROUGE scores.
    • Seq2Seq mentions 22.6% of entities more than three times, compared with 4.0% in human-written summaries.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢TencentShenzhen · China66 works · 17,945 citations🎓Northeastern UniversityBoston · United States28 works · 6,164 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Summarization · 159 works

Cite

Cite as Eva Sharma, Chen Li and Lu Wang. 2019. BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1212

@inproceedings{sharma2019bigpatent,
  title = {{BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization}},
  author = {Eva Sharma, Chen Li and Lu Wang},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1212},
  url = {https://doi.org/10.18653/v1/p19-1212}
}

References

37 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
  2. Federico Barrios, Federico López, Luis Argerich, and Rosa Wachenchauzer. 2016. Variations of the similarity function of textrank for automated summarization. arXiv preprint arXiv:1602.03606. — [arXiv:1602.03606](https://arxiv.org/abs/1602.03606) find
  3. Regina Barzilay and Mirella Lapata. 2008. Modeling local coherence: An entity-based approach. Computational Linguistics, 34(1).
  4. Regina Barzilay and Lillian Lee. 2004. Catching the drift: Probabilistic content models, with applications to generation and summarization. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004.
  5. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. ” O’Reilly Media, Inc.”. find
  6. Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. In Proceedings of the Association for the Advancement of Artificial Intelligence (AAAI). find
  7. Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 675–686. Association for Computational Linguistics.
  8. David Cinciruk. 2015. Patent summarization and paraphrasing. http://www.ece.drexel.edu/ walsh/David_PatentSummarization. pdf. ↗
  9. Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational…
  10. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul):2121–2159. find
  11. Günes Erkan and Dragomir R Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of artificial intelligence research, 22:457–479. find
  12. Lisa Fan, Dong Yu, and Lu Wang. 2018. Robust neural abstractive summarization systems and evaluation against adversarial information. In Workshop on Interpretability and Robustness in Audio, Speech, and Language (IRASL). Neural Information Processing Systems. find
  13. Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018a. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098–4109. Association for Computational Linguistics.
  14. Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018b. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098–4109.
  15. Google. 2018. Google patents public datasets: connecting public, paid, and private patent data. https://console.cloud.google.com/marketplace/details/ google_patents_public_datasets/ google-patents-public-data?_ga=2. 148226999.-1648178590.1534442735& pli=1. Accessed: 2018-08-30. ↗
  16. Barbara J. Grosz, Scott Weinstein, and Aravind K. Joshi. 1995. Centering: A framework for modeling the local coherence of discourse. Computational Linguistics, 21(2).
  17. Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pa…
  18. Aria Haghighi and Lucy Vanderwende. 2009. Exploring content models for multi-document summarization. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 362–370. Association for Computational…
  19. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1693– 1701.
  20. Chin-Yew Lin and Eduard Hovy. 2003. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics.
  21. Edward Loper and Steven Bird. 2002. Nltk: The natural language toolkit. In Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics. find
  22. Kathleen R McKeown. 1985. Discourse strategies for generating natural-language text. Artificial Intelligence, 27(1):1–41. find
  23. Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into text. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing. find
  24. Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280–290. Association for Computational Lin… find
  25. Courtney Napoles, Matthew Gormley, and Benjamin Van Durme. 2012. Annotated gigaword. In Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction (AKBC-WEKEX), pages 95–100. Association for Computational Linguistics.
  26. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807. Association for Compu… find
  27. Ani Nenkova and Lucy Vanderwende. 2005. The impact of frequency on summarization. Microsoft Research, Redmond, Washington, Tech. Rep. MSR-TR-2005, 101. find
  28. Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304.
  29. Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379–389. find
  30. Evan Sandhaus. 2008. The new york times annotated corpus. Linguistic Data Consortium, Philadelphia, 6(12):e26752. find
  31. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073– 1083. Association for Computational Linguistic… find
  32. Michael Strube and Udo Hahn. 1999. Functional centering grounding referential coherence in information structure. Computational Linguistics, 25(3).
  33. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112. find
  34. Jiwei Tan, Xiaojun Wan, and Jianguo Xiao. 2017. Abstractive document summarization with a graph-based attentional neural model. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1171–1181. Association for Computational Linguisti…
  35. USPTO. 2013. Cooperative patent classification scheme. https://www.uspto.gov/web/ patents/classification/cpc/html/ cpc.html. Accessed: 2018-08-30. ↗
  36. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. find
  37. Yinfei Yang, Forrest Bao, and Ani Nenkova. 2017. Detecting (un)important content for single-document news summarization. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 707–712. Association for Computationa… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sharma-2019-bigpatent-large-scale-dataset-abstractive/