Elamur Research
☰
home / papers / 2019 / PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization

PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1912.08777

Summary

Pre-trained Transformer encoder-decoders had already improved text generation, but no pre-training task had been designed specifically for abstractive summarization, and models were rarely compared across many domains. The authors introduce PEGASUS, which is pre-trained on a self-supervised task called Gap Sentences Generation (GSG). In GSG, whole sentences judged important are hidden from a document, and the model must reproduce them, joined into one target, from the remaining text. This closely mimics writing a summary. Importance is estimated with ROUGE1-F1 between each sentence and the rest of the document. This "principal" selection outperformed picking the leading or random sentences. A 568M-parameter model was pre-trained on C4 and on HugeNews, a newly assembled collection of news-like articles. It was then fine-tuned on 12 benchmarks covering news, science, stories, how-to guides, emails, patents and legislative bills. It equalled or beat the previous best ROUGE on every one, adapted well from very few labelled pairs, and received human ratings comparable to reference summaries. This makes strong summarization practical where annotated data is scarce.

Key points

  • Pre-training objective. GSG can be combined with BERT-style MLM. Principal sentence selection comes in four variants: independent or sequential scoring (Ind/Seq), and original or set-based n-gram counting (Orig/Uniq). The gap sentences ratio (GSR) acts like a mask rate.
  • Benchmark results. All 12 datasets saw state-of-the-art ROUGE: arXiv, PubMed, BIGPATENT, BillSum, AESLC, Gigaword, Multi-News, XSum, WikiHow, CNN/DailyMail, Reddit TIFU and NEWSROOM.
  • Low-resource setting. With only 1000 fine-tuning examples, the model exceeded the prior state of the art on 6 datasets.
  • Human evaluation. Amazon Mechanical Turk workers rated summaries on a 1–5 scale, with significance checked by a paired t-test. Model outputs were rated at least as good as references on XSum, CNN/DailyMail and Reddit TIFU. Reddit TIFU required full supervision to reach that level.
  • Improved variant and data checks. A later "mixed, stochastic" model was trained on C4 plus HugeNews for 1.5M steps, with GSR sampled between 15% and 45%. It scored 47.60/24.83/39.64 ROUGE-1/2/L on XSum. Test-set overlap with C4 (15–20% for XSum) changed scores by under 1%. Code and checkpoints are available at github.com/google-research/pegasus.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

846 papers of our library cite it (of the 140673 whose bibliographies we read); 25 of them have a page here, 821 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1912.08777 · arXiv 1912.08777

@misc{zhang2019pegasus,
  title = {{PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization}},
  author = {Zhang, Jingqing and Zhao, Yao and Saleh, Mohammad and Liu, Peter J.},
  year = {2019},
  doi = {10.48550/arxiv.1912.08777},
  eprint = {1912.08777},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1912.08777}
}

References

49 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014. find
  2. Cohan, A., Dernoncourt, F., Kim, D. S., Bui, T., Kim, S., Chang, W., and Goharian, N. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Languag… find our card
  3. Dai, A. M. and Le, Q. V. Semi-supervised sequence learning. In Cortes, C., Lawrence, N. D., Lee, D. D., Sugiyama, M., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 28, pp. 3079–3087. Curran Associates, Inc., 2015. URL http://papers.nips.cc/paper/5949-semi-supervised-seque… ↗
  4. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and… find
  5. Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., and Hon, H.-W. Unified language model pre-training for natural language understanding and generation. In 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), 2019. find our card
  6. Fabbri, A., Li, I., She, T., Li, S., and Radev, D. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1074–1084, Florence, Italy, July 2019. Association for… find our card
  7. Goodman, S., Lan, Z., and Soricut, R. Multi-stage pretraining for abstractive summarization, 2019. find
  8. Graff, D., Kong, J., Chen, K., and Maeda, K. English gigaword. Linguistic Data Consortium, Philadelphia, 4 (1):34, 2003. find
  9. Grusky, M., Naaman, M., and Artzi, Y. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 2018. doi: 10… find our card
  10. Hermann, K. M., Kocisky, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., and Blunsom, P. Teaching machines to read and comprehend. In Advances in neural information processing systems, pp. 1693–1701, 2015. find our card
  11. Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural Comput., 9(8):1735–1780, November 1997. ISSN 0899-7667. doi: 10.1162/neco.1997.9.8.1735. URL http://dx.doi.org/10.1162/neco.1997.9.8.1735. ↗
  12. Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., and Levy, O. SpanBERT: Improving pre-training by representing and predicting spans. arXiv preprint arXiv:1907.10529, 2019. find our card
  13. Khandelwal, U., Clark, K., Jurafsky, D., and Kaiser, L. Sample efficient text summarization using a single pre-trained transformer. arXiv preprint arXiv:1905.08836, 2019. — [arXiv:1905.08836](https://arxiv.org/abs/1905.08836) find
  14. Kim, B., Kim, H., and Kim, G. Abstractive summarization of Reddit posts with multi-level memory networks. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 2519–2531… find
  15. Klimt, B. and Yang, Y. The enron corpus: A new dataset for email classification research. In Proceedings of the 15th European Conference on Machine Learning, ECML’04, pp. 217–226, Berlin, Heidelberg, 2004. Springer-Verlag. ISBN 3-540-23105-6, 978-3-540-23105-9. doi: 10.1007/978-3-540-30115-8 22. URL… — [doi:10.1007/978-3-540-30115-8](https://doi.org/10.1007/978-3-540-30115-8) find
  16. Kornilova, A. and Eidelman, V. BillSum: A corpus for automatic summarization of US legislation. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pp. 48–56, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-5406. URL https://www.aclw… ↗
  17. Koupaee, M. and Wang, W. Y. Wikihow: A large scale text summarization dataset. arXiv preprint arXiv:1810.09305, 2018. — [arXiv:1810.09305](https://arxiv.org/abs/1810.09305) find
  18. Kryscinski, W., Keskar, N. S., McCann, B., Xiong, C., and Socher, R. Neural text summarization: A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), p… find our card
  19. Kudo, T. Subword regularization: Improving neural network translation models with multiple subword candidates. arXiv preprint arXiv:1804.10959, 2018. find
  20. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019. find our card
  21. Lin, C.-Y. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https: //www.aclweb.org/anthology/W04-1013. find
  22. Liu, P. J., Saleh, M., Pot, E., Goodrich, B., Sepassi, R., Kaiser, L., and Shazeer, N. Generating wikipedia by summarizing long sequences. In International Conference on Learning Representations, 2018. URL https: //openreview.net/forum?id=Hyg0vbWC-. find our card
  23. Nallapati, R., Zhou, B., dos Santos, C., Gulçehre, Ç., and Xiang, B. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pp. 280–290, Berlin, Germany, August 2016. Association for Comput… find
  24. Nallapati, R., Zhai, F., and Zhou, B. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI’17, pp. 3075–3081. AAAI Press, 2017. URL http://dl.acm.org/citation.cfm?id… ↗ our card
  25. Narayan, S., Cohen, S. B., and Lapata, M. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797–1807, Brussels, Belgium, October-November 20… find
  26. Paulus, R., Xiong, C., and Socher, R. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304, 2017. find our card
  27. Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assets/researchcovers/languageunsupervised/language understanding paper. pdf, 2018a. find
  28. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. 2018b. URL https: //d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf. find
  29. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer, 2019. find
  30. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. Squad: 100,000+ questions for machine comprehension of text. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016. doi: 10.18653/v1/d16-1264. URL http://dx.doi.org/10.18653/v1/D16-1264. ↗ our card
  31. Ramachandran, P., Liu, P., and Le, Q. Unsupervised pretraining for sequence to sequence learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 383–391, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/D… find
  32. Rothe, S., Narayan, S., and Severyn, A. Leveraging pre-trained checkpoints for sequence generation tasks. arXiv preprint arXiv:1907.12461, 2019. find our card
  33. Rush, A. M., Chopra, S., and Weston, J. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 379–389, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10.18653/… find
  34. See, A., Liu, P. J., and Manning, C. D. Get to the point: Summarization with pointer-generator networks. CoRR, abs/1704.04368, 2017. URL http://arxiv.org/abs/1704.04368. ↗
  35. Sennrich, R., Haddow, B., and Birch, A. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1715–1725, Berlin, Germany, August 2016. Association for Computational Linguist… find
  36. Sharma, E., Li, C., and Wang, L. BIGPATENT: A large-scale dataset for abstractive and coherent summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2204–2213, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.1865… find
  37. Shazeer, N. and Stern, M. Adafactor: Adaptive learning rates with sublinear memory cost. arXiv preprint arXiv:1804.04235, 2018. find
  38. Shi, T., Wang, P., and Reddy, C. K. LeafNATS: An open-source toolkit and live demo system for neural abstractive text summarization. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pp. 66–71, Minneapolis, Minnesot… find
  39. Song, K., Tan, X., Qin, T., Lu, J., and Liu, T.-Y. Mass: Masked sequence to sequence pre-training for language generation. In International Conference on Machine Learning, pp. 5926–5936, 2019. find our card
  40. Subramanian, S., Li, R., Pilault, J., and Pal, C. On extractive and abstractive neural document summarization with transformer language models. arXiv preprint arXiv:1909.03186, 2019. find
  41. Sutskever, I., Vinyals, O., and Le, Q. V. Sequence to sequence learning with neural networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, NIPS’14, pp. 3104–3112, Cambridge, MA, USA, 2014. MIT Press. URL http://dl.acm.org/citation.cfm?id=29… ↗
  42. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. In Advances in neural information processing systems, pp. 5998–6008, 2017. find
  43. Völske, M., Potthast, M., Syed, S., and Stein, B. TL;DR: Mining Reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pp. 59–63, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/W17-4508. URL https:… find
  44. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. Glue: A multi-task benchmark and analysis platform for natural language understanding. Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 2018. doi: 10.18653/v1/w18-5446. URL htt… find our card
  45. Welleck, S., Kulikov, I., Roller, S., Dinan, E., Cho, K., and Weston, J. Neural text generation with unlikelihood training. arXiv preprint arXiv:1908.04319, 2019. find our card
  46. Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016. find
  47. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., and Le, Q. V. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, pp. 5754– 5764, 2019. URL http://papers.nips.cc/paper/8812-xlnet-generalized-autoregressive-pre… ↗ our card
  48. Zhang, R. and Tetreault, J. This email could save your life: Introducing the task of email subject line generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 446–456, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.… find
  49. Zhong, M., Liu, P., Wang, D., Qiu, X., and Huang, X. Searching for effective neural extractive summarization: What works and whats next. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19-1100. URL http://dx.doi.org/10.18653/v1/p19-110… ↗