Elamur Research
☰
home / papers / 2022 / GLM: General Language Model Pretraining with Autoregressive Blank Infilling

GLM: General Language Model Pretraining with Autoregressive Blank Infilling

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2022.acl-long.26.pdf

Summary

Each common pretraining family has a weakness. GPT-style left-to-right models generate long text well but read context in only one direction. BERT-style masked encoders excel at understanding but cannot generate text directly. Encoder-decoder systems like T5 need extra parameters to match RoBERTa-class models. The authors propose GLM, a single Transformer that learns to fill in blanks autoregressively. Contiguous spans are cut from a passage, each replaced by one [MASK], and the model regenerates them token by token in a randomly shuffled span order. A 2D positional encoding records where each span sits in the corrupted text and each token's offset inside its span. Attention masks keep the corrupted context bidirectional and the generated part unidirectional. Varying the number and length of blanks produces objectives for NLU, seq2seq or long-form generation, and NLU tasks are recast as cloze questions. Given equal model size and data, GLM outperforms BERT, T5 and GPT. This shows that one pretrained model can serve both understanding and generation.

Key points

  • On SuperGLUE, GLM beats BERT by 4.6–5.0% at equal parameters and compute. Trained on a 158GB corpus, it also surpasses RoBERTa and BART.
  • Spans for the NLU objective follow a Poisson length distribution (λ = 3) and cover about 15% of tokens. The authors found this ratio crucial for downstream NLU.
  • Multi-task variants add two objectives:
    • a document-level objective that masks a single span covering 50–100% of the text;
    • a sentence-level objective that masks whole sentences. A multi-task model with 1.25× the parameters of BERT Large performs best across task types.
  • Ablations show the main design choices matter:
    • cloze-style finetuning adds about 7 points for the large model;
    • removing span shuffling sharply hurts SuperGLUE;
    • using several sentinel tokens instead of one [MASK] lowers scores;
    • dropping the second positional dimension weakens long-text generation.
  • Code and pretrained models are released at github.com/THUDM/GLM.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

494 papers of our library cite it (of the 140673 whose bibliographies we read); 40 of them have a page here, 454 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.acl-long.26 · arXiv 2103.10360

@inproceedings{du2022glm,
  title = {{GLM: General Language Model Pretraining with Autoregressive Blank Infilling}},
  author = {Du, Zhengxiao and Qian, Yujie and Liu, Xiao and Ding, Ming and Qiu, Jiezhong and Yang, Zhilin and Tang, Jie},
  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.26},
  eprint = {2103.10360},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2022.acl-long.26}
}

References

52 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ben Athiwaratkun, Cicero dos Santos, Jason Krone, and Bing Xiang. 2020. Augmented natural language for generative sequence labeling. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 375–385. find
  2. Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Jianfeng Gao, Songhao Piao, Ming Zhou, and Hsiao-Wuen Hon. 2020. Unilmv2: Pseudo-masked language models for unified language model pre-training. In ICML 2020, volume 119, pages 642–652. find our card
  3. Bin Bi, Chenliang Li, Chen Wu, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, and Luo Si. 2020. PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation. In EMNLP 2020, pages 8681–8691. find
  4. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… find
  5. Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14. find
  6. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. In ICLR 2020. find
  7. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, pages 177–190. Springer. find
  8. Michael Denkowski and Alon Lavie. 2014. Meteor Universal: Language Specific Translation Evaluation for Any Target Language. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 376–380. find our card
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL 2019, pages 4171–4186. find
  10. Chris Donahue, Mina Lee, and Percy Liang. 2020. Enabling language models to fill in the blanks. pages 2492–2501. find
  11. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In NeurIPS 2019, pages 13042– 13054. find our card
  12. Xinya Du, Junru Shao, and Claire Cardie. 2017. Learning to Ask: Neural Question Generation for Reading Comprehension. In ACL 2017, pages 1342–1352. find our card
  13. Aaron Gokaslan and Vanya Cohen. 2019. Openwebtext corpus. http://Skylion007.github. io/OpenWebTextCorpus. ↗
  14. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decoding-enhanced bert with disentangled attention. ArXiv, abs/2006.03654. find our card
  15. Dan Hendrycks and Kevin Gimpel. 2016. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, abs/1606.08415. find
  16. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving Pre-training by Representing and Predicting Spans. Trans. Assoc. Comput. Linguistics, 8:64–77. find our card
  17. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In ICLR 2020. find
  18. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In ACL 2020, pages 7871–7880. find our card
  19. Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. pages 74–81. find
  20. Yang Liu and Mirella Lapata. 2019. Text Summarization with Pretrained Encoders. In EMNLP 2019, pages 3730–3740. find
  21. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. find
  22. Joel Mackenzie, Rodger Benham, Matthias Petri, Johanne R. Trippas, J. Shane Culpepper, and Alistair Moffat. 2020. find
  23. CC-News-En: A Large English News Corpus. In CIKM 2020, pages 3077–3084. find
  24. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization. In EMNLP 2018, pages 1797–1807. find
  25. Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, Rishita Anubhai, Cicero Nogueira dos Santos, Bing Xiang, and Stefano Soatto. 2020. Structured Prediction as Translation between Augmented Natural Languages. find
  26. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016. The LAMBADA dataset: Word prediction requiring a broad discourse context. In ACL 2016. find our card
  27. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A Method for Automatic Evaluation of Machine Translation. In ACL 2002, pages 311–318. find our card
  28. Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E. Hinton. 2017. Regularizing neural networks by penalizing confident output distributions. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings… find our card
  29. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018a. Improving Language Understanding by Generative Pre-Training. find
  30. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2018b. Language models are unsupervised multitask learners. find
  31. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. J. Mach. Learn. Res., 21:140:1– 140:67. find
  32. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know What You Don’t Know: Unanswerable Questions for SQuAD. In ACL 2018, pages 784–789. find our card
  33. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In EMNLP 2016, pages 2383–2392. find our card
  34. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In KDD 2020, pages 3505–3506. find
  35. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In EMNLP 2015, pages 379– 389. find
  36. Timo Schick and Hinrich Schütze. 2020a. Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference. pages 255–269. find our card
  37. Timo Schick and Hinrich Schütze. 2020b. It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners. pages 2339–2352. find our card
  38. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get To The Point: Summarization with Pointer-Generator Networks. In ACL 2017, pages 1073–1083. find
  39. Towards story-like visual explanations by watching movies and reading books. In ICCV 2015, pages 19– 27. find
  40. Tianxiao Shen, Victor Quach, Regina Barzilay, and Tommi S. Jaakkola. 2020. Blank language models. pages 5186–5198. find
  41. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. CoRR, abs/1909.08053. find
  42. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In EMNLP 2013, pages 1631–1642. find
  43. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019. MASS: Masked Sequence to Sequence Pre-training for Language Generation. In ICML 2019, volume 97, pages 5926–5936. find our card
  44. Trieu H. Trinh and Quoc V. Le. 2019. A Simple Method for Commonsense Reasoning. arXiv:1806.02847 [cs]. find
  45. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. In NeurIPS 2019, pages 3261–3275. find our card
  46. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In ICLR 2019, pages 353–355. find our card
  47. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In NAACL 2018, pages 1112–1122. find
  48. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized Autoregressive Pretraining for Language Understanding. In NeurIPS 2019, pages 5754–5764. find our card
  49. Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg-Kirkpatrick. 2017. Improved variational autoencoders for text modeling using dilated convolutions. In ICML 2017, volume 70, pages 3881–3890. find
  50. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020. PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization. In ICML 2020, pages 11328–11339. find our card
  51. Wanrong Zhu, Zhiting Hu, and Eric Xing. 2019. Text infilling. arXiv preprint arXiv:1901.00158. — [arXiv:1901.00158](https://arxiv.org/abs/1901.00158) find
  52. Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: find