Elamur Research
☰
home / papers / 2021 / A Robustly Optimized BERT Pre-training Approach with Post-training

A Robustly Optimized BERT Pre-training Approach with Post-training

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2021.ccl-1.108.pdf

Summary

Pre-trained language models like BERT usually move directly from unsupervised pre-training to fine-tuning. That jump can be fragile when the target task has few labelled examples or belongs to a specialised domain such as finance. The authors add an intermediate "post-training" stage, which produces a three-stage scheme and a model called PPBERT. After standard BERT pre-training, the model is trained further on supervised data from related tasks: CoQA (question answering), SNLI (inference) and YELP (sentiment). This uses a multi-task continual learning procedure. Each new task starts from the previously learned weights and is trained together with earlier tasks to reduce forgetting. Every task gets a fixed iteration budget, and gradients are accumulated over sub-batches. Only after this is the model fine-tuned on the target task. PPBERT beats BERT baselines on GLUE, SuperGLUE and several QA benchmarks. Applying the same step to ALBERT (PPALBERT) gives leading SQuAD and other QA/NLI scores. The approach is a model-agnostic add-on for low-resource and domain-specific settings, and the code and checkpoints are released.

Key points

  • PPBERT outperforms BERT on all 16 GLUE and SuperGLUE tasks. It reaches a GLUE average of 87.02 (+2.97) and a SuperGLUE score of 74.55 (+5.55). The abstract claims 24 tasks overall, but the listed breakdown adds up to 22.
  • Post-training corpora are CoQA, SNLI and YELP, used simultaneously. The PPALBERT LARGE-QA variant is additionally post-trained on SearchQA.
  • On the SQuAD 1.1 dev set, post-training lifts BERT from 84.1/90.9 to 85.2/92.1 EM/F1. PPALBERT LARGE-QA scores 89.7/95.1 on SQuAD 1.1 and 87.7/90.5 on SQuAD 2.0.
  • PPALBERT variants slightly exceed ALBERT LARGE on NewsQA, TriviaQA, HotpotQA, RACE, QNLI and MNLI-m. Examples are 95.9 on QNLI and 90.9 on MNLI-m.
  • The authors report particularly strong gains in small-data settings, including experiments that use only 1/20 of the training set.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

182 papers of our library cite it (of the 140673 whose bibliographies we read).

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zhuang2021robustly,
  title = {{A Robustly Optimized BERT Pre-training Approach with Post-training}},
  author = {Zhuang, Liu and Wayne, Lin and Ya, Shi and Jun, Zhao},
  booktitle = {Proceedings of the 20th Chinese National Conference on Computational Linguistics},
  year = {2021},
  url = {https://aclanthology.org/2021.ccl-1.108/}
}

References

29 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samuel R. Bowman, Ellie Pavlick, and Edouard Grave. 2018. Looking for elmo’s friends: Sentence-level pre-training beyond language modeling. CoRR, abs/1812.10860. — [arXiv:1812.10860](https://arxiv.org/abs/1812.10860) find
  2. Zhiyuan Chen and Bing Liu. 2018. Lifelong Machine Learning, Second Edition. Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool Publishers. find
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association… find
  4. Ming Hou, Xinqi Chen, Shifeng Huang, Shengli Xie, and Guoxu Zhou. 2020. Generalizing deep multi-task learning with heterogeneous structured networks. In In Proceedings of ICLR. find
  5. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min-Yen Kan, editors, Proceedings Proceedings of the 20th China National Conference on Computational Linguistics, pa… find our card
  6. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2019. Spanbert: Improving pre-training by representing and predicting spans. CoRR, abs/1907.10529. find our card
  7. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard H. Hovy. 2017. RACE: large-scale reading comprehension dataset from examinations. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 20… find our card
  8. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. find
  9. Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. find
  10. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. find
  11. Xiaodong Liu, Yu Wang, Jianshu Ji, Hao Cheng, Xueyun Zhu, Emmanuel Awa, Pengcheng He, Weizhu Chen, Hoifung Poon, Guihong Cao, and Jianfeng Gao. 2020a. The microsoft toolkit of multi-task deep neural networks for natural language understanding. In Asli Celikyilmaz and Tsung-Hsien Wen, editors, Procee… find
  12. Zhuang Liu, Degen Huang, Kaiyu Huang, Zhuang Li, and Jun Zhao. 2020b. Finbert: A pre-trained financial language representation model for financial text mining. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, January 5-10, 2021, Yokohama, Japa… find
  13. Zhuang Liu, Kaiyu Huang, Degen Huang, Zhuang Liu, and Jun Zhao. 2020c. Dual head-wise coattention network for machine comprehension with multiple-choice questions. In Mathieu d’Aquin, Stefan Dietze, Claudia Hauff, Edward Curry, and Philippe Cudré-Mauroux, editors, CIKM ’20: The 29th ACM Internation… find
  14. Macedo Maia, Siegfried Handschuh, Andre Freitas, Brian Davis, Ross Mcdermott, Manel Zarrouk, Alexandra Balahur, and Ross Mc-Dermott, editors. 2018. In Proceedings of WWW. ACM. find
  15. L Yasuhito Ohsugi, Itsumi Saito, Kyosuke Nishida, Hisako Asano, and Junji Tomita. 2019. A simple but effective method to incorporate multi-turn context with BERT for conversational machine comprehension. CoRR, abs/1905.12848. find
  16. CC German Ignacio Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. 2019. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54–71. find our card
  17. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Marilyn A. Walker, Heng Ji, and Amanda Stent, editors, Proceedings of the 2018 Conference of the North American Chapter of the Associati… find
  18. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. In Proceedings of Technical report, OpenAI. find
  19. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In Jian Su, Xavier Carreras, and Kevin Duh, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texa… find our card
  20. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for squad. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volu… find our card
  21. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. Coqa: A conversational question answering challenge. Trans. Assoc. Comput. Linguistics, 7:249–266. find our card
  22. Proceedings of the 20th China National Conference on Computational Linguistics, pages 1218-1227, Hohhot, China, Augest 13 - 15, 2021. (c) Technical Committee on Computational Linguistics, Chinese Information Processing Society of China Yu Sun, Shuohuan Wang, and Yu-Kun Li. 2019. ERNIE: enhanced repr… find our card
  23. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. Newsqa: A machine comprehension dataset. Proc. of the 2ndWorkshop on Representation Learning for NLP. find our card
  24. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. Superglue: A stickier benchmark for general-purpose language understanding systems. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily… find our card
  25. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRevi… find our card
  26. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors, Proceedings of the… find our card
  27. Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019a. End-to-end open-domain question answering with bertserini. In Waleed Ammar, Annie Louis, and Nasrin Mostafazadeh, editors, Proceedings of the 2019 Conference of the North American Chapter of the Associ… find our card
  28. CC L Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019b. Xlnet: Generalized autoregressive pretraining for language understanding. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, edito… find our card
  29. Proceedings of the 20th China National Conference on Computational Linguistics, pages 1218-1227, Hohhot, China, Augest 13 - 15, 2021. (c) Technical Committee on Computational Linguistics, Chinese Information Processing Society of China find