Elamur Research
☰
home / papers / 2020 / MPNet: Masked and Permuted Pre-training for Language Understanding

MPNet: Masked and Permuted Pre-training for Language Understanding

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2004.09297

Summary

BERT and XLNet each use a pre-training objective with its own weakness. BERT's masked language modeling (MLM) predicts each masked token independently of the others. XLNet's permuted language modeling (PLM) captures dependencies between predicted tokens, but it hides the positions of tokens that come later. As a result, the model sees an incomplete sentence during pre-training, which does not match what it gets at fine-tuning. The authors show that both objectives can be described in one framework: a permuted sequence is split into a part used as context and a part to be predicted. Building on this, MPNet predicts tokens one after another in permuted order, as PLM does. It also adds mask symbols that carry the positions of all predicted tokens (called "position compensation"), so the model always knows the full sentence layout. This is implemented with two-stream self-attention. MPNet was pre-trained on over 160GB of text and fine-tuned on GLUE, SQuAD, RACE and IMDB. With the same base model size, it beats BERT, XLNet and RoBERTa. The work matters because it combines the strengths of both objectives in one model, and the model is publicly released.

Key points

  • On GLUE dev sets (BERT-BASE setting), MPNet scores 4.8, 3.4 and 1.5 points higher than BERT, XLNet and RoBERTa respectively.
  • On RACE, MPNet pre-trained on only Wikipedia + BooksCorpus (16GB) reaches 70.4 accuracy, compared with 65.0 for BERT and 66.8 for XLNet. With the full 160GB corpus it reaches 76.1. On IMDB, its error rate drops to 4.4 (BERT: 5.4, XLNet: 4.9).
  • Ablation (16GB, 1M steps): removing position compensation turns MPNet into PLM and costs 0.6–2.3 points. Reducing it to plain MLM costs 0.5–3.7 points, measured on SQuAD v1.1/v2.0, MNLI and SST-2.
  • MPNet also clearly beats BERT, XLNet and RoBERTa on SQuAD v1.1 and v2.0 (single model, no data augmentation).
  • Published at NeurIPS 2020. The code and pre-trained models are available at github.com/microsoft/MPNet.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

430 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 421 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2004.09297 · arXiv 2004.09297

@misc{song2020mpnet,
  title = {{MPNet: Masked and Permuted Pre-training for Language Understanding}},
  author = {Song, Kaitao and Tan, Xu and Qin, Tao and Lu, Jianfeng and Liu, Tie-Yan},
  year = {2020},
  doi = {10.48550/arxiv.2004.09297},
  eprint = {2004.09297},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2004.09297}
}

References

32 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assets/researchcovers/languageunsupervised/language understanding paper. pdf, 2018. find
  2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, pages 4171–4186, 2019. find
  3. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1(8), 2019. find
  4. Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. MASS: Masked sequence to sequence pre-training for language generation. In ICML, pages 5926–5936, 2019. find our card
  5. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In NIPS, pages 5754–5764, 2019. find our card
  6. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. In NIPS, pages 13042–13054, 2019. find our card
  7. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. find
  8. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019. find
  9. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. arXiv preprint arXiv:1907.10529, 2019. find our card
  10. Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555, 2020. find
  11. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 5998–6008, 2017. find
  12. Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu. Pre-training with whole word masking for chinese BERT. CoRR, abs/1906.08101, 2019. find our card
  13. Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In NAACL, pages 464–468, June 2018. find our card
  14. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019. find
  15. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In ICCV, pages 19–27, December 2015. find our card
  16. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. find
  17. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692, 2019. find
  18. Trieu H. Trinh and Quoc V. Le. A simple method for commonsense reasoning. CoRR, abs/1806.02847, 2018. find
  19. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014. find
  20. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019. find our card
  21. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. CoRR, abs/1805.12471, 2018. find
  22. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, pages 1631–1642, October 2013. find
  23. William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005. find our card
  24. Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, August 2017. find
  25. Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL, pages 1112–1122, June 2018. find
  26. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP, pages 2383–2392, 2016. find our card
  27. Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, pages 177–190, 2006. find
  28. Hector J. Levesque, Ernest Davis, and Leora Morgenstern. The Winograd Schema Challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning, pages 552–561. Rome, Italy, 2012. find
  29. Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for SQuAD. In ACL, pages 784–789, 2018. find our card
  30. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. In EMNLP, pages 785–794, 2017. find our card
  31. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, June 2011. find our card
  32. Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. How to fine-tune BERT for text classification? CoRR, abs/1905.05583, 2019. find our card