Elamur Research
☰
home / papers / 2020 / UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training

UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2002.12804

Summary

Pre-trained Transformers typically commit to one objective. Autoencoding (BERT-style), which predicts masked tokens independently, suits understanding tasks. Autoregressive objectives suit generation. UniLMv2 trains a single model to do both. Its pseudo-masked language model (PMLM) handles each corrupted position in two ways. Conventional [M] masks drive bidirectional autoencoding (AE). Appended [P] pseudo-mask tokens, carrying the original positions, drive a partially autoregressive (PAR) objective, where each factorization step predicts either one token or a whole span. Carefully designed self-attention masks and position embeddings let both losses be computed in one forward pass, with the context encodings shared rather than recomputed. AE trains the model as a bidirectional encoder, and PAR trains it as a sequence-to-sequence decoder. Span-level prediction also pushes the model away from local shortcuts toward longer-range dependencies. After fine-tuning, the BASE-size UniLMv2 reports state-of-the-art results on understanding benchmarks (SQuAD, MNLI, SST-2) and generation tasks (abstractive summarization, question generation). It often beats larger models, which shows that the two objectives complement each other.

Key points

  • Masking scheme: 15% of tokens are masked. 40% of the time the masked unit is a contiguous n-gram of 2–6 tokens, otherwise a single token. One factorization order is randomly sampled per text, similar to XLNet's permutation idea but allowing span-level steps.
  • Ablation setup: Models were trained on English Wikipedia + BookCorpus for 1M steps (batch size 256, Adam, lr 1e-4, 10K warmup steps). AE+PAR came out best of the five objectives (AE, AR, PAR, AE+AR, AE+PAR).
  • Ablation results: The best AE+PAR UniLMv2 BASE configuration reaches 92.0 F1 on SQuAD v1.1 (BERT BASE: 88.5), 83.6 F1 on SQuAD v2.0, 86.1 on MNLI and 93.2 on SST-2.
  • Generation results: On question generation, UniLMv2 BASE outperforms UniLM LARGE despite having about three times fewer parameters. On summarization (ROUGE), it beats other BASE-size pre-trained models such as MASS, T5 and BERT-SumAbs.
  • Availability: Code and models are released at github.com/microsoft/unilm (Microsoft Research with Harbin Institute of Technology).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

126 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 118 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2002.12804 · arXiv 2002.12804

@misc{bao2020unilmv2,
  title = {{UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training}},
  author = {Bao, Hangbo and Dong, Li and Wei, Furu and Wang, Wenhui and Yang, Nan and Liu, Xiaodong and Wang, Yu and Piao, Songhao and Gao, Jianfeng and Zhou, Ming and Hon, Hsiao-Wuen},
  year = {2020},
  doi = {10.48550/arxiv.2002.12804},
  eprint = {2002.12804},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2002.12804}
}

References

42 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Banerjee, S. and Lavie, A. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 65–72, 2005. find our card
  2. Bar-Haim, R., Dagan, I., Dolan, B., Ferro, L., and Giampiccolo, D. The second PASCAL recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, 2006. find
  3. Bentivogli, L., Dagan, I., Dang, H. T., Giampiccolo, D., and Magnini, B. The fifth pascal recognizing textual entailment challenge. In In Proc Text Analysis Conference, 2009. find
  4. Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., and Specia, L. Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055, 2017. find
  5. Dagan, I., Glickman, O., and Magnini, B. The PASCAL recognising textual entailment challenge. In Proceedings of the First International Conference on Machine Learning Challenges, pp. 177–190, 2006. find
  6. Devlin, J., Chang, M., Lee, K., and Toutanova, K. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018. find
  7. Dolan, W. B. and Brockett, C. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing, 2005. find our card
  8. Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., and Hon, H.-W. Unified language model pre-training for natural language understanding and generation. In 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), 2019. find our card
  9. Du, X. and Cardie, C. Harvesting paragraph-level question-answer pairs from wikipedia. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pp. 1907–1917, 2018. find
  10. Giampiccolo, D., Magnini, B., Dagan, I., and Dolan, B. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pp. 1–9, 2007. find
  11. Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., and Levy, O. SpanBERT: Improving pre-training by representing and predicting spans. arXiv preprint arXiv:1907.10529, 2019. find our card
  12. Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, San Diego, CA, 2015. find
  13. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. ALBERT: A lite bert for self-supervised learning of language representations. ArXiv, abs/1909.11942, 2019. find
  14. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019. find our card
  15. Lin, C.-Y. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pp. 74–81, Barcelona, Spain, 2004. find
  16. Liu, Y. and Lapata, M. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pp. 3730–3740, Hong Kong, China, 2019. find
  17. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692, 2019. find
  18. Narayan, S., Cohen, S. B., and Lapata, M. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797–1807, Brussels, Belgium, 2018. find
  19. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311–318, Philadelphia, Pennsylvania, USA, 2002. find our card
  20. Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2227–2237, New Orl… find
  21. Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. Improving language understanding by generative pre-training. 2018. find
  22. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019. find
  23. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383– 2392, Austin, Texas, 2016. find our card
  24. Rajpurkar, P., Jia, R., and Liang, P. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers, pp. 784–789, 2018. find our card
  25. See, A., Liu, P. J., and Manning, C. D. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pp. 1073–1083, Vancouver, Canada, 2017. find
  26. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631–1642, 2013. find
  27. Song, K., Tan, X., Qin, T., Lu, J., and Liu, T.-Y. Mass: Masked sequence to sequence pre-training for language generation. arXiv preprint arXiv:1905.02450, 2019. find our card
  28. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016. find
  29. Taylor, W. L. Cloze procedure: A new tool for measuring readability. Journalism Bulletin, 30(4):415–433, 1953. find
  30. Trinh, T. H. and Le, Q. V. A simple method for commonsense reasoning. ArXiv, abs/1806.02847, 2018. find
  31. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. In Advances in Neural Information Processing Systems 30, pp. 5998–6008, 2017. find
  32. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019. find our card
  33. Warstadt, A., Singh, A., and Bowman, S. R. Neural network acceptability judgments. arXiv preprint arXiv:1805.12471, 2018. find
  34. Williams, A., Nangia, N., and Bowman, S. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1112–1122, New Orleans, Loui… find
  35. Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., Klingner, J., Shah, A., Johnson, M., Liu, X., Kaiser, L., Gouws, S., Kato, Y., Kudo, T., Kazawa, H., Stevens, K., Kurian, G., Patil, N., Wang, W., Young, C., Smith, J., Riesa, J., Rudnic… find
  36. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., and Le, Q. V. XLNet: Generalized autoregressive pre-training for language understanding. In 33rd Conference on Neural Information Processing Systems, 2019. find
  37. Zhang, S. and Bansal, M. Addressing semantic drift in question generation for semi-supervised question answering. CoRR, abs/1909.06356, 2019. find
  38. Zhao, Y., Ni, X., Ding, Y., and Ke, Q. Paragraph-level neural question generation with maxout pointer and gated self-attention networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3901–3910, Brussels, Belgium, 2018. find
  39. Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., and Fidler, S. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pp. 19–27, 2015. find our card
  40. A. Hyperparameters for Pre-Training As shown in Table 7, we present the hyperparameters used for pre-training U NI LMv2 BASE . We use the same Word-Piece (Wu et al., 2016) vocabulary and model size as BERT BASE (Devlin et al., 2018). We follow the optimization hyperparameters of RoBERTa BASE (Liu et… find
  41. Layers Hidden size FFN inner hidden size Attention heads Attention head size Max relative position Training steps Batch size Adam  Adam β Learning rate Learning rate schedule Warmup ratio Gradient clipping Dropout Weight decay 12 768 3072 12 64 128 0.5M 7680 1e-6 (0.9, 0.98) 6e-4 Linear 0.048 0.0 0… find
  42. Dataset \#Train/\#Dev/\#Test Single-Sentence Classification CoLA (Acceptability) 8.5k/1k/1k SST-2 (Sentiment) 67k/872/1.8k Pairwise Text Classification MNLI (NLI) 393k/20k/20k RTE (NLI) 2.5k/276/3k QNLI (NLI) 105k/5.5k/5.5k WNLI (NLI) 634/71/146 QQP (Paraphrase) 364k/40k/391k MRPC (Paraphrase) 3.7k/408… find