Elamur Research
☰
home / papers / 2019 / XLNet: Generalized Autoregressive Pretraining for Language Understanding

XLNet: Generalized Autoregressive Pretraining for Language Understanding

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/1906.08237

Summary

Before XLNet, language pretraining used one of two approaches. Autoregressive language models estimate text likelihood in one direction, so each prediction uses context from only one side. BERT-style denoising autoencoders use context from both sides. However, they train on [MASK] tokens that never appear during finetuning, and they predict the masked tokens as if those tokens were independent of each other. XLNet, from Carnegie Mellon University and the Google AI Brain Team, keeps the autoregressive setup but samples a random factorization order for each sequence. The parameters are shared across all orders, so on average every token learns to use context from both directions, and the input is never corrupted. To make this work, the authors added a two-stream self-attention design that tells the model which position it is predicting. They also used Transformer-XL's segment recurrence and relative positional encoding. In comparable settings, XLNet beat BERT on 20 tasks: GLUE, SQuAD, RACE, sentiment and topic classification, and ClueWeb09-B document ranking. The largest gains were on reading tasks with long contexts. XLNet became an important reference point among the pretraining objectives that followed BERT.

Key points

  • Permutation language modeling: only the order of prediction is permuted, using attention masks. Tokens keep their original positional encodings, so the model still sees text in natural order during finetuning. The content stream can see the current token. The query stream sees only its position, which keeps the prediction target unambiguous.
  • GLUE dev, single models of BERT-Large size (24 layers): XLNet scores MNLI 90.8/90.8, SST-2 97.0 and CoLA 69.0. RoBERTa scores 90.2/90.2, 96.4 and 68.0, and BERT scores 86.6, 93.2 and 60.6. On the public leaderboard (as of Oct 28, 2019), the XLNet ensemble led on most tasks, e.g. WNLI 92.5.
  • Text classification error rates: Yelp-5 27.05% vs BERT 29.32%, Amazon-5 31.67% vs 34.17%, IMDB 3.20% vs 4.51%, DBpedia 0.60% vs 0.64%.
  • Ablation with 12-layer Base models trained only on Wikipedia and BooksCorpus: XLNet-Base reaches RACE 66.05–66.66 vs BERT-Base 64.3, and SQuAD2.0 F1 81.33 vs 76.30. Removing the Transformer-XL memory hurts most on RACE. Next-sentence prediction gave no benefit, so it was dropped.
  • The code and pretrained models are released at github.com/zihangdai/xlnet. The paper was published at NeurIPS 2019.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

1992 papers of our library cite it (of the 140673 whose bibliographies we read); 102 of them have a page here, the first 100 below, 1890 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1906.08237 · arXiv 1906.08237

@misc{yang2019xlnet,
  title = {{XLNet: Generalized Autoregressive Pretraining for Language Understanding}},
  author = {Yang, Zhilin and Dai, Zihang and Yang, Yiming and Carbonell, Jaime and Salakhutdinov, Ruslan and Le, Quoc V.},
  year = {2019},
  doi = {10.48550/arxiv.1906.08237},
  eprint = {1906.08237},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1906.08237}
}

References

40 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. Character-level language modeling with deeper self-attention. arXiv preprint arXiv:1808.04444, 2018. find
  2. Anonymous. Bam! born-again multi-task networks for natural language understanding. anonymous preprint under review, 2018. find
  3. Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. arXiv preprint arXiv:1809.10853, 2018. find
  4. Yoshua Bengio and Samy Bengio. Modeling high-dimensional discrete data with multi-layer neural networks. In Advances in Neural Information Processing Systems, pages 400–406, 2000. find
  5. Jamie Callan, Mark Hoy, Changkuk Yoo, and Le Zhao. Clueweb09 data set, 2009. find
  6. Common Crawl. Common crawl. URl: http://http://commoncrawl. org, 2019. find
  7. Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. In Advances in neural information processing systems, pages 3079–3087, 2015. find
  8. Zhuyun Dai, Chenyan Xiong, Jamie Callan, and Zhiyuan Liu. Convolutional neural networks for soft-matching n-grams in ad-hoc search. In Proceedings of the eleventh ACM international conference on web search and data mining, pages 126–134. ACM, 2018. find
  9. Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860, 2019. find
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. find
  11. William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: better text generation via filling in the\_. arXiv preprint arXiv:1801.07736, 2018. — [arXiv:1801.07736](https://arxiv.org/abs/1801.07736) find
  12. Mathieu Germain, Karol Gregor, Iain Murray, and Hugo Larochelle. Made: Masked autoencoder for distribution estimation. In International Conference on Machine Learning, pages 881–889, 2015. find
  13. Jiafeng Guo, Yixing Fan, Qingyao Ai, and W Bruce Croft. A deep relevance matching model for ad-hoc retrieval. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pages 55–64. ACM, 2016. find
  14. Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146, 2018. find
  15. Rie Johnson and Tong Zhang. Deep pyramid convolutional neural networks for text categorization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 562–570, 2017. find
  16. Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu, Yordan Yordanov, and Thomas Lukasiewicz. A surprisingly robust trick for winograd schema challenge. arXiv preprint arXiv:1905.06290, 2019. — [arXiv:1905.06290](https://arxiv.org/abs/1905.06290) find
  17. Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226, 2018. find our card
  18. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683, 2017. find our card
  19. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942, 2019. find
  20. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. arXiv preprint arXiv:1901.11504, 2019. find our card
  21. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. find
  22. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems, pages 6294–6305, 2017. find our card
  23. Takeru Miyato, Andrew M Dai, and Ian Goodfellow. Adversarial training methods for semi-supervised text classification. arXiv preprint arXiv:1605.07725, 2016. find our card
  24. Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. arXiv preprint arXiv:1601.06759, 2016. — [arXiv:1601.06759](https://arxiv.org/abs/1601.06759) find
  25. Xiaoman Pan, Kai Sun, Dian Yu, Heng Ji, and Dong Yu. Improving question answering with external knowledge. arXiv preprint arXiv:1902.00993, 2019. find
  26. Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. English gigaword fifth edition, linguistic data consortium. Technical report, Technical Report. Linguistic Data Consortium, Philadelphia, Tech. Rep., 2011. find
  27. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.05365, 2018. find
  28. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assets/research-covers/languageunsupervised/language understanding paper. pdf, 2018. find
  29. Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822, 2018. find our card
  30. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016. find our card
  31. Devendra Singh Sachan, Manzil Zaheer, and Ruslan Salakhutdinov. Revisiting lstm networks for semi-supervised text classification via mixed objective function. 2018. find
  32. Benigno Uria, Marc-Alexandre Côté, Karol Gregor, Iain Murray, and Hugo Larochelle. Neural autoregressive distribution estimation. The Journal of Machine Learning Research, 17(1):7184– 7220, 2016. find
  33. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017. find
  34. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. 2019. In the Proceedings of ICLR. find our card
  35. Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V. Le. Unsupervised data augmentation. arXiv preprint arXiv:1904.12848, 2019. find our card
  36. Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. End-to-end neural ad-hoc ranking with kernel pooling. In Proceedings of the 40th International ACM SIGIR conference on research and development in information retrieval, pages 55–64. ACM, 2017. find
  37. Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W Cohen. Breaking the softmax bottleneck: A high-rank rnn language model. arXiv preprint arXiv:1711.03953, 2017. — [arXiv:1711.03953](https://arxiv.org/abs/1711.03953) find
  38. Shuailiang Zhang, Hai Zhao, Yuwei Wu, Zhuosheng Zhang, Xi Zhou, and Xiang Zhou. Dual co-matching network for multi-choice reading comprehension. arXiv preprint arXiv:1901.09381, 2019. — [arXiv:1901.09381](https://arxiv.org/abs/1901.09381) find
  39. Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Advances in neural information processing systems, pages 649–657, 2015. find our card
  40. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19–27, 20… find our card