Elamur Research⌕
☰
home / papers / 2019 / Cloze-driven Pretraining of Self-attention Networks

Cloze-driven Pretraining of Self-attention Networks

Paper · 2019 · Alexei Baevski, Sergey Edunov, Yinhan Liu et al.

Summary

Pretrained language models had boosted language understanding, but earlier approaches either read text in one direction (GPT) or trained two directional models with separate losses (ELMo). Baevski and colleagues at Facebook AI Research propose a two-tower self-attention network. One stack of transformer decoder blocks reads text left-to-right and another reads it right-to-left. A final self-attention combination layer merges both sides to guess the token at each position. Neither tower sees the target token, so every word in the corpus provides a training signal. BERT, by contrast, reconstructs only a masked subset of tokens and adds next-sentence prediction. The model is pretrained on up to 18B tokens drawn from Common Crawl, News Crawl, BooksCorpus and Wikipedia. It is then fine-tuned with masking in the combination layer switched off. The model beats GPT on every GLUE task while trailing BERT slightly. With task-specific architectures stacked on top, it reaches new state-of-the-art results for NER and constituency parsing. Extensive ablations show how data size, data domain, multi-sentence examples and the training objective affect the gains.

Key points

  • Architecture: forward and backward towers of transformer blocks (16 attention heads, pre-layer normalization, character CNN input for word models), joined by an attention-based combination module that predicts the center word. Fine-tuning uses the outputs at the <s> boundary tokens (dimension 1024) and adds a <sep> token for sentence pairs.
  • On GLUE, the model gains 9.1 points on RTE over Radford et al. (2018). Its results come close to the concurrent BERT model but fall slightly short of it.
  • The cloze objective outperforms separate left/right next-word prediction (bilm). Cloze perplexity is about 4, versus roughly 27–30 for bilm. Mixing the two losses (bilm term scaled by 0.15) brings no benefit over cloze alone.
  • More Common Crawl data keeps helping. The average GLUE dev score rises from 79.9 at 562M tokens to 81.3 at 18B tokens, with no sign of saturation. CoLA and RTE benefit the most.
  • Paragraph-level training examples beat single sentences. At 4.5B tokens, News Crawl (about 23-word sentences) trails Common Crawl by 14 accuracy points on RTE. Training on fixed 512-token blocks of BooksCorpus plus Wikipedia did no better than using the examples as they are.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MetaMenlo Park · United States214 works · 241,492 citations🏢Meta (Israel)Tel Aviv · Israel46 works · 42,810 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer and Michael Auli. 2019. Cloze-driven Pretraining of Self-attention Networks. arXiv:1903.07785.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1903.07785 · arXiv 1903.07785

@misc{baevski2019cloze,
  title = {{Cloze-driven Pretraining of Self-attention Networks}},
  author = {Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer and Michael Auli},
  year = {2019},
  doi = {10.48550/arxiv.1903.07785},
  eprint = {1903.07785},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1903.07785}
}

References

36 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alexei Baevski and Michael Auli. 2018. Adaptive input representations for neural language modeling. arXiv, abs/1809.10853.
  2. Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. The fifth pascal recognizing textual entailment challenge. In Proc. of TAC. find
  3. Ondřej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 conference on machine translation (WMT18). In Proc. of WMT.
  4. Daniel M. Cer, Mona T. Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2018. Semeval-2017 task 1: Semantic textual similarity - multilingual and cross-lingual focused evaluation. In Proc. of SemEval.
  5. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognizing textual entailment challenge. Machine learning challenges, evaluating predictive uncertainty, visual object classification, and recognizing textual entailment, pages 177– 190. find
  6. Andrew M. Dai and Quoc V. Le. 2015. Semi-supervised sequence learning. arXiv, abs/1511.01432. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
  8. William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proc. of IWP.
  9. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The pascal recognizing textual entailment challenge. Proc. of the ACL-PASCAL workshop on textual entailment and paraphrasing. find
  10. Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proc. of LREC.
  11. Edouard Grave, Armand Joulin, Moustapha Cissé, David Grangier, and Hervé Jégou. 2017. Efficient softmax approximation for gpus. In Proc. of ICML.
  12. Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The pascal recognising textual entailment challenge. find
  13. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep Residual Learning for Image Recognition. In Proc. of CVPR.
  14. Hakan Inan, Khashayar Khosravi, and Richard Socher. 2016. Tying word vectors and word classifiers: A loss framework for language modeling. arXiv, abs/1611.01462.
  15. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2016. Character-aware neural language models. In AAAI, pages 2741–2749.
  16. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In Proc. of ICLR.
  17. Nikita Kitaev and Dan Klein. 2018. Constituency parsing with a self-attentive encoder. In Proc. of ACL.
  18. Ilya Loshchilov and Frank Hutter. 2016. SGDR: stochastic gradient descent with restarts. arXiv, abs/1608.03983. find
  19. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In Proc. of NIPS.
  20. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proc. of NAACL System Demonstrations.
  21. Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. In Proc. of WMT.
  22. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In Proc. of ICML. find
  23. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proc. of ACL.
  24. Jason Phang, Thibault Fevry, and Samuel R. Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv, abs/1811.01088.
  25. Ofir Press and Lior Wolf. 2017. Using the output embedding to improve language models. In Proc. of EACL.
  26. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/ language-unsupervised/language_ understanding_paper.pdf. ↗
  27. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. arXiv, abs/1606.05250.
  28. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proc. of ACL.
  29. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proc. of EMNLP.
  30. Ilya Sutskever, James Martens, George E. Dahl, and Geoffrey E. Hinton. 2013. On the importance of initialization and momentum in deep learning. In Proc. of ICML. find
  31. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Proc. of NIPS.
  32. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. arXiv, abs/1804.07461.
  33. Alex Warstadt, Amanpreet Singh, and Sam Bowman. 2018. Corpus of linguistic acceptability. https://nyu-mll.github.io/CoLA. ↗
  34. Adina Williams, Nikita Nangia, and Samuel R Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proc. of NAACL.
  35. Kelly W. Zhang and Samuel R. Bowman. 2018. Language modeling teaches you more syntax than translation does: Lessons learned through auxiliary task analysis. arXiv, abs/1809.10040. find
  36. Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. arXiv, abs/1506.06724.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/baevski-2019-cloze-driven-pretraining-self-attention/