Elamur Research
☰
home / papers / 2019 / Well-Read Students Learn Better: On the Importance of Pre-training Compact Models

Well-Read Students Learn Better: On the Importance of Pre-training Compact Models

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1908.08962

Summary

Large pre-trained Transformers such as BERT perform well but are too slow and memory-hungry for many deployments. Earlier compression work used elaborate schemes, while the obvious baseline was largely ignored: running ordinary masked-LM pre-training and fine-tuning directly on a small network. The authors test this baseline and find it competitive. They then extend it into Pre-trained Distillation (PD), a three-stage recipe:

  1. MLM pre-training of a compact student on unlabeled general text.
  2. Standard knowledge distillation from a large fine-tuned teacher (BERT BASE or BERT LARGE), using the teacher's soft labels on unlabeled task data.
  3. An optional fine-tuning pass on labeled data.

On GLUE tasks, PD matches or beats more complex concurrent methods such as Patient-KD and DistilBERT, and it works with any architecture. Systematic experiments vary student size and the amount and domain similarity of the transfer data. Pre-training and distillation help each other even when both use the same data. The released compact models give practitioners strong starting points across a range of compute budgets.

Key points

  • Released models: 24 miniature BERT models, from about 4M parameters (L=2, H=128, "Tiny") to 110M (L=12, H=768, "Base"). The smallest runs roughly 65× faster than BERT LARGE on TPU v2.
  • Comparison with Patient-KD: With 6-layer/768-hidden students, PD reaches a GLUE test meta score of 82.1, versus 81.7 for Patient-KD (Sun et al., 2019a) and 81.6 for plain pre-training+fine-tuning.
  • Comparison with DistilBERT: On dev, PD averages 84.4, versus 82.3 for DistilBERT (Sanh, 2019).
  • Depth vs. width: Pre-trained students gain more from added depth than from added width. Randomly initialized students show no such pattern.
  • Robustness and compounding: PD tolerates smaller or less in-domain transfer sets better than distillation from scratch. Applying pre-training and then distillation to the same dataset outperforms either step alone.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

273 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 265 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1908.08962 · arXiv 1908.08962

@misc{turc2019well,
  title = {{Well-Read Students Learn Better: On the Importance of Pre-training Compact Models}},
  author = {Turc, Iulia and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  year = {2019},
  doi = {10.48550/arxiv.1908.08962},
  eprint = {1908.08962},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1908.08962}
}

References

41 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. The fifth pascal recognizing textual entailment challenge. In Proc Text Analysis Conference (TAC09, 2009. find
  2. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2015. find
  3. C Bucilă, R Caruana, and A Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 535– 541, 2006. find
  4. Z. Chen, H. Zhang, X. Zhang, and L. Zhao. Quora question pairs. 2018. find
  5. Yew Ken Chia, Sam Witteveen, and Martin Andrews. Transformer to cnn: Label-scarce distillation for efficient text classification. 2018. find
  6. Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D. Manning, and Quoc V. Le. BAM! born-again multi-task networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. Association for C… find
  7. Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. In Advances in neural information processing systems, pp. 3079–3087, 2015. find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. find
  9. Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In ICML, 2015. find
  10. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. Annotation artifacts in natural language inference data. In NAACL, 2018. find
  11. Suchin Gururangan, Tam Dang, Dallas Card, and Noah A. Smith. Variational pretraining for semi-supervised text classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. find
  12. Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In NIPS, 2016. find
  13. Ruining He and Julian McAuley. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In proceedings of the 25th international conference on world wide web, pp. 507–517. International World Wide Web Conferences Steering Committee, 2016. find
  14. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. find our card
  15. Minghao Hu, Yuxing Peng, Furu Wei, Zhen Huang, Dongsheng Li, Nan Yang, and Ming Zhou. Attention-guided answer distillation for machine reading comprehension. arXiv preprint arXiv:1808.07644, 2018. find
  16. Johansson, Stig, and Knut Hofland. Frequency Analysis of English vocabulary and grammar, based on the LOB corpus. Claredon, Oxford, 1989. find
  17. Rie Johnson and Tong Zhang. Semi-supervised convolutional neural networks for text categorization via region embedding. In Advances in neural information processing systems, pp. 919–927, 2015. find
  18. Adam Kilgarriff and Tony Rose. Measures for corpus similarity and homogeneity. In Proceedings of the Third Conference on Empirical Methods for Natural Language Processing, pp. 46–52, 1998. find
  19. Young-Bum Kim, Karl Stratos, and Dongchan Kim. Adversarial adaptation of synthetic or stale data. In ACL, 2017. find
  20. Akisato Kimura, Zoubin Ghahramani, Koh Takeuchi, Tomoharu Iwata, and Naonori Ueda. Few-shot learning of neural networks from scratch by pseudo example optimization. arXiv preprint arXiv:1802.03039, 2018. — [arXiv:1802.03039](https://arxiv.org/abs/1802.03039) find
  21. Jinyu Li, Rui Zhao, Jui-Ting Huang, and Yifan Gong. Learning small-size dnn with output-distribution-based criteria. In Fifteenth annual conference of the international speech communication association, 2014. find
  22. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019. find
  23. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pp. 3111–3119, 2013. find
  24. Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532–1543, 2014. find
  25. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.05365, 2018. find
  26. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assets/research-covers/languageunsupervised/language understanding paper. pdf, 2018. find
  27. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. URL https://openai. com/blog/better-language-models, 2019. find
  28. Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014. find
  29. Victor Sanh. Smaller, faster, cheaper, lighter: Introducing DistilBERT, a distilled version of BERT. https://medium.com/huggingface/distilbert-8cf3380435b5, 2019. ↗
  30. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631–1642, 201… find
  31. Spearman. The proof and measurement of association between two things. In American Journal of Psychology., pp. 72–101, 1904. find
  32. Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. arXiv preprint arXiv:1908.09355, 2019a. find our card
  33. Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. Ernie 2.0: A continual pre-training framework for language understanding, 2019b. find our card
  34. Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. Distilling task-specific knowledge from bert into simple neural networks, 2019. find
  35. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998–6008, 2017. find
  36. Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018. find our card
  37. Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers),… find
  38. Ze Yang, Linjun Shou, Ming Gong, Wutao Lin, and Daxin Jiang. Model compression with multi-task knowledge distillation for web-scale question answering system. arXiv preprint arXiv:1904.09636, 2019a. — [arXiv:1904.09636](https://arxiv.org/abs/1904.09636) find
  39. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237, 2019b. find our card
  40. Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017. find
  41. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pp. 19–27, 2015… find our card