Elamur Research⌕
☰
home / papers / 2019 / DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

Paper · 2019 · Victor Sanh, Lysandre Debut, Julien Chaumond et al.

Summary

Large pre-trained language models such as BERT deliver high quality. However, they require a lot of memory and computation, which makes them hard to run on smartphones or on a limited budget. The authors from Hugging Face propose DistilBERT: a compact general-purpose model obtained through knowledge distillation already during pre-training rather than for a specific task. The student's architecture is the same as BERT-base, but with half as many layers, and the token-type embeddings and the pooler are removed. The initial weights are taken from every second layer of the teacher. Training combines three loss functions: a distillation loss on the teacher's softened distributions, masked language modeling, and a cosine loss that aligns the directions of the hidden states. The corpus is the same as BERT's: English Wikipedia and the Toronto Book Corpus. As a result, DistilBERT is 40% smaller and 60% faster than BERT and retains about 97% of its performance on GLUE. The weights and code are released openly in the Transformers library. The model has therefore become a practical foundation for cheaper inference and for NLP on mobile devices.

Key points

  • Size and speed: 66M parameters versus 110M in BERT-base and 180M in ELMo. A full pass over the STS-B dev set on CPU with batch size 1 takes 410 s versus 668 s for BERT.
  • Quality: GLUE macro-score of 77.0 versus 79.5 for BERT-base and 68.7 for the ELMo baseline. On IMDb, accuracy is 92.82 versus 93.46. On SQuAD 1.1, EM/F1 is 77.7/85.8 versus 81.2/88.5, and with a second distillation step during fine-tuning it is 79.1/86.9 (according to Table 2; the main text gives different numbers for this variant).
  • Training: about 90 hours on 8 V100 GPUs with 16 GB each. Batches of up to 4K examples via gradient accumulation, dynamic masking, no next sentence prediction objective (following the RoBERTa recipe).
  • Ablation (change in GLUE macro-score): without the distillation loss L_ce −2.96, without the cosine loss L_cos −1.46, without L_mlm only −0.31. Random initialization instead of the teacher's weights gives −3.69.
  • On-device: in a question answering app on the iPhone 7 Plus, DistilBERT runs 71% faster than BERT (excluding tokenization), and the model weighs 207 MB. The code is published in the swift-coreml-transformers repository.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Robert BoschStuttgart · Germany12 works · 9,057 citations🎓Freie Universität BerlinBerlin · Germany13 works · 7,043 citations🔬Fraunhofer Institute for Open Communication SystemsBerlin · Germany2 works · 6,805 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Efficient models & compression · 297 works

Cite

Cite as Victor Sanh, Lysandre Debut, Julien Chaumond and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv:1910.01108.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1910.01108 · arXiv 1910.01108

@misc{sanh2019distilbert,
  title = {{DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter}},
  author = {Victor Sanh, Lysandre Debut, Julien Chaumond and Thomas Wolf},
  year = {2019},
  doi = {10.48550/arxiv.1910.01108},
  eprint = {1910.01108},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1910.01108}
}

References

22 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2018.
  2. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. find
  3. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar S. Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke S. Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692, 2019.
  4. Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai. ArXiv, abs/1907.10597, 2019.
  5. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. In ACL, 2019.
  6. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
  7. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Transformers: State-of-the-art natural language processing, 2019.
  8. Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In KDD, 2006. find
  9. Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. ArXiv, abs/1503.02531, 2015.
  10. Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. 2015 IEEE International Conference on Computer Vision (ICCV), pages 19–27, 2015.
  11. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2018.
  12. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACL, 2018.
  13. Alex Wang, Ian F. Tenney, Yada Pruksachatkun, Katherin Yu, Jan Hula, Patrick Xia, Raghu Pappagari, Shuning Jin, R. Thomas McCoy, Roma Patel, Yinghui Huang, Jason Phang, Edouard Grave, Najoung Kim, Phu Mon Htut, Thibault F’evry, Berlin Chen, Nikita Nangia, Haokun Liu, Anhad Mohananey, Shikha Bordia,… find
  14. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In ACL, 2011.
  15. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. In EMNLP, 2016. — SQuAD: 100,000+ Questions for Machine Comprehension of Text
  16. Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. Distilling task-specific knowledge from bert into simple neural networks. ArXiv, abs/1903.12136, 2019. find
  17. Debajyoti Chatterjee. Making neural machine reading comprehension faster. ArXiv, abs/1904.00796, 2019. — [arXiv:1904.00796](https://arxiv.org/abs/1904.00796) find
  18. Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: The impact of student initialization on knowledge distillation. ArXiv, abs/1908.08962, 2019.
  19. Ze Yang, Linjun Shou, Ming Gong, Wutao Lin, and Daxin Jiang. Model compression with multi-task knowledge distillation for web-scale question answering system. ArXiv, abs/1904.09636, 2019. — [arXiv:1904.09636](https://arxiv.org/abs/1904.09636) find
  20. Henry Tsai, Jason Riesa, Melvin Johnson, Naveen Arivazhagan, Xin Li, and Amelia Archer. Small and practical bert models for sequence labeling. In EMNLP-IJCNLP, 2019. find
  21. Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In NeurIPS, 2019.
  22. Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In ICML, 2015. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sanh-2019-distilbert-distilled-version-bert-smaller/