Elamur Research⌕
☰
home / papers / 2020 / TinyBERT: Distilling BERT for Natural Language Understanding

TinyBERT: Distilling BERT for Natural Language Understanding

Paper · 2020 · Xiaoqi Jiao, Yichun Yin, Lifeng Shang et al.

Summary

Large pre-trained language models such as BERT achieve very high performance. However, they have many parameters and slow inference, which makes them difficult to run on phones and other low-resource devices. The authors compress BERT with a knowledge distillation method designed specifically for the Transformer architecture. The small "student" model TinyBERT learns to reproduce not only the final predictions of the "teacher". It also mimics the teacher's embedding layer and the hidden states and attention matrices of its intermediate layers. Training has two stages. First comes general distillation on a large unlabeled corpus, where the teacher is the original BERT without fine-tuning. Then comes task-specific distillation: the teacher is a fine-tuned BERT, and the training set is expanded through data augmentation. The four-layer TinyBERT4 retains more than 96.8% of the performance of BERT BASE on GLUE. At the same time, it is 7.5 times smaller and 9.4 times faster. The six-layer TinyBERT6 performs practically on par with its teacher. The work offers a practical way to obtain compact NLP models for systems with limited computational resources.

Key points

  • Loss functions: distillation operates at three levels. The student is fitted to the output of the embedding layer, to the hidden states and attention matrices of the Transformer layers (via MSE), and to the logits of the prediction layer. Attention is included because it carries syntactic and coreference information.
  • Two-stage framework: general distillation yields a general-purpose General TinyBERT. It serves as the starting point for task-specific distillation on the augmented dataset. According to the authors, both stages are necessary for performance and generalization ability.
  • Comparison with other models: TinyBERT4 has only about 13.3% of the parameters of BERT BASE and about 10.6% of its inference time. It clearly outperforms other four-layer distilled models (BERT-PKD, DistilBERT), despite having roughly 28% of their parameters and 31% of their inference time.
  • Choice of teacher layers: the uniform strategy, in which the student draws on layers from the full depth of BERT, achieved an average score of 75.6 on the dev sets of MNLI, MRPC and CoLA. For strategies that use only the top or only the bottom layers, the scores are 70.9 and 71.3, respectively. Hence, different tasks require knowledge from different layers.
  • Resources: the code and models are open-sourced in the huawei-noah/Pretrained-Language-Model repository on GitHub (TinyBERT folder). Among directions for future work, the authors name distillation from BERT LARGE and combining distillation with quantization or pruning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Huawei TechnologiesShenzhen · China96 works · 15,432 citations🎓Huazhong University of Science and TechnologyWuhan · China59 works · 8,310 citations

Also on the byline: Wuhan National Laboratory for Optoelectronics.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Natural language understanding · 52 works

Cite

Cite as Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang and Qun Liu. 2020. TinyBERT: Distilling BERT for Natural Language Understanding. Findings of the Association for Computational Linguistics: EMNLP 2020.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.findings-emnlp.372 · arXiv 1909.10351

@inproceedings{jiao2020tinybert,
  title = {{TinyBERT: Distilling BERT for Natural Language Understanding}},
  author = {Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang and Qun Liu},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2020},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.372},
  eprint = {1909.10351},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.findings-emnlp.372}
}

References

42 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. L. Bentivogli, P. Clark, I. Dagan, and D. Giampiccolo. 2009. The fifth pascal recognizing textual entailment challenge. find
  2. D. Cer, M. Diab, E. Agirre, I. Lopez-Gazpio, and L. Specia. 2017. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation.
  3. Z. Chen, H. Zhang, X. Zhang, and L. Zhao. 2018. Quora question pairs. find
  4. K. Clark, U. Khandelwal, O. Levy, and C. D. Manning. 2019. What does bert look at? an analysis of bert’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. find
  5. K. Clark, M. Luong, Q. V. Le, and C. D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. In ICLR. find
  6. B. Cui, Y. Li, M. Chen, and Z. Zhang. 2019. Fine-tune bert with sparse self-attention mechanism. In EMNLP. find
  7. M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and L. Kaiser. 2019. Universal transformers. In ICLR. find
  8. G. Hinton, O. Vinyals, and J. Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
  9. L. Hou, L. Shang, X. Jiang, and Q. Liu. 2020. Dyn-abert: Dynamic bert with adaptive width and depth. arXiv preprint arXiv:2004.04037. find
  10. M. Hu, Y. Peng, F. Wei, Z. Huang, D. Li, N. Yang, and M. Zhou. 2018. Attention-guided answer distillation for machine reading comprehension. In EMNLP. find
  11. Y. Kim and A. M. Rush. 2016. Sequence-level knowledge distillation. In EMNLP.
  12. O. Kovaleva, A. Romanov, A. Rogers, and A. Rumshisky. 2019. Revealing the dark secrets of bert. In EMNLP.
  13. Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In ICLR.
  14. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. find
  15. W. Liu, P. Zhou, Z. Zhao, Z. Wang, H. Deng, and Q. Ju. 2020. Fastbert: a self-distilling bert with adaptive inference time. arXiv preprint arXiv:2004.02178.
  16. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  17. X. Ma, P. Zhang, S. Zhang, N. Duan, Y. Hou, M. Zhou, and D. Song. 2019. A tensorized transformer for language modeling. In NIPS. find
  18. J. S. McCarley. 2019. Pruning a bert-based question answering model. arXiv preprint arXiv:1910.06360. — [arXiv:1910.06360](https://arxiv.org/abs/1910.06360) find
  19. P. Michel, O. Levy, and G. Neubig. 2019. Are sixteen heads really better than one? In NIPS.
  20. J. Pennington, R. Socher, and C. D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP.
  21. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
  22. P. Rajpurkar, R. Jia, and P. Liang. 2018. Know what you don’t know: Unanswerable questions for squad. In ACL.
  23. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In EMNLP.
  24. A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio. 2014. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550. find
  25. H. Sajjad, F. Dalvi, N. Durrani, and P. Nakov. 2020. Poor man’s bert: Smaller and faster transformer models. arXiv preprint arXiv:2004.03844. — [arXiv:2004.03844](https://arxiv.org/abs/2004.03844) find
  26. V. Sanh, L. Debut, J. Chaumond, and T. Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
  27. S. Shen, Z. Dong, J. Ye, L. Ma, Z. Yao, A. Gholami, M. W. Mahoney, and K. Keutzer. 2019. Q-bert: Hessian based ultra low precision quantization of bert. arXiv preprint arXiv:1909.05840. find
  28. R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP.
  29. S. Sun, Y. Cheng, Z. Gan, and J. Liu. 2019. Patient knowledge distillation for bert model compression. In EMNLP.
  30. Z. Sun, H. Yu, X. Song, R. Liu, Y. Yang, and D. Zhou. 2020. Mobilebert: a compact task-agnostic bert for resource-limited devices. arXiv preprint arXiv:2004.02984.
  31. R. Tang, Y. Lu, L. Liu, L. Mou, O. Vechtomova, and J. Lin. 2019. Distilling task-specific knowledge from bert into simple neural networks. arXiv preprint arXiv:1903.12136. find
  32. I. Turc, M. Chang, K. Lee, and K. Toutanova. 2019. Well-read students learn better: The impact of student initialization on knowledge distillation. arXiv preprint arXiv:1908.08962.
  33. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. 2017. Attention is all you need. In NIPS.
  34. E. Voita, D. Talbot, F. Moiseev, R. Sennrich, and I. Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In ACL.
  35. A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP.
  36. W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. arXiv preprint arXiv:2002.10957. find
  37. A. Warstadt, A. Singh, and S. R. Bowman. 2019. Neural network acceptability judgments. TACL. find
  38. A. Williams, N. Nangia, and S. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL.
  39. X. Wu, S. Lv, L. Zang, J. Han, and S. Hu. 2019. Conditional bert contextual augmentation. In International Conference on Computational Science. find
  40. C. Xu, W. Zhou, T. Ge, F. Wei, and M. Zhou. 2020. Bert-of-theseus: Compressing bert by progressive module replacing. arXiv preprint arXiv:2002.02925.
  41. Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In NIPS.
  42. O. Zafrir, G. Boudoukh, P. Izsak, and M. Wasserblat. 2019. Q8bert: Quantized 8bit bert. arXiv preprint arXiv:1910.06188. — [arXiv:1910.06188](https://arxiv.org/abs/1910.06188) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jiao-2020-tinybert-distilling-bert-natural-language/