Elamur Research
☰
home / papers / 2019 / Patient Knowledge Distillation for BERT Model Compression

Patient Knowledge Distillation for BERT Model Compression

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1908.09355

Summary

Pre-trained encoders like BERT are expensive to fine-tune and deploy. BERT-Base alone stacks 12 Transformer layers with about 110M parameters. This paper asks how to shrink such a model into a shallower network while preserving task accuracy. The authors start from standard knowledge distillation, where a small student fits both gold labels and a fine-tuned teacher's temperature-softened predictions. They note that this setup plateaus early on test data, possibly from overfitting. Their remedy, Patient Knowledge Distillation (Patient-KD), also makes the student imitate the teacher's intermediate-layer representations. There are two layer-matching schemes. PKD-Skip aligns the student with every k-th teacher layer, and PKD-Last uses the teacher's final k layers. Students are initialized from the lower layers of BERT and tested on seven datasets spanning sentiment classification, paraphrase matching, NLI and machine reading comprehension. Patient-KD beats vanilla KD. It produces 6- and 3-layer models that are much smaller and faster, with accuracy the authors describe as comparable to the teacher. This matters for running pre-trained models on limited hardware.

Key points

  • Footprint: BERT 6 has 67.0M parameters (1.64× smaller than BERT-Base) and BERT 3 has 45.7M (2.40× smaller). The embedding layer stays fixed at 23.8M parameters.
  • Inference speed is nearly linear in depth: 1.94× faster for BERT 6 and 3.73× for BERT 3. This was measured on 105k QNLI samples on one Titan RTX GPU (batch size 128, sequence length 128, FP16).
  • Teacher size matters little. Distilling BERT 6 [Base] with vanilla KD from BERT 12 versus BERT 24 gives similar GLUE scores (e.g., SST-2 91.5 vs 91.2, MNLI-m 79.7 vs 80.2).
  • A 6-layer student built from BERT-Large layers performs clearly worse. The authors attribute this to a harsher 4:1 compression ratio and an initialization mismatch.
  • Replacing vanilla KD with PKD-Skip improves the BERT 6 [Large] student on almost all tasks. This suggests the method works regardless of which teacher is chosen.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

308 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 299 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1908.09355 · arXiv 1908.09355

@misc{sun2019patient,
  title = {{Patient Knowledge Distillation for BERT Model Compression}},
  author = {Sun, Siqi and Cheng, Yu and Gan, Zhe and Liu, Jingjing},
  year = {2019},
  doi = {10.48550/arxiv.1908.09355},
  eprint = {1908.09355},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1908.09355}
}

References

40 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. TACL. find
  2. Soravit Changpinyo, Mark Sandler, and Andrey Zhmoginov. 2017. The power of sparsity in convolutional neural networks. arXiv preprint arXiv:1702.06257. — [arXiv:1702.06257](https://arxiv.org/abs/1702.06257) find
  3. Tianqi Chen, Ian J. Goodfellow, and Jonathon Shlens. 2015. Net2net: Accelerating learning via knowledge transfer. arXiv preprint arXiv:1511.05641. — [arXiv:1511.05641](https://arxiv.org/abs/1511.05641) find
  4. Yu Cheng, Felix X. Yu, Rogerio S. Feris, Sanjiv Kumar, Alok Choudhary, and Shi-Fu Chang. 2015. An exploration of parameter redundancy in deep networks with circulant projections. In ICCV. find
  5. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In EMNLP. find our card
  6. Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In NIPS. find
  7. Misha Denil, Babak Shakibi, Laurent Dinh, Marc’Aurelio Ranzato, and Nando de Freitas. 2013. Predicting parameters in deep learning. In NIPS. find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
  9. William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing. find our card
  10. Yunchao Gong, Liu Liu, Ming Yang, and Lubomir D. Bourdev. 2014. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115. — [arXiv:1412.6115](https://arxiv.org/abs/1412.6115) find
  11. Song Han, Huizi Mao, and William J Dally. 2016. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In ICLR. find
  12. Song Han, Jeff Pool, John Tran, and William J. Dally. 2015. Learning both weights and connections for efficient neural networks. In NIPS. find
  13. Yihui He, Xiangyu Zhang, and Jian Sun. 2017. Channel pruning for accelerating very deep neural networks. In ICCV. find
  14. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. find our card
  15. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In ACL. find
  16. Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. 2016. Deep networks with stochastic depth. In ECCV. find
  17. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In NIPS. find
  18. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683. find our card
  19. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019a. Improving multi-task deep neural networks via knowledge distillation for natural language understanding. arXiv preprint arXiv:1904.09482. — [arXiv:1904.09482](https://arxiv.org/abs/1904.09482) find
  20. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. find
  21. Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations. In ICLR. find our card
  22. Liang Lu, Michelle Guo, and Steve Renals. 2017. Knowledge distillation for small-footprint highway networks. In ICASSP. find
  23. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In NIPS. find our card
  24. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In NIPS. find
  25. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In EMNLP. find
  26. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL. find
  27. Antonio Polino, Razvan Pascanu, and Dan Alistarh. 2018. Model compression via distillation and quantization. arXiv preprint arXiv:1802.05668. — [arXiv:1802.05668](https://arxiv.org/abs/1802.05668) find
  28. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. arXiv. find
  29. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In EMNLP. find our card
  30. Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. 2015. Fitnets: Hints for thin deep nets. In ICLR. find
  31. Bharat Bhusan Sau and Vineeth N Balasubramanian. 2016. Deep model compression: Distilling knowledge from noisy teachers. arXiv preprint arXiv:1610.09650. — [arXiv:1610.09650](https://arxiv.org/abs/1610.09650) find
  32. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP. find
  33. Xu Tan, Yi Ren, Di He, Tao Qin, Zhou Zhao, and Tie-Yan Liu. 2019. Multilingual neural machine translation with knowledge distillation. In ICLR. find
  34. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS. find
  35. Dmitry P. Vetrov, Jonathan Huang, Li Zhang, Maxwell Collins, Michael Figurnov, Ruslan Salakhutdinov, and Yukun Zhu. 2017. Spatially adaptive computation time for residual networks. In CVPR. find
  36. Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. find our card
  37. Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426. find
  38. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237. find our card
  39. Shuangfei Zhai, Yu Cheng, Weining Lu, and Zhongfei Mark Zhang. 2016. Doubly convolutional neural networks. In NIPS. find
  40. Chenguang Zhu, Michael Zeng, and Xuedong Huang. 2018. Sdnet: Contextualized attention-based deep network for conversational question answering. arXiv preprint arXiv:1812.03593. — [arXiv:1812.03593](https://arxiv.org/abs/1812.03593) find