Elamur Research⌕
☰
home / papers / 2020 / FastBERT: a Self-distilling BERT with Adaptive Inference Time

FastBERT: a Self-distilling BERT with Adaptive Inference Time

Summary

Models like BERT are accurate but expensive to serve. Fixed-size distilled students cannot adapt to fluctuating request volumes or to the fact that some inputs are easy and others hard. FastBERT addresses this by attaching a lightweight student classifier after each Transformer layer of a 12-layer BERT backbone, so a sample can exit early once a branch is confident. Training has three stages. First, the backbone is pre-trained, or an existing checkpoint is loaded. Second, the backbone and a final teacher classifier are fine-tuned on the task. Third, in self-distillation, the student branches learn to imitate the teacher's soft predictions via a summed KL-divergence loss, which requires no labels. At inference, one tunable Speed threshold sets how much uncertainty a sample may carry before it is passed to the next layer. On six Chinese and six English NLP datasets, FastBERT ran about 2–3× faster than BERT without accuracy loss. Accepting some degradation allows speedups anywhere from 1× to 12×, which suits industrial deployment with variable load.

Key points

  • Architecture: a BERT-compatible backbone with a teacher classifier and per-layer student classifiers. The teacher reduces dimensions from 768 to 128, then applies self-attention and a softmax output. Weights from BERT-WWM, RoBERTa or ERNIE can initialize the model.
  • Exit behavior: on the Book review dataset with Speed = 0.8, 61% of samples finished inference at the first layer (Transformer0).
  • Ablation, Book review:
    • FastBERT reached 86.98 accuracy at 2.23× speedup (Speed 0.2) and 85.69 at 6.01×.
    • Without self-distillation, it scored 86.22 at 2.19× and 85.02 at 5.08×.
  • Fixed depth without adaptive inference: two layers gave 82.88 (Book review) and 93.11 (Yelp.P) at 5.87×. On Yelp.P, adaptive FastBERT reached 94.67 at 7.90×.
  • Code: released at github.com/autoliuweijie/FastBERT.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

111 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 106 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.537

@inproceedings{liu2020fastbert,
  title = {{FastBERT: a Self-distilling BERT with Adaptive Inference Time}},
  author = {Liu, Weijie and Zhou, Peng and Wang, Zhiruo and Zhao, Zhe and Deng, Haotang and Ju, Qi},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.537},
  url = {https://doi.org/10.18653/v1/2020.acl-main.537}
}

References

29 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. ALBERT: A lite BERT for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942. find
  2. Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. 2020. K-BERT: Enabling language representation with knowledge graph. In Proceedings of AAAI. find our card
  3. Xin Liu, Qingcai Chen, Chong Deng, Huajun Zeng, Jing Chen, Dongfang Li, and Buzhou Tang. 2018. Lcqmc: A large-scale chinese question matching corpus. In Proceedings of the ICCL, pages 1952– 1962. find
  4. Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu. 2019. Pre-training with whole word masking for chinese BERT. arXiv preprint arXiv:1906.08101. find our card
  5. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of ACL, pages 4171–4186. find
  7. Ilya Loshchilov and Frank Hutter. decay regularization in adam. arXiv:1711.05101. find
  8. Michael Figurnov, Maxwell D Collins, Yukun Zhu, Li Zhang, Jonathan Huang, Dmitry Vetrov, and Ruslan Salakhutdinov. 2017. Spatially adaptive computation time for residual networks. In Proceedings of CVPR, pages 1790–1799. find
  9. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL-HLT, pages 2227–2237. find
  10. Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. 2014. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115. — [arXiv:1412.6115](https://arxiv.org/abs/1412.6115) find
  11. Yuanyuan Qiu, Hongzheng Li, Shen Li, Yingdi Jiang, Renfen Hu, and Lijiao Yang. 2018. Revisiting correlations between intrinsic and extrinsic evaluations of word embeddings. In Proceedings of CCL, pages 209–221. Springer. find
  12. Alex Graves. 2016. Adaptive computation time for recurrent neural networks. arXiv preprint arXiv:1603.08983. find
  13. Song Han, Jeff Pool, John Tran, and William Dally. 2015. Learning both weights and connections for efficient neural network. In Advances in NeurIPS, pages 1135–1143. find
  14. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. Computer Science, 14(7):38–39. find our card
  15. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019. TinyBERT: Distilling BERT for natural language understanding. arXiv preprint arXiv:1909.10351. find
  16. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of EMNLP, pages 1746–1751. find our card
  17. Fixing weight arXiv preprint Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. Technical report, OpenAI. find
  18. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. find
  19. Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. 2014. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550. find
  20. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter. In NeurIPS EMC2 Workshop. find
  21. Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019a. Patient knowledge distillation for bert model compression. In Proceedings of EMNLP-IJCNLP, pages 4314–4323. find our card
  22. Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019b. ERNIE: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223. find our card
  23. Surat Teerapittayanon, Bradley McDanel, and Hsiang-Tsung Kung. 2016. Branchynet: Fast inference via early exiting from deep neural networks. In Proceedings of ICPR, pages 2464–2469. find
  24. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in NeurIPS, pages 5998– 6008. find
  25. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of EMNLP, pages 353–355. find our card
  26. Baoxin Wang. 2018. Disconnected recurrent neural networks for text categorization. In Proceedings of ACL, pages 2311–2320. find
  27. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237. find our card
  28. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in NeurIPS, pages 649–657. find our card
  29. Zhe Zhao, Hui Chen, Jinbin Zhang, Xin Zhao, Tao Liu, Wei Lu, Xi Chen, Haotang Deng, Qi Ju, and Xiaoyong Du. 2019. UER: An open-source toolkit for pre-training models. In Proceedings of EMNLP-IJCNLP 2019, page 241. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2020-fastbert-self-distilling-bert-adaptive/