Elamur Research
☰
home / papers / 2020 / DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference

DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.acl-main.204.pdf

Summary

Pre-trained transformers such as BERT and RoBERTa are accurate but slow at inference, which makes real-time and on-device use difficult. DeeBERT tackles this by letting easy inputs leave the network early. It attaches a small classifier, called an "off-ramp", after every transformer layer. Fine-tuning has two stages. First, the model is fine-tuned as usual with only the final classifier. Then those weights are frozen, and the intermediate off-ramps are trained on the sum of their cross-entropy losses. At inference, each off-ramp computes the entropy of its prediction. If the entropy falls below a threshold S, the answer is returned; otherwise the input moves to the next layer. Raising S trades accuracy for speed.

The authors tested the method with BERT and RoBERTa on six GLUE tasks. It cut inference time by up to about 40% with very small quality losses. Unlike distillation or LayerDrop, it needs no extra pre-training. It also gives users a range of speed–accuracy settings rather than a single compressed model. Layer-by-layer analysis further shows that some transformer layers are redundant.

Key points

  • Evaluation covered SST-2, MRPC, QNLI, RTE, QQP and MNLI from GLUE, using BERT-base and RoBERTa-base, plus large variants in the analysis. The implementation builds on the HuggingFace Transformers library, and timing was measured on one NVIDIA Tesla P100.
  • On QQP, BERT-base saved about 24% of runtime with no F1 loss, and RoBERTa-base saved about 32% with a +0.1 F1 change. Larger savings, around 44–49%, cost roughly 1–3 points.
  • The paper introduces an "expected saving" metric, defined as the fraction of transformer-layer computations skipped. It correlates linearly with measured wall-clock savings.
  • BERT's per-layer accuracy rises steadily with depth. RoBERTa's accuracy stays flat for several layers, then jumps and saturates sooner. This lets RoBERTa save more time at first, but its quality degrades faster beyond a turning point.
  • Code is released at github.com/castorini/DeeBERT.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

151 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 147 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.204

@inproceedings{xin2020deebert,
  title = {{DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference}},
  author = {Xin, Ji and Tang, Raphael and Lee, Jaejun and Yu, Yaoliang and Lin, Jimmy},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.204},
  url = {https://doi.org/10.18653/v1/2020.acl-main.204}
}

References

19 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. B. Barla Cambazoglu, Hugo Zaragoza, Olivier Chapelle, Jiang Chen, Ciya Liao, Zhaohui Zheng, and Jon Degenhardt. 2010. Early exit optimizations for additive machine learned ranking systems. In Proceedings of the Third ACM International Conference on Web Search and Data Mining (WSDM 2010), pages 411–4… find
  2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  3. Angela Fan, Edouard Grave, and Armand Joulin. 2019. Reducing transformer depth on demand with structured dropout. arXiv:1909.11556. find our card
  4. Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Weinberger. 2018. Multi-scale dense networks for resource efficient image classification. In Proceedings of the 6th International Conference on Learning Representations, Vancouver, BC, Canada. find
  5. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019. TinyBERT: Distilling BERT for natural language understanding. arXiv:1909.10351. find
  6. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv:1907.11692. find
  7. Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? In Advances in Neural Information Processing Systems 32, pages 14014–14024. find our card
  8. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… find
  9. Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. 2019. Q-BERT: Hessian based ultra low precision quantization of BERT. arXiv:1909.05840. — [arXiv:1909.05840](https://arxiv.org/abs/1909.05840) find
  10. Raphael Tang, Yao Lu, and Jimmy Lin. 2019. Natural language generation for effective knowledge distillation. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 202–208, Hong Kong, China. find
  11. Surat Teerapittayanon, Bradley McDanel, and Hsiang-Tsung Kung. 2017. BranchyNet: Fast inference via early exiting from deep neural networks. arXiv:1709.01686. — [arXiv:1709.01686](https://arxiv.org/abs/1709.01686) find
  12. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30, pages 5998–6008. find
  13. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797–5808, Florence, I… find our card
  14. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,… find our card
  15. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. HuggingFace’s Transformers: State-of-the-art natural language processing. arXiv:1910.03771. find our card
  16. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: generalized autoregressive pretraining for language understanding. arXiv:1906.08237. find our card
  17. Matthew D. Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In Proceedings of the 13th European Conference on Computer Vision (ECCV 2014), pages 818–833, Zürich, Switzerland. find our card
  18. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog. find
  19. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv:1910.01108. find