Elamur Research⌕
☰
home / papers / 2021 / BinaryBERT: Pushing the Limit of BERT Quantization

BinaryBERT: Pushing the Limit of BERT Quantization

Paper · 2021 · Haoli Bai, Wei Zhang, Lu Hou et al.

Summary

Large pre-trained language models are hard to run on edge devices, and quantization shrinks them without redesigning the architecture. Earlier BERT quantization work reached ternary (2-bit) weights but never full 1-bit binarization. The authors show that binarizing weights directly causes a sudden accuracy loss. They trace this to the loss landscape, which becomes much steeper and more irregular for binary weights. They show this through surface visualizations and Hessian top-eigenvalue estimates. Their fix, ternary weight splitting (TWS), first trains a half-width ternary BERT. Its latent full-precision weights and quantized weights are then split, preserving equivalence, into a full-width binary model, which is fine-tuned further. An adaptive variant keeps the most important modules ternary under a size or FLOPs budget. On GLUE and SQuAD, BinaryBERT stays close to full-precision BERT-base while being 24× smaller, setting state-of-the-art BERT compression results. This makes very compact BERT models practical for constrained hardware.

Key points

  • Going from 32-bit to 2-bit weights (with 8-bit activations) costs about 0.6% on MRPC and 0.2% on MNLI-m. Moving to 1-bit costs roughly 3.8% and 0.9%, respectively.
  • For the MHA-O layers, the largest Hessian eigenvalue of the binary model is about 15× that of the full-precision one, indicating a much sharper loss surface.
  • The baselines are TWN (ternary), BWN (binary) and LAQ (8/4/3-bit), with uniformly quantized activations. TWS also beat LAB, BiReal and gradual-quantization BWN variants in most settings.
  • Fine-tuning after splitting from TWN 0.5× brought consistent gains on SQuAD v1.1, MNLI-m, QNLI and MRPC, under both 8-bit and 4-bit activations.
  • Adaptive splitting produces mixed ternary/binary models whose arithmetic precision remains easy to deploy. Its importance-based criterion outperformed other splitting criteria across model sizes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Chinese University of Hong KongHong Kong · China53 works · 12,921 citations🏢Huawei TechnologiesShenzhen · China29 works · 9,991 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Computer Vision and Pattern Recognition · 1,282 works

🧭 Topics 1 Efficient models & compression · 77 works

Cite

Cite as Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jin Jin, Xin Jiang, Qun Liu, Michael Lyu and Irwin King. 2021. BinaryBERT: Pushing the Limit of BERT Quantization. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.acl-long.334

@inproceedings{bai2021binarybert,
  title = {{BinaryBERT: Pushing the Limit of BERT Quantization}},
  author = {Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jin Jin, Xin Jiang, Qun Liu, Michael Lyu and Irwin King},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.334},
  url = {https://doi.org/10.18653/v1/2021.acl-long.334}
}

References

54 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. H. Bai, J. Wu, I. King, and M. Lyu. 2020. Few shot network compression via cross distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 3203–3210. find
  2. Y. Bai, Y. Wang, and E. Liberty. 2018. Proxquant: Quantized neural networks via proximal operators. In International Conference on Machine Learning. find
  3. A. Bulat, B. Martinez, and G. Tzimiropoulos. 2020. Bats: Binary architecture search. In European Conference on Computer Vision, pages 309–325. find
  4. T. Chen, I. Goodfellow, and J. Shlens. 2016. Net2net: Accelerating learning via knowledge transfer. In International Conference on Learning Representations. find
  5. M. Courbariaux, Y. Bengio, and J. David. 2015. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances in neural information processing systems. find
  6. M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and L. Kaiser. 2019. Universal transformers. In International Conference on Learning Representations. find
  7. J. Devlin, M. Chang, K. Lee, and K. Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics. our card
  8. Z. Dong, Z. Yao, A. Gholami, M. Mahoney, and K. Keutzer. 2019. Hawq: Hessian aware quantization of neural networks with mixed-precision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 293–302. find
  9. S. K. Esser, J. L. McKinstry, D. Bablani, R. Appuswamy, and D. S. Modha. 2019. Learned step size quantization. In International Conference on Learning Representations. find
  10. A. Fan, E. Grave, and A. Joulin. 2019. Reducing transformer depth on demand with structured dropout. In International Conference on Learning Representations. our card
  11. A. Fan, P. Stock, B. Graham, E. Grave, R. Gribonval, H. Jegou, and A. Joulin. 2020. Training with quantization noise for extreme model compression. Preprint arXiv:2004.07320. — [arXiv:2004.07320](https://arxiv.org/abs/2004.07320) find
  12. H. Habi, R. Jennings, and A. Netzer. 2020. Hmq: Hardware friendly mixed precision quantization block for cnns. In European Conference on Computer Vision, pages 448–463. find
  13. Y. Hao, L. Dong, F. Wei, and K. Xu. 2019. Visualizing and understanding the effectiveness of BERT. In Conference on Empirical Methods in Natural Language Processing. our card
  14. L. Hou, Z. Huang, L. Shang, X. Jiang, X. Chen, and Q. Liu. 2020. Dynabert: Dynamic bert with adaptive width and depth. In Advances in Neural Information Processing Systems. find
  15. L. Hou and J. T. Kwok. 2018. Loss-aware weight quantization of deep networks. In International Conference on Learning Representations. find
  16. L. Hou, Yao Q., and J. T. Kwok. 2017. Loss-aware binarization of deep networks. In International Conference on Learning Representations. find
  17. Z. Huang, L Hou, L. Shang, X. Jiang, X. Chen, and Q. Liu. 2021. Ghostbert: Generate more features with cheap operations for bert. In Annual Meeting of the Association for Computational Linguistics. find
  18. I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio. 2016. Binarized neural networks. In Advances in neural information processing systems. find
  19. X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu. 2020. Tinybert: Distilling bert for natural language understanding. In Findings of Empirical Methods in Natural Language Processing. find
  20. D. Kim, K Singh, and J. Choi. 2020. Learning architectures for binary networks. In European Conference on Computer Vision, pages 575–591. find
  21. H. Kim, K. Kim, J. Kim, and J. Kim. 2019. Bina-ryduo: Reducing gradient mismatch in binary activation network by coupling binary activations. In International Conference on Learning Representations. find
  22. Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations. find
  23. F. Li, B. Zhang, and B. Liu. 2016. Ternary weight networks. Preprint arXiv:1605.04711. — [arXiv:1605.04711](https://arxiv.org/abs/1605.04711) find
  24. H. Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein. 2018. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems. find
  25. Y. Li, X. Dong, and W. Wang. 2020a. Additive powersof-two quantization: a non-uniform discretization for neural networks. In International Conference on Learning Representations. find
  26. Y. Li, W. Wang, H. Bai, R. Gong, X. Dong, and F. Yu. 2020b. Efficient bitwidth search for practical mixed precision neural network. Preprint arXiv:2003.07577. — [arXiv:2003.07577](https://arxiv.org/abs/2003.07577) find
  27. H. Liu, K. Simonyan, and Y. Yang. 2019. Darts: Differentiable architecture search. In International Conference on Learning Representations. find
  28. Z. Liu, Z. Shen, M. Savvides, and K. Cheng. 2020. Reactnet: Towards precise binary neural network with generalized activation functions. In European Conference on Computer Vision, pages 143–159. find
  29. Z. Liu, B. Wu, W. Luo, X. Yang, W. Liu, and K. Cheng. 2018. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. In European Conference on Computer Vision. find
  30. X. Ma, P. Zhang, S. Zhang, N. Duan, Y. Hou, D. Song, and M. Zhou. 2019. A tensorized transformer for language modeling. In Advances in Neural Information Processing Systems. find
  31. B. Martinez, J. Yang, A. Bulat, and G. Tzimiropoulos. 2020. Training binary neural networks with real-to-binary convolutions. In International Conference on Learning Representations. find
  32. P. Michel, O. Levy, and G. Neubig. 2019. Are sixteen heads really better than one? In Advances in Neural Information Processing Systems. our card
  33. Y. Nahshan, B. Chmiel, C. Baskin, E. Zheltonozhskii, R. Banner, A. M. Bronstein, and A. Mendelson. 2019. Loss aware post-training quantization. Preprint arXiv:1911.07190. — [arXiv:1911.07190](https://arxiv.org/abs/1911.07190) find
  34. P. Rajpurkar, R. Jia, and P. Liang. 2018. Know what you don’t know: Unanswerable questions for squad. Preprint arXiv:1806.03822. our card
  35. J. Wang, H. Bai, J. Wu, X. Shi, J. Huang, I. King, M. Lyu, and J. Cheng. 2020. Revisiting parameter sharing for automatic neural channel number search. In Advances in Neural Information Processing Systems, volume 33. find
  36. K. Wang, Z. Liu, Y. Lin, J. Lin, and S. Han. 2019. Haq: Hardware-aware automated quantization with mixed precision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8612–8620. find
  37. B. Wu, Y. Wang, P. Zhang, Y. Tian, P. Vajda, and K. Keutzer. 2018a. Mixed precision quantization of convnets via differentiable neural architecture search. Preprint arXiv:1812.00090. — [arXiv:1812.00090](https://arxiv.org/abs/1812.00090) find
  38. J. Wu, Y. Zhang, H. Bai, H. Zhong, J. Hou, W. Liu, and J Huang. 2018b. Pocketflow: An automated framework for compressing and accelerating deep neural networks. In Advances in Neural Information Processing Systems, Workshop on Compact Deep Neural Networks with Industrial Applications. find
  39. L. Wu, D. Wang, and Q. Liu. 2019. Splitting steepest descent for growing neural architectures. In Advances in Neural Information Processing Systems, volume 32. find
  40. J. Xin, R. Tang, J. Lee, Y. Yu, and J. Lin. 2020. Deebert: Dynamic early exiting for accelerating bert inference. In Annual Meeting of the Association for Computational Linguistics. our card
  41. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. Preprint arXiv:1606.05250. our card
  42. A. H. Zadeh and A. Moshovos. 2020. Gobo: Quantizing attention-based nlp models for low latency and energy efficient inference. Preprint arXiv:2005.03842. find
  43. M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi. 2016. Xnor-net: Imagenet classification using binary convolutional neural networks. In European Conference on Computer Vision. find
  44. O. Zafrir, G. Boudoukh, P. Izsak, and M. Wasserblat. 2019. Q8bert: Quantized 8bit bert. Preprint arXiv:1910.06188. — [arXiv:1910.06188](https://arxiv.org/abs/1910.06188) find
  45. V. Sanh, L. Debut, J. Chaumond, and T. Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. Preprint arXiv:1910.01108. find
  46. D. Zhang, J. Yang, D. Ye, and G. Hua. 2018. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In European conference on computer vision, pages 365–382. find
  47. S. Shen, Z. Dong, J. Ye, L. Ma, Z. Yao, A. Gholami, M. W. Mahoney, and K. Keutzer. 2020. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence. find
  48. W. Zhang, L. Hou, Y. Yin, L. Shang, X. Chen, X. Jiang, and Q. Liu. 2020. Ternarybert: Distillation-aware ultra-low bit bert. In Conference on Empirical Methods in Natural Language Processing. our card
  49. S. Sun, Y. Cheng, Z. Gan, and J. Liu. 2019. Patient knowledge distillation for bert model compression. In Conference on Empirical Methods in Natural Language Processing. our card
  50. R. Zhao, Y. Hu, J. Dotzel, C. De Sa, and Z. Zhang. 2019. Improving neural network quantization without retraining using outlier channel splitting. In International Conference on Machine Learning. find
  51. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. our card
  52. A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. Preprint arXiv:1804.07461. our card
  53. S. Zhou, Y. Wu, Z. Ni, X. Zhou, H. Wen, and Y. Zou. 2016. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. Preprint arXiv:1606.06160. find
  54. W. Zhou, C. Xu, T. Ge, J. McAuley, K. Xu, and F. Wei. 2020. Bert loses patience: Fast and robust inference with early exit. In Advances in Neural Information Processing Systems. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bai-2021-binarybert-pushing-limit-bert-quantization/