Elamur Research⌕
☰
home / papers / 2020 / TernaryBERT: Distillation-aware Ultra-low Bit BERT

TernaryBERT: Distillation-aware Ultra-low Bit BERT

Paper · 2020 · Wei Zhang, Lu Hou, Yichun Yin et al.

Summary

Fine-tuned BERT models are large: BERT-base has 109M parameters, over 400 MB in 32-bit floats. This makes them hard to run on phones and other constrained hardware. Earlier quantization work kept accuracy at 8 bits. Going lower usually needed mixed precision, which some hardware handles poorly, or product quantization, which adds a clustering step.

TernaryBERT limits the weights of a fine-tuned BERT to three values (−1, 0, +1) plus a scaling factor, so each weight needs only 2 bits. The authors adapt two existing ternarization schemes, the approximation-based TWN and the loss-aware LAT. They also test how finely to apply scaling in different parts of the model. Activations are quantized to 8 bits. To make up for the lost capacity, a full-precision teacher of the same size guides the ternary student. Distillation targets both the Transformer layer outputs and the final logits, while full-precision weight copies are kept for gradient updates.

On GLUE and SQuAD, TernaryBERT beats other BERT quantization methods and nearly matches full-precision accuracy while being 14.9x smaller. This shows that extremely low-bit BERT is practical for deployment.

Key points

  • Bit settings and granularity: weights and word embedding use 2 bits, activations use 8 bits. The word embedding is ternarized row-wise, and Transformer weights layer-wise. Min-max activation quantization did better than symmetric quantization on SQuAD v1.1 (EM/F1 79.9/87.4 vs 79.0/86.9).
  • What is left unquantized: segment and position embeddings, biases, softmax, layer normalization and the task-specific output layer stay in full precision.
  • MNLI-m results: TernaryBERT reaches 83.5% at 28 MB, and TernaryTinyBERT reaches 83.4% at 18 MB. For comparison, Q-BERT (2/3-bit mixed precision, 46 MB) gets 81.8% and GOBO with 2-bit weights gets 71.0%.
  • Effect of distillation: removing distillation on both Transformer layers and logits causes large drops, e.g. CoLA falls from 55.7 to 45.4 and RTE from 72.9 to 56.3.
  • Other helpful factors: starting from a fine-tuned rather than a pre-trained BERT, and using data augmentation, both improve results, most visibly on CoLA and RTE. Replacing LAT with 3-bit LAQ gives a 3-bit BERT that reaches 84.2% on MNLI-m at 41 MB.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Huawei TechnologiesShenzhen · China29 works · 9,991 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Computer Vision and Pattern Recognition · 1,282 works

🧭 Topics 1 Efficient models & compression · 77 works

Cite

Cite as Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang and Qun Liu. 2020. TernaryBERT: Distillation-aware Ultra-low Bit BERT. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.37

@inproceedings{zhang2020ternarybert,
  title = {{TernaryBERT: Distillation-aware Ultra-low Bit BERT}},
  author = {Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang and Qun Liu},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.37},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.37}
}

References

40 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, and Y. Bengio. 2016. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. In Advances in Neural Information Processing Systems. find
  2. M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and L. Kaiser. 2019. Universal transformers. In International Conference on Learning Representations. find
  3. J. Devlin, M. Chang, K. Lee, and K. Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171–4186. our card
  4. A. Fan, E. Grave, and A. Joulin. 2019. Reducing transformer depth on demand with structured dropout. In International Conference on Learning Representations. our card
  5. J. Kim, Y. Bhalgat, J. Lee, C. Patel, and N. Kwak. 2019. Qkd: Quantization-aware knowledge distillation. Preprint arXiv:1911.12491. — [arXiv:1911.12491](https://arxiv.org/abs/1911.12491) find
  6. Y. Kim and A. M. Rush. 2016. Sequence-level knowledge distillation. In Conference on Empirical Methods in Natural Language Processing. our card
  7. D. P. Kingma and J. Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations. find
  8. Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations. find
  9. Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations. find
  10. C. Leng, Z. Dou, H. Li, S. Zhu, and R. Jin. 2018. Extremely low bit neural network: Squeeze the last bit out with admm. In AAAI Conference on Artificial Intelligence. find
  11. F. Li, B. Zhang, and B. Liu. 2016. Ternary weight networks. Preprint arXiv:1605.04711. — [arXiv:1605.04711](https://arxiv.org/abs/1605.04711) find
  12. Z. Lin, M. Courbariaux, R. Memisevic, and Y. Bengio. 2016. Neural networks with few multiplications. In International Conference on Learning Representations. find
  13. W. Liu, P. Zhou, Z. Zhao, Z. Wang, H. Deng, and Q. Ju. 2020. Fastbert: a self-distilling bert with adaptive inference time. In Annual Conference of the Association for Computational Linguistics. our card
  14. A. Fan, P. Stock, B. Graham, E. Grave, R. Gribonval, H. Jegou, and A. Joulin. 2020. Training with quantization noise for extreme model compression. Preprint arXiv:2004.07320. — [arXiv:2004.07320](https://arxiv.org/abs/2004.07320) find
  15. I. Loshchilov and F. Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations. find
  16. G. Hinton, O. Vinyals, and J. Dean. 2015. Distilling the knowledge in a neural network. Preprint arXiv:1503.02531. our card
  17. X. Ma, P. Zhang, S. Zhang, N. Duan, Y. Hou, D. Song, and M. Zhou. 2019. A tensorized transformer for language modeling. In Advances in Neural Information Processing Systems. find
  18. L. Hou and J. T. Kwok. 2018. Loss-aware weight quantization of deep networks. In International Conference on Learning Representations. find
  19. L. Hou, Yao Q., and J. T. Kwok. 2017. Loss-aware binarization of deep networks. In International Conference on Learning Representations. find
  20. Y. Mao, Y. Wang, C. Wu, C. Zhang, Y. Wang, Y. Yang, Q. Zhang, Y. Tong, and J. Bai. 2020. Ladabert: Lightweight adaptation of bert through hybrid model compression. Preprint arXiv:2004.04124. find
  21. J. S. McCarley. 2019. Pruning a bert-based question answering model. Preprint arXiv:1910.06360. — [arXiv:1910.06360](https://arxiv.org/abs/1910.06360) find
  22. L. Hou, L. Shang, X. Jiang, and Q. Liu. 2020. Dynabert: Dynamic bert with adaptive width and depth. Preprint arXiv:2004.04037. find
  23. P. Michel, O. Levy, and G. Neubig. 2019. Are sixteen heads really better than one? Preprint arXiv:1905.10650. our card
  24. X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu. 2019. Tinybert: Distilling bert for natural language understanding. Preprint arXiv:1909.10351. find
  25. A. Polino, R. Pascanu, and D. Alistarh. 2018. Model compression via distillation and quantization. In International Conference on Learning Representations. find
  26. G. Prato, E. Charlaix, and M. Rezagholizadeh. 2019. Fully quantized transformer for improved translation. Preprint arXiv:1910.10485. — [arXiv:1910.10485](https://arxiv.org/abs/1910.10485) find
  27. O. Zafrir, G. Boudoukh, P. Izsak, and M. Wasserblat. 2019. Q8bert: Quantized 8bit bert. Preprint arXiv:1910.06188. — [arXiv:1910.06188](https://arxiv.org/abs/1910.06188) find
  28. P. Rajpurkar, R. Jia, and P. Liang. 2018. Know what you don’t know: Unanswerable questions for squad. Preprint arXiv:1806.03822. our card
  29. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. Preprint arXiv:1606.05250. our card
  30. M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi. 2016. Xnor-net: Imagenet classification using binary convolutional neural networks. In European Conference on Computer Vision, pages 525–542. find
  31. A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio. 2014. Fitnets: Hints for thin deep nets. Preprint arXiv:1412.6550. find
  32. V. Sanh, L. Debut, J. Chaumond, and T. Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. Preprint arXiv:1910.01108. find
  33. S. Shen, Z. Dong, J. Ye, L. Ma, Z. Yao, A. Gholami, M. W. Mahoney, and K. Keutzer. 2020. Q-bert: Hessian based ultra low precision quantization of bert. In AAAI Conference on Artificial Intelligence. find
  34. P. Stock, A. Joulin, R. Gribonval, B. Graham, and H. Jégou. 2020. And the bit goes down: Revisiting the quantization of neural networks. In International Conference on Learning Representations. find
  35. S. Sun, Y. Cheng, Z. Gan, and J. Liu. 2019. Patient knowledge distillation for bert model compression. In Conference on Empirical Methods in Natural Language Processing, pages 4314–4323. our card
  36. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998– 6008. our card
  37. E. Voita, D. Talbot, F. Moiseev, R. Sennrich, and I. Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Annual Conference of the Association for Computational Linguistics. our card
  38. A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. Preprint arXiv:1804.07461. our card
  39. W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Preprint arXiv:2002.10957. find
  40. A. H. Zadeh and A. Moshovos. 2020. Gobo: Quantizing attention-based nlp models for low latency and energy efficient inference. Preprint arXiv:2005.03842. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2020-ternarybert-distillation-aware-ultra-low/