Elamur Research
☰
home / papers / 2021 / 8-bit Optimizers via Block-wise Quantization

8-bit Optimizers via Block-wise Quantization

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2110.02861

Summary

Stateful optimizers such as Adam and SGD with momentum store per-parameter gradient statistics. These statistics take up a large share of training memory (33–75% by the authors' estimate), which limits how big a model can fit on given hardware. Dettmers et al. show that these statistics can be kept in 8 bits instead of 32 without hurting results.

Their method, block-wise dynamic quantization, has three parts:

  • Block-wise quantization: tensors are cut into small blocks, and each block is normalized by its own absolute maximum. This confines outliers to one block and avoids synchronization across GPU cores.
  • Dynamic quantization: a non-linear scheme that keeps precision for both tiny and large values.
  • Stable embedding layer: reduces gradient variance caused by skewed token frequencies.

The 8-bit optimizers matched 32-bit quality across many tasks without retuning hyperparameters. These included 1.5B-parameter language modeling, GLUE finetuning, ImageNet, WMT'14/WMT'16 translation, MoCo v2 and RoBERTa pretraining. The authors release CUDA kernels and a PyTorch implementation that needs only a two-line code change. This makes training or finetuning larger models feasible on limited GPU memory.

Key points

  • Memory savings: 32-bit Momentum and Adam need 4 and 8 bytes per parameter (4 GB / 8 GB for a 1B model). The 8-bit versions cut this to 1 GB and 2 GB.
  • Scale of the problem: Adam states for the largest GPT-2 and T5 models occupy 11 GB and 41 GB, respectively.
  • Blocks contain 2048 elements. State conversion between 8 and 32 bits happens in registers, which makes the 8-bit optimizers faster on GPUs than standard 32-bit ones.
  • Ablations indicate that block-wise quantization, dynamic quantization and the stable embedding layer are each necessary. Within the stable embedding layer, Xavier initialization and layer norm both lower perplexity.
  • Limitation: on a 209M-parameter model trained on the RoBERTa corpus, 8-bit Adam matched 32-bit Adam (16.4 vs 16.7 perplexity). However, 8-bit AdaGrad lagged its 32-bit version (19.7 vs 19.4).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Computing & GPUs

Cited by

172 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 167 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2110.02861 · arXiv 2110.02861

@misc{dettmers20218,
  title = {{8-bit Optimizers via Block-wise Quantization}},
  author = {Dettmers, Tim and Lewis, Mike and Shleifer, Sam and Zettlemoyer, Luke},
  year = {2021},
  doi = {10.48550/arxiv.2110.02861},
  eprint = {2110.02861},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2110.02861}
}

References

67 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anil, R., Gupta, V., Koren, T., and Singer, Y. (2019). Memory efficient adaptive optimization. In Wallach, H. M., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E. B., and Garnett, R., editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Pro… find
  2. Ba, J. L., Kiros, J. R., and Hinton, G. E. (2016). arXiv:1607.06450. find our card
  3. Layer normalization. arXiv preprint Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. arXiv preprint arXiv:2005.14165. find
  4. Cambier, L., Bhiwandiwalla, A., Gong, T., Elibol, O. H., Nekuii, M., and Tang, H. (2020). Shifted and squeezed 8-bit floating point format for low-precision training of deep neural networks. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2… find
  5. Chen, E. J. and Kelton, W. D. (2001). Quantile and histogram estimation. In Proceeding of the 2001 Winter Simulation Conference (Cat. No. 01CH37304), volume 1, pages 451–459. IEEE. find
  6. Chen, T., Xu, B., Zhang, C., and Guestrin, C. (2016). Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174. find
  7. Chen, X., Agarwal, N., Hazan, E., Zhang, C., and Zhang, Y. (2020a). Extreme tensoring for low-memory preconditioning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. find
  8. Chen, X., Fan, H., Girshick, R., and He, K. (2020b). Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297. — [arXiv:2003.04297](https://arxiv.org/abs/2003.04297) find
  9. Choi, J., Venkataramani, S., Srinivasan, V., Gopalakrishnan, K., Wang, Z., and Chuang, P. (2019). Accurate and efficient 2-bit quantized neural networks. In Talwalkar, A., Smith, V., and Zaharia, M., editors, Proceedings of Machine Learning and Systems 2019, MLSys 2019, Stanford, CA, USA, March 31 -… find
  10. Courbariaux, M. and Bengio, Y. (2016). Binarynet: Training deep neural networks with weights and activations constrained to +1 or -1. CoRR, abs/1602.02830. find
  11. Courbariaux, M., Bengio, Y., and David, J. (2015). Binaryconnect: Training deep neural networks with binary weights during propagations. In Cortes, C., Lawrence, N. D., Lee, D. D., Sugiyama, M., and Garnett, R., editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neur… find
  12. Courbariaux, M., Bengio, Y., and David, J.-P. (2014). Training deep neural networks with low precision multiplications. arXiv preprint arXiv:1412.7024. — [arXiv:1412.7024](https://arxiv.org/abs/1412.7024) find
  13. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee. find
  14. Dettmers, T. (2016). 8-bit approximations for parallelism in deep learning. International Conference on Learning Representations (ICLR). find
  15. Devlin, J., Chang, M., Lee, K., and Toutanova, K. (2019). BERT: pre-training of deep bidirectional transformers for language understanding. In Burstein, J., Doran, C., and Solorio, T., editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Ling… find
  16. Drumond, M., Lin, T., Jaggi, M., and Falsafi, B. (2018). Training dnns with hybrid block floating point. In Bengio, S., Wallach, H. M., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R., editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Informati… find
  17. Duchi, J., Hazan, E., and Singer, Y. (2011). Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7). find
  18. Dunning, T. and Ertl, O. (2019). Computing extremely accurate quantiles using t-digests. arXiv preprint arXiv:1902.04023. — [arXiv:1902.04023](https://arxiv.org/abs/1902.04023) find
  19. Fedus, W., Zoph, B., and Shazeer, N. (2021). Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961. find
  20. Glorot, X. and Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256. JMLR Workshop and Conference Proceedings. find
  21. Gokaslan, A. and Cohen, V. (2019). Openwebtext corpus. find
  22. Gomez, A. N., Ren, M., Urtasun, R., and Grosse, R. B. (2017). The reversible residual network: Backpropagation without storing activations. arXiv preprint arXiv:1707.04585. — [arXiv:1707.04585](https://arxiv.org/abs/1707.04585) find
  23. Gong, R., Liu, X., Jiang, S., Li, T., Hu, P., Lin, J., Yu, F., and Yan, J. (2019). Differentiable soft quantization: Bridging full-precision and low-bit neural networks. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, page… find
  24. Govindaraju, N. K., Raghuvanshi, N., and Manocha, D. (2005). Fast and approximate stream mining of quantiles and frequencies using graphics processors. In Proceedings of the 2005 ACM SIGMOD international conference on Management of data, pages 611–622. find
  25. Greenwald, M. and Khanna, S. (2001). Space-efficient online computation of quantile summaries. ACM SIGMOD Record, 30(2):58–66. find
  26. Harlap, A., Narayanan, D., Phanishayee, A., Seshadri, V., Devanur, N., Ganger, G., and Gibbons, P. (2018). Pipedream: Fast and efficient pipeline parallel dnn training. arXiv preprint arXiv:1806.03377. find
  27. He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778. find
  28. Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., et al. (2020). Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701. find
  29. Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, M. X., Chen, D., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., et al. (2018). Gpipe: Efficient training of giant neural networks using pipeline parallelism. arXiv preprint arXiv:1811.06965. find
  30. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. find
  31. Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., and Socher, R. (2019). CTRL: A conditional transformer language model for controllable generation. CoRR, abs/1909.05858. find our card
  32. Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  33. Krizhevsky, A., Hinton, G., et al. (2009). Learning multiple layers of features from tiny images. find
  34. Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. (2020). Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668. find
  35. Lewis, M., Bhosale, S., Dettmers, T., Goyal, N., and Zettlemoyer, L. (2021). Base layers: Simplifying training of large, sparse models. arXiv preprint arXiv:2103.16716. find
  36. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. (2020). BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. R., editors… find our card
  37. Li, J. B., Qu, S., Li, X., Strubell, E., and Metze, F. (2021). End-to-end quantized training via log-barrier extensions. find
  38. Li, R., Wang, Y., Liang, F., Qin, H., Yan, J., and Fan, R. (2019). Fully quantized network for object detection. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 2810–2819. Computer Vision Foundation / IEEE. find
  39. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  40. Loshchilov, I. and Hutter, F. (2018). Fixing weight decay regularization in adam. find
  41. Macháček, M. and Bojar, O. (2014). Results of the wmt14 metrics shared task. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 293–301. find our card
  42. Mellempudi, N., Srinivasan, S., Das, D., and Kaul, B. (2019). Mixed precision training with 8-bit floating point. CoRR, abs/1905.12334. — [arXiv:1905.12334](https://arxiv.org/abs/1905.12334) find
  43. Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Houston, M., Kuchaiev, O., Venkatesh, G., et al. (2017). Mixed precision training. arXiv preprint arXiv:1710.03740. find
  44. Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., and Auli, M. (2019). fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint arXiv:1904.01038. find our card
  45. Ott, M., Edunov, S., Grangier, D., and Auli, M. (2018). Scaling neural machine translation. arXiv preprint arXiv:1806.00187. find our card
  46. Pappalardo, A. (2021). Xilinx/brevitas. find
  47. Pudipeddi, B., Mesmakhosroshahi, M., Xi, J., and Bharadwaj, S. (2020). Training large neural networks with constant memory using a new execution algorithm. arXiv preprint arXiv:2002.05645. find
  48. Qian, N. (1999). On the momentum term in gradient descent learning algorithms. Neural networks : the official journal of the International Neural Network Society, 12 1:145–151. find
  49. Qin, H., Gong, R., Liu, X., Bai, X., Song, J., and Sebe, N. (2020). Binary neural networks: A survey. CoRR, abs/2004.03333. — [arXiv:2004.03333](https://arxiv.org/abs/2004.03333) find
  50. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  51. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2019). Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. find
  52. Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. (2020). Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE. find
  53. Rajbhandari, S., Ruwase, O., Rasley, J., Smith, S., and He, Y. (2021). Zero-infinity: Breaking the gpu memory wall for extreme scale deep learning. arXiv preprint arXiv:2104.07857. find
  54. Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. (2021). Zero-shot text-to-image generation. arXiv preprint arXiv:2102.12092. find our card
  55. Rastegari, M., Ordonez, V., Redmon, J., and Farhadi, A. (2016). Xnor-net: Imagenet classification using binary convolutional neural networks. In Leibe, B., Matas, J., Sebe, N., and Welling, M., editors, Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14… find
  56. Sennrich, R., Haddow, B., and Birch, A. (2015). Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909. find
  57. Sennrich, R., Haddow, B., and Birch, A. (2016). Edinburgh neural machine translation systems for wmt 16. arXiv preprint arXiv:1606.02891. find our card
  58. Shazeer, N. and Stern, M. (2018). Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pages 4596–4604. PMLR. find
  59. Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. (2019). Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053. find
  60. Sun, X., Choi, J., Chen, C., Wang, N., Venkataramani, S., Srinivasan, V., Cui, X., Zhang, W., and Gopalakrishnan, K. (2019). Hybrid 8-bit floating point (HFP8) training and inference for deep neural networks. In Wallach, H. M., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E. B., and Garne… find
  61. Trinh, T. H. and Le, Q. V. (2018). A simple method for commonsense reasoning. arXiv preprint arXiv:1806.02847. find
  62. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. arXiv preprint arXiv:1706.03762. find
  63. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. (2018a). Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. find our card
  64. Wang, N., Choi, J., Brand, D., Chen, C., and Gopalakrishnan, K. (2018b). Training deep neural networks with 8-bit floating point numbers. In Bengio, S., Wallach, H. M., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R., editors, Advances in Neural Information Processing Systems 31: Annu… find
  65. Wenzek, G., Lachaux, M.-A., Conneau, A., Chaudhary, V., Guzmán, F., Joulin, A., and Grave, E. (2020). CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4003–4012, Marseille, France. European Langua… find our card
  66. Zhu, C., Han, S., Mao, H., and Dally, W. J. (2017). Trained ternary quantization. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net. find
  67. Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., and Fidler, S. (2015). Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19–27. find our card