Elamur Research⌕
☰
home / papers / 2022 / The case for 4-bit precision: k-bit Inference Scaling Laws

The case for 4-bit precision: k-bit Inference Scaling Laws

Summary

Quantizing an LLM shrinks its memory use and, at small batch sizes, its latency. This raises a practical question: for a fixed budget of total model bits, is it better to run a larger model at low precision or a smaller one at higher precision? Dettmers and Zettlemoyer frame this as inference scaling laws that relate total bits to mean zero-shot accuracy. They ran over 35,000 zero-shot quantization experiments on OPT, BLOOM, BLOOMZ, Pythia/NeoX and GPT-2 models, ranging from 19M to 176B parameters. Weights were quantized to 3–16 bits while inputs stayed at 16 bits. The experiments also varied the data type (Int, Float, quantile, dynamic exponent), the block size, and an outlier-aware scheme the authors call proxy quantization. Across every model family and scale, moving from 16 down to 4 bits improves accuracy per bit, but at 3 bits the trend reverses. Only smaller blocks and the choice of data type improved the scaling curves. The practical advice is to deploy 4-bit models and adjust model size, not precision, to hit a memory or accuracy target.

Key points

  • Scope: more than 35,000 experiments covering 19M–176B parameters in the OPT, BLOOM/BLOOMZ, Pythia/NeoX and GPT-2 families, all using zero-shot quantization (no calibration data or optimization).
  • Optimal precision: 4-bit weights were almost universally best for accuracy per total model bit, and 3-bit performed worse. None of the tested methods improved scaling at 6–8 bits.
  • What helps: quantile quantization and Float data types scaled best, while Int and dynamic exponent scaled worse. Block sizes of 64–128 were generally optimal. A block size of 64 with 16-bit constants adds about 0.25 bits per parameter.
  • Outliers: proxy quantization stabilized unstable 3-bit OPT and Pythia models but still did not beat 4-bit.
  • One-shot methods: 2-bit GPTQ with blocking outperformed zero-shot 3-bit Float, which marks one-shot methods as promising below 4 bits. Repeating the study with GPTQ on just OPT-175B and BLOOM-176B would cost an estimated 5,120 GPU days.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics Efficient models & compression

Cite

Cite as Tim Dettmers and Luke Zettlemoyer. 2022. The case for 4-bit precision: k-bit Inference Scaling Laws. arXiv:2212.09720.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2212.09720 · arXiv 2212.09720

@misc{dettmers2022case,
  title = {{The case for 4-bit precision: k-bit Inference Scaling Laws}},
  author = {Tim Dettmers and Luke Zettlemoyer},
  year = {2022},
  doi = {10.48550/arxiv.2212.09720},
  eprint = {2212.09720},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2212.09720}
}

References

51 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bisk, Y., Zellers, R., LeBras, R., Gao, J., and Choi, Y. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth A… our card
  2. AAAI Press, 2020. URL https://ojs.aaai.org/ index.php/AAAI/article/view/6239. ↗
  3. Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al. Gpt-neox-20b: An open-source autoregressive language model. arXiv preprint arXiv:2204.06745, 2022. our card
  4. Bondarenko, Y., Nagel, M., and Blankevoort, T. Understanding and overcoming the challenges of efficient transformer quantization. arXiv preprint arXiv:2109.12948, 2021. find
  5. Choi, Y., El-Khamy, M., and Lee, J. Towards the limit of network quantization. arXiv preprint arXiv:1612.01543, 2016. — [arXiv:1612.01543](https://arxiv.org/abs/1612.01543) find
  6. Dettmers, T. 8-bit approximations for parallelism in deep learning. International Conference on Learning Representations (ICLR), 2016. find
  7. Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. LLM.int8(): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, 2022a. find
  8. Dettmers, T., Lewis, M., Shleifer, S., and Zettlemoyer, L. 8-bit optimizers via block-wise quantization. 9th International Conference on Learning Representations, ICLR, 2022b. our card
  9. Dongarra, J. A not so simple matter of software. https://www.youtube.com/watch?v= cSO0Tc2w5Dg, November 2022. ↗
  10. Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022. find
  11. Gao, J., He, D., Tan, X., Qin, T., Wang, L., and Liu, T.-Y. Representation degeneration problem in training natural language generation models. arXiv preprint arXiv:1907.12009, 2019. our card
  12. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. find
  13. Gao, L., Tow, J., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., McDonell, K., Muennighoff, N., Phang, J., Reynolds, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, September 2021. URL https://doi.org/10.5281/zenod… — [doi:10.5281/zenodo.5371628](https://doi.org/10.5281/zenodo.5371628) ↗
  14. Gong, Y., Liu, L., Yang, M., and Bourdev, L. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115, 2014. — [arXiv:1412.6115](https://arxiv.org/abs/1412.6115) find
  15. Han, S., Mao, H., and Dally, W. J. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015. find
  16. Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701, 2020. our card
  17. Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M., Ali, M., Yang, Y., and Zhou, Y. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017. our card
  18. Hou, L., Yao, Q., and Kwok, J. T. Loss-aware binarization of deep networks. arXiv preprint arXiv:1611.01600, 2016. — [arXiv:1611.01600](https://arxiv.org/abs/1611.01600) find
  19. Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D. Quantization and training of neural networks for efficient integer-arithmetic-only inference. arxiv e-prints, art. arXiv preprint arXiv:1712.05877, 2017. find
  20. Jain, S., Gural, A., Wu, M., and Dick, C. Trained quantization thresholds for accurate and efficient fixed-point inference of deep neural networks. Proceedings of Machine Learning and Systems, 2:112–128, 2020. find
  21. Jaszczur, S., Chowdhery, A., Mohiuddin, A., Kaiser, L., Gajewski, W., Michalewski, H., and Kanerva, J. Sparse is enough in scaling transformers. Advances in Neural Information Processing Systems, 34:9895–9907, 2021. find
  22. Jia, Z., Maggioni, M., Smith, J., and Scarpazza, D. P. Dissecting the nvidia turing t4 gpu via microbenchmarking. arXiv preprint arXiv:1903.07486, 2019. find
  23. Jin, Q., Ren, J., Zhuang, R., Hanumante, S., Li, Z., Chen, Z., Wang, Y., Yang, K., and Tulyakov, S. F8net: Fixed-point 8-bit only multiplication for network quantization. arXiv preprint arXiv:2202.05239, 2022. — [arXiv:2202.05239](https://arxiv.org/abs/2202.05239) find
  24. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. our card
  25. Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Levskaya, A., Heek, J., Xiao, K., Agrawal, S., and Dean, J. Efficiently scaling transformer inference. arXiv preprint arXiv:2211.05102, 2022. find
  26. Kovaleva, O., Kulshreshtha, S., Rogers, A., and Rumshisky, A. Bert busters: Outlier dimensions that disrupt transformers. arXiv preprint arXiv:2105.06990, 2021. find
  27. Puccetti, G., Rogers, A., Drozd, A., and Dell’Orletta, F. Outliers dimensions that disrupt transformers are driven by frequency. arXiv preprint arXiv:2205.11380, 2022. — [arXiv:2205.11380](https://arxiv.org/abs/2205.11380) find
  28. Krishnamoorthi, R. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint arXiv:1806.08342, 2018. — [arXiv:1806.08342](https://arxiv.org/abs/1806.08342) find
  29. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. find
  30. Leng, C., Dou, Z., Li, H., Zhu, S., and Jin, R. Extremely low bit neural network: Squeeze the last bit out with admm. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018. find
  31. Rastegari, M., Ordonez, V., Redmon, J., and Farhadi, A. Xnor-net: Imagenet classification using binary convolutional neural networks. In Leibe, B., Matas, J., Sebe, N., and Welling, M. (eds.), Computer Vision -ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Pro… find
  32. Springer, 2016. doi: 10.1007/ 978-3-319-46493-0\\ 32. URL https://doi.org/ 10.1007/978-3-319-46493-0_32. — [doi:10.1007/978-3-319-46493-0_32](https://doi.org/10.1007/978-3-319-46493-0_32) find
  33. Luo, Z., Kulmizev, A., and Mao, X. Positional artefacts propagate through masked language model embeddings. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), p… find
  34. Rosenfeld, J. S., Rosenfeld, A., Belinkov, Y., and Shavit, N. A constructive prediction of the generalization error across scales. arXiv preprint arXiv:1909.12673, 2019. find
  35. Micikevicius, P., Stosic, D., Burgess, N., Cornea, M., Dubey, P., Grisenthwaite, R., Ha, S., Heinecke, A., Judd, P., Kamalu, J., et al. Fp8 formats for deep learning. arXiv preprint arXiv:2209.05433, 2022. find
  36. Rusci, M., Capotondi, A., and Benini, L. Memory-driven mixed low precision quantization for enabling deep network inference on microcontrollers. Proceedings of Machine Learning and Systems, 2:326–335, 2020. find
  37. Nagel, M., Baalen, M. v., Blankevoort, T., and Welling, M. Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1325– 1334, 2019. find
  38. Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: an adversarial winograd schema challenge at scale. Commun. ACM, 64(9):99–106, 2021. doi: 10.1145/3474381. URL https://doi.org/10. 1145/3474381. find
  39. Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N. Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume… our card
  40. Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon, F., Gallé, M., et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022. find
  41. Park, E., Ahn, J., and Yoo, S. Weighted-entropy-based quantization for deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5456–5464, 2017. find
  42. Wei, X., Zhang, Y., Zhang, X., Gong, R., Zhang, S., Zhang, Q., Yu, F., and Liu, X. Outlier suppression: Pushing the limit of low-bit transformer language models. arXiv preprint arXiv:2209.13325, 2022. — [arXiv:2209.13325](https://arxiv.org/abs/2209.13325) find
  43. Park, G., Park, B., Kwon, S. J., Kim, B., Lee, Y., and Lee, D. nuqmm: Quantized matmul for efficient inference of large-scale generative language models. arXiv preprint arXiv:2206.09557, 2022. find
  44. Wu, H., Judd, P., Zhang, X., Isaev, M., and Micikevicius, P. Integer quantization for deep learning inference: Principles and empirical evaluation. arXiv preprint arXiv:2004.09602, 2020. find
  45. Timkey, W. and van Schijndel, M. All bark and no bite: Rogue dimensions in transformer language models obscure representational quality. arXiv preprint arXiv:2109.04404, 2021. find
  46. Xiao, G., Lin, J., Seznec, M., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. arXiv preprint arXiv:2211.10438, 2022. find
  47. Yao, Z., Aminabadi, R. Y., Zhang, M., Wu, X., Li, C., and He, Y. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. arXiv preprint arXiv:2206.01861, 2022. our card
  48. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? In Korhonen, A., Traum, D. R., and Màrquez, L. (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August… find
  49. Zeng, A., Liu, X., Du, Z., Wang, Z., Lai, H., Ding, M., Yang, Z., Xu, Y., Zheng, W., Xia, X., et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414, 2022. our card
  50. Zhang, D., Yang, J., Ye, D., and Hua, G. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In Proceedings of the European conference on computer vision (ECCV), pp. 365–382, 2018. find
  51. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dettmers-2022-case-4-bit-precision-k/