Elamur Research⌕
☰
home / papers / 2023 / Atom: Low-bit Quantization for Efficient and Accurate LLM Serving

Atom: Low-bit Quantization for Efficient and Accurate LLM Serving

Paper · 2023 · Yilong Zhao, Chien-Yu Lin, Kan Zhu et al.

Summary

LLM serving relies on batching to keep GPUs busy, and quantization can further reduce memory use and compute cost. Current methods fall short. Weight-only schemes still multiply in FP16. INT8 weight-activation schemes leave INT4 Tensor Cores unused. Earlier 4-bit attempts such as OmniQuant and QLLM lose noticeable perplexity. Atom targets 4-bit quantization of both weights and activations (W4A4) without that loss.

It keeps a small set of outlier channels at higher precision (8-bit), after reordering channels so memory access stays regular. It also uses fine-grained group quantization, computes activation scales dynamically for each input, and stores the KV-cache in low bit-width. Quantization and reordering are fused into existing operators. Custom CUDA kernels for low-bit Tensor Cores are integrated, together with FlashInfer attention, into an end-to-end serving framework.

On Llama models from 7B to 65B, Atom stays close to FP16 quality. At the same latency target, it raises serving throughput by up to 7.73× over FP16 and 2.53× over INT8. This shows that accurate 4-bit serving is practical.

Key points

  • On Llama-65B with 4-bit weights and activations, the average zero-shot accuracy drop is about 1.4%, and WikiText2 perplexity rises by only about 0.3.
  • At similar latency, end-to-end throughput improves by up to 7.7× vs FP16, 5.5× vs W4A16 (weight-only), and 2.5× vs W8A8.
  • The motivation is hardware peak throughput on A100: INT4 reaches 1248 TOPS, INT8 reaches 624 TOPS, and FP16 Tensor Cores reach only 312 TFLOPS.
  • A runtime profile of Llama-7b shows that dense layers plus self-attention take over 90% of execution time. These are therefore the parts worth quantizing.
  • Symmetric quantization is used to avoid the extra cross-term computations that asymmetric zero points would add to integer matrix multiplication.

Interesting ideas

  • I1. Reorder channels before mixed precision: Outlier channels are moved to the end of the matrix so the high-precision part forms one contiguous block. This keeps mixed-precision kernels regular and fast, a trick that applies to any format mixing two precisions.
  • I2. Fuse cheap preprocessing into neighbouring kernels: Dynamic quantization and reordering are folded into operators that already run, which hides their cost. This is a general way to make per-input adaptive steps affordable.
  • I3. Use a middle tier for outliers: Outliers are stored in INT8 rather than FP16, trading a little accuracy for much better efficiency. A graded precision scheme like this may beat a strict "low vs full" split elsewhere too.
  • I4. Profile before optimizing: The paper first breaks down where serving time goes across batch sizes, then targets those layers. This is a useful evaluation habit for any efficiency project.
  • I5. Measure throughput under a fixed latency target: Results are reported at matched latency rather than as raw speed alone. This gives a fairer, deployment-relevant comparison between methods.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States598 works · 191,225 citations🎓University of WashingtonSeattle · United States402 works · 212,762 citations🎓Peking UniversityBeijing · China315 works · 43,268 citations🎓Shanghai Jiao Tong UniversityShanghai · China212 works · 19,544 citations

Also on the byline: Seattle University.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 worksComputational Theory and Mathematics · 321 worksEngineering · 188 worksElectrical and Electronic Engineering · 33 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksEfficient models & compression · 297 works

Cite

Cite as Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen and Baris Kasikci. 2023. Atom: Low-bit Quantization for Efficient and Accurate LLM Serving. MLSys 2024.

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 2310.19102

@inproceedings{zhao2023atom,
  title = {{Atom: Low-bit Quantization for Efficient and Accurate LLM Serving}},
  author = {Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen and Baris Kasikci},
  booktitle = {MLSys 2024},
  year = {2023},
  eprint = {2310.19102},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.19102}
}

References

58 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdelkhalik, H., Arafa, Y., Santhi, N., and Badawy, A.-H. Demystifying the nvidia ampere architecture through microbenchmarking and instruction-level analysis, 2022. find
  2. Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., Tumanov, A., and Ramjee, R. Taming throughput-latency tradeoff in llm inference with sarathi-serve, 2024.
  3. Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023.
  4. Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. Piqa: Reasoning about physical commonsense in natural language, 2019.
  5. Chee, J., Cai, Y., Kuleshov, V., and Sa, C. D. Quip: 2-bit quantization of large language models with guarantees, 2023. find
  6. Chen, L. Dissecting batching effects in gpt inference, May 2023. URL https://le.qun.ch/en/blog/2023/05/13/transformer-batching/. ↗
  7. Chen, L., Ye, Z., Wu, Y., Zhuo, D., Ceze, L., and Krishnamurthy, A. Punica: Multi-tenant lora serving, 2023.
  8. Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions, 2019.
  9. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018.
  10. Dai, D., Deng, C., Zhao, C., Xu, R. X., Gao, H., Chen, D., Li, J., Zeng, W., Yu, X., Wu, Y., Xie, Z., Li, Y. K., Huang, P., Luo, F., Ruan, C., Sui, Z., and Liang, W. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models, 2024. find
  11. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022. find
  12. Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Llm.int8(): 8-bit matrix multiplication for transformers at scale, 2022. find
  13. Duarte, F. Number of chatgpt users, Jul 2023. URL https://explodingtopics.com/blog/chatgpt-users. ↗
  14. Elimian, G. Chatgpt costs 700,000 to run daily, openai may go bankrupt in 2024, Aug 2023. URL https://technext24.com/2023/08/14/chatgpt-costs-700000-daily-openai. ↗
  15. Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre-trained transformers, 2023. find
  16. Gao, L., Tow, J., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., McDonell, K., Muennighoff, N., Phang, J., Reynolds, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, September 2021. URL https://doi.org/10.5281/zenod… — [doi:10.5281/zenodo.5371628](https://doi.org/10.5281/zenodo.5371628) ↗
  17. Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces, 2023. find
  18. Gu, A., Goel, K., and Ré, C. Efficiently modeling long sequences with structured state spaces, 2022. find
  19. Guo, C., Tang, J., Hu, W., Leng, J., Zhang, C., Yang, F., Liu, Y., Guo, M., and Zhu, Y. OliVe: Accelerating large language models via hardware-friendly outlier-victim pair quantization. In Proceedings of the 50th Annual International Symposium on Computer Architecture. ACM, jun 2023. doi: 10.1145/35… find
  20. Han, S., Mao, H., and Dally, W. J. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, 2016. find
  21. HuggingFace. Sharegpt vicuna unfiltered, May 2023. URL https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered. ↗
  22. Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D. Quantization and training of neural networks for efficient integer-arithmetic-only inference, 2017. find
  23. Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., de las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L., Lachaux, M.-A., Stock, P., Subramanian, S., Yang, S., Antoniak, S., Scao, T. L., Gervet, T., La… find
  24. Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K. Squeezellm: Dense-and-sparse quantization, 2023.
  25. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention, 2023.
  26. Lee, C., Jin, J., Kim, T., Kim, H., and Park, E. Owq: Lessons learned from activation outliers for weight quantization in large language models. ArXiv, abs/2306.02272, 2023. URL https://api.semanticscholar.org/CorpusID:259076427. ↗
  27. Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., Gan, C., and Han, S. Awq: Activation-aware weight quantization for llm compression and acceleration, 2023. find
  28. Liu, J., Gong, R., Wei, X., Dong, Z., Cai, J., and Zhuang, B. Qllm: Accurate and efficient low-bitwidth quantization for large language models, 2023a. find
  29. Liu, S., Liu, Z., Huang, X., Dong, P., and Cheng, K. Llm-fp4: 4-bit floating-point quantized transformers, 2023b. find
  30. Marcus, M., Kim, G., Marcinkiewicz, M. A., MacIntyre, R., Bies, A., Ferguson, M., Katz, K., and Schasberger, B. The penn treebank: Annotating predicate argument structure. In Proceedings of the Workshop on Human Language Technology, HLT ’94, pp. 114–119, USA, 1994. Association for Computational Ling…
  31. Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models, 2016.
  32. Micikevicius, P., Stosic, D., Burgess, N., Cornea, M., Dubey, P., Grisenthwaite, R., Ha, S., Heinecke, A., Judd, P., Kamalu, J., Mellempudi, N., Oberman, S., Shoeybi, M., Siu, M., and Wu, H. Fp8 formats for deep learning, 2022. find
  33. Nagel, M., Fournarakis, M., Amjad, R. A., Bondarenko, Y., van Baalen, M., and Blankevoort, T. A white paper on neural network quantization, 2021. find
  34. NVIDIA. Nvidia a100 specifications, a. URL https://www.nvidia.com/en-us/data-center/a100/. NVIDIA. Nvidia tensor core, b. URL https://www.nvidia.com/en-us/data-center/tensor-cores/. NVIDIA. Nvidia blackwell platform arrives to power a new era of computing, March 2024a. URL https://nvidianews.nvidi… ↗
  35. Patel, P., Choukse, E., Zhang, C., Íñigo Goiri, Shah, A., Maleki, S., and Bianchini, R. Splitwise: Efficient generative llm inference using phase splitting, 2023.
  36. Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Levskaya, A., Heek, J., Xiao, K., Agrawal, S., and Dean, J. Efficiently scaling transformer inference. ArXiv, abs/2211.05102, 2022. URL https://api.semanticscholar.org/CorpusID:253420623. ↗
  37. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(1), jan 2020. ISSN 1532-4435.
  38. Rouhani, B. D., Zhao, R., More, A., Hall, M., Khodamoradi, A., Deng, S., Choudhary, D., Cornea, M., Dellinger, E., Denolf, K., Dusan, S., Elango, V., Golub, M., Heinecke, A., James-Roxby, P., Jani, D., Kolhe, G., Langhammer, M., Li, A., Melnick, L., Mesmakhosroshahi, M., Rodriguez, A., Schulte, M.,… — Microscaling Data Formats for Deep Learning
  39. Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale, 2019. find
  40. Shao, W., Chen, M., Zhang, Z., Xu, P., Zhao, L., Li, Z., Zhang, K., Gao, P., Qiao, Y., and Luo, P. Omniquant: Omnidirectionally calibrated quantization for large language models, 2023.
  41. Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Fu, D. Y., Xie, Z., Chen, B., Barrett, C. W., Gonzalez, J., Liang, P., Ré, C., Stoica, I., and Zhang, C. High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning, 2023. URL… find
  42. Thakkar, V., Ramani, P., Cecka, C., Shivam, A., Lu, H., Yan, E., Kosaian, J., Hoemmen, M., Wu, H., Kerr, A., Nicely, M., Merrill, D., Blasig, D., Qiao, F., Majcher, P., Springer, P., Hohnerbach, M., Wang, J., and Gupta, M. CUTLASS, January 2023. URL https://github.com/NVIDIA/cutlass. ↗
  43. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hossei… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  44. Trauzettel-Klosinski, S., Dietz, K., and the IReST Study Group. Standardized Assessment of Reading Performance: The New International Reading Speed Texts IReST. Investigative Ophthalmology & Visual Science, 53(9):5452–5461, 08 2012. ISSN 1552-5783. doi: 10.1167/iovs.11-8284. URL https://doi.org/10.1… — [doi:10.1167/iovs.11-8284](https://doi.org/10.1167/iovs.11-8284) find
  45. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need, 2023. find
  46. Wei, X., Zhang, Y., Li, Y., Zhang, X., Gong, R., Guo, J., and Liu, X. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling, 2023. find
  47. Wikipedia contributors. List of qualcomm snapdragon systems on chips — Wikipedia, the free encyclopedia, 2023. URL https://en.wikipedia.org/w/index.php?title=List_of_Qualcomm_Snapdragon_systems_on_chips&oldid=1182026635. [Online; accessed 26-October-2023]. ↗
  48. Williams, S., Waterman, A., and Patterson, D. Roofline: an insightful visual performance model for multicore architectures. Communications of the ACM, 52(4):65–76, 2009.
  49. Wu, X., Yao, Z., and He, Y. Zeroquant-fp: A leap forward in llms post-training w4a8 quantization using floating-point formats, 2023. find
  50. Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models, 2023. find
  51. Yao, Z., Aminabadi, R. Y., Zhang, M., Wu, X., Li, C., and He, Y. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers, 2022.
  52. Ye, Z., Chen, L., Lai, R., Zhao, Y., Zheng, S., Shao, J., Hou, B., Jin, H., Zuo, Y., Yin, L., Chen, T., and Ceze, L. Accelerating self-attentions for llm serving with flashinfer, February 2024. URL https://flashinfer.ai/2024/02/02/introduce-flashinfer.html. ↗
  53. Yu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S., and Chun, B.-G. Orca: A distributed serving system for Transformer-Based generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp. 521–538, Carlsbad, CA, July 2022. USENIX Association. ISBN 978-1-939133-28-…
  54. Yuan, Z., Niu, L., Liu, J., Liu, W., Wang, X., Shang, Y., Sun, G., Wu, Q., Wu, J., and Wu, B. Rptq: Reorder-based post-training quantization for large language models, 2023. find
  55. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence?, 2019. find
  56. Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., Wang, Z., and Chen, B. H 2 o: Heavy-hitter oracle for efficient generative inference of large language models, 2023. — H$_2$O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
  57. Zhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X., and Zhang, H. Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving, 2024.
  58. Zhou, C., Richard, V., Savarese, P., Hassman, Z., Maire, M., DiBrino, M., and Li, Y. Sysmol: A hardware-software co-design framework for ultra-low and fine-grained mixed-precision neural networks, 2023. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhao-2023-atom-low-bit-quantization-efficient/