Elamur Research⌕
☰
home / papers / 2024 / QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving

QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving

Paper · 2024 · Yujun Lin, Haotian Tang, Shang Yang et al.

Summary

Low-bit integer quantization promises faster LLM inference, but INT4 approaches have mostly helped small-batch edge scenarios. In large-batch cloud serving they often lose to W8A8 or W4A16. The authors trace this to dequantization work running on slow CUDA cores, which costs 20–90% extra runtime on GPUs. Their answer is a co-designed algorithm and system. QoQ is a W4A8KV4 scheme: 4-bit weights, 8-bit activations and a 4-bit KV cache. QServe is the inference library built to run it.

QoQ uses two-stage ("progressive") weight quantization so that every GEMM runs on INT8 tensor cores. It also adds SmoothAttention, which limits accuracy loss when keys are stored in 4 bits. On the system side, QServe reorders weights and exploits register-level parallelism to make dequantization cheap, and it keeps KV4 attention memory-bound.

Across seven LLMs on A100 and L40S, QServe beats TensorRT-LLM, Atom and QuaRot in maximum throughput. A cheaper L40S running QServe often outperforms an A100 running TensorRT-LLM, which cuts serving cost by about 3×.

Key points

  • Measured throughput gains over TensorRT-LLM:
    • Llama-3-8B: 1.2× on A100 and 1.4× on L40S.
    • Qwen1.5-72B: 2.4× on A100 and 3.5× on L40S.
  • Gains over W4A4 systems: on A100, QServe delivers 2.5–2.9× the throughput of Atom and QuaRot. The paper notes that Atom itself runs 20–25% slower than TensorRT-LLM's W4A16/W8A8 on Llama-2-7B.
  • Progressive group quantization:
    • Weights are first mapped to INT8 with per-channel FP16 scales, using a protective range of [−119, 119].
    • These INT8 values are then group-quantized to INT4.
    • At runtime, INT4→INT8 dequantization multiplies first and subtracts afterward.
  • Roofline analysis on A100:
    • W4A16 is theoretically faster when the batch size is below about 78; W8A8 wins above that.
    • W4A8 dominates both across batch sizes.
    • One CUDA-core operation costs roughly as much as 50 INT4 tensor-core operations.
  • Release and reproducibility:
    • Code is in the mit-han-lab/omniserve repository under Apache 2.0, with a Docker artifact.
    • Accuracy results can be reproduced with the deepcompressor library.

Interesting ideas

  • I1. Profile the slow units, not the peak numbers. The paper shows that lower nominal precision can lose to higher precision once you count overhead on slower hardware units. Judging any accelerated pipeline by where its bottleneck sits, rather than by headline TOPS, is a habit that transfers well beyond LLM serving.
  • I2. Quantize in stages with headroom. Weights pass through an 8-bit intermediate whose range is deliberately clipped so the later 4-bit step cannot overflow. That spare range is what makes cheap, parallel unpacking possible, so the same trick could help any custom compact encoding.
  • I3. Move the difficulty to the side you keep precise. SmoothAttention rescales keys and queries so that outliers end up in the queries, which are never quantized. This rebalancing applies to other paired computations where only one side must be compressed.
  • I4. Use roofline reasoning to choose formats. Plotting compute intensity against hardware limits makes it clear which precision suits which batch-size regime. The same approach could guide deployment choices for Ukrainian NLP models on modest GPUs.
  • I5. Optimize the cost-per-token metric. The authors frame success as getting more throughput from a cheaper GPU than from a premium one. That makes a practical evaluation target for budget-limited projects.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 worksHardware and Architecture · 167 worksEngineering · 188 worksElectrical and Electronic Engineering · 33 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksEfficient models & compression · 297 works

Cite

Cite as Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan and Song Han. 2024. QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving. MLSys 2025.

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 2405.04532

@inproceedings{lin2024qserve,
  title = {{QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving}},
  author = {Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan and Song Han},
  booktitle = {MLSys 2025},
  year = {2024},
  eprint = {2405.04532},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.04532}
}

References

44 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245, 2023.
  2. Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J. Quarot: Outlier-free 4-bit inference in rotated llms. arXiv preprint arXiv:2404.00456, 2024.
  3. Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
  4. Chee, J., Cai, Y., Kuleshov, V., and Sa, C. D. Quip: 2-bit quantization of large language models with guarantees, 2024. find
  5. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018.
  6. Contributors, L. Lightllm: A light and fast inference service for llm. https://github.com/ModelTC/lightllm, 2023a. ↗
  7. Contributors, L. Lmdeploy: A toolkit for compressing, deploying, and serving llm. https://github.com/InternLM/lmdeploy, 2023b. ↗
  8. Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. GPT3.int8(): 8-bit matrix multiplication for transformers at scale. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. find
  9. Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314, 2023a.
  10. Dettmers, T., Svirschevski, R., Egiazarian, V., Kuznedelev, D., Frantar, E., Ashkboos, S., Borzunov, A., Hoefler, T., and Alistarh, D. Spqr: A sparse-quantized representation for near-lossless llm weight compression, 2023b. find
  11. Fang, C., Zhou, A., and Wang, Z. An algorithm–hardware co-optimized framework for accelerating n: M sparse transformers. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 30(11):1573–1586, 2022. find
  12. Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. GPTQ: Accurate post-training compression for generative pretrained transformers. arXiv preprint arXiv:2210.17323, 2022. find
  13. Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac’h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework… find
  14. Ham, T. J., Jung, S. J., Kim, S., Oh, Y. H., Park, Y., Song, Y., Park, J.-H., Lee, S., Park, K., Lee, J. W., et al. Aˆ 3: Accelerating attention mechanisms in neural networks with approximation. In 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA), pp. 328–341. find
  15. Ham, T. J., Lee, Y., Seo, S. H., Kim, S., Choi, H., Jung, S. J., and Lee, J. W. Elsa: Hardware-software co-design for efficient, lightweight self-attention mechanism in neural networks. In 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA), pp. 692– 705. find
  16. Hong, S., Moon, S., Kim, J., Lee, S., Kim, M., Lee, D., and Kim, J.-Y. Dfx: A low-latency multi-fpga appliance for accelerating transformer-based text generation. In 2022 55th IEEE/ACM International Symposium on Microarchitecture (MICRO), pp. 616–630. find
  17. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. find
  18. Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. find
  19. Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K. Squeezellm: Dense-and-sparse quantization, 2024.
  20. Kim, Y. J., Henry, R., Fahim, R., and Awadalla, H. H. Who says elephants can’t run: Bringing large scale moe models into cloud scale production. arXiv preprint arXiv:2211.10017, 2022. find
  21. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611–626, 2023a.
  22. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023b.
  23. Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. Awq: Activation-aware weight quantization for llm compression and acceleration. In MLSys, 2024. find
  24. Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models, 2016.
  25. NVIDIA. TensorRT-LLM: A TensorRT Toolbox for Optimized Large Language Model Inference, 2023. URL https://github.com/NVIDIA/TensorRT-LLM. ↗
  26. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. Pytorch: An imperative style, high-perfo… — PyTorch: An Imperative Style, High-Performance Deep Learning Library
  27. Qu, Z., Liu, L., Tu, F., Chen, Z., Ding, Y., and Xie, Y. Dota: detect and omit weak attentions for scalable transformer acceleration. In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 14– 26, 2022. find
  28. Rouhani, B. D., Zhao, R., More, A., Hall, M., Khodamoradi, A., Deng, S., Choudhary, D., Cornea, M., Dellinger, E., Denolf, K., et al. Microscaling data formats for deep learning. arXiv preprint arXiv:2310.10537, 2023.
  29. Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint arXiv:1907.10641, 2019. find
  30. Shao, W., Chen, M., Zhang, Z., Xu, P., Zhao, L., Li, Z., Zhang, K. Z., Gao, P., Qiao, Y., and Luo, P. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308.13137, 2023.
  31. Tambe, T., Hooper, C., Pentecost, L., Jia, T., Yang, E.-Y., Donato, M., Sanh, V., Whatmough, P., Rush, A. M., Brooks, D., et al. Edgebert: Sentence-level energy optimizations for latency-aware multi-task nlp inference. In MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture, p… find
  32. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation language models, 2023a.
  33. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.
  34. Wang, H., Zhang, Z., and Han, S. Spatten: Efficient sparse attention architecture with cascade token and head pruning. In 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pp. 97–110.
  35. Wei, X., Zhang, Y., Zhang, X., Gong, R., Zhang, S., Zhang, Q., Yu, F., and Liu, X. Outlier suppression: Pushing the limit of low-bit transformer language models. arXiv preprint arXiv:2209.13325, 2022. — [arXiv:2209.13325](https://arxiv.org/abs/2209.13325) find
  36. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T. L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. M. Huggingface’s transformers: State-of-th… — Transformers: State-of-the-Art Natural Language Processing
  37. Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. SmoothQuant: Accurate and efficient post-training quantization for large language models. In Proceedings of the 40th International Conference on Machine Learning, 2023. find
  38. Young, A., Chen, B., Li, C., Huang, C., Zhang, G., Zhang, G., Li, H., Zhu, J., Chen, J., Chang, J., Yu, K., Liu, P., Liu, Q., Yue, S., Yang, S., Yang, S., Yu, T., Xie, W., Huang, W., Hu, X., Ren, X., Niu, X., Nie, P., Xu, Y., Liu, Y., Wang, Y., Cai, Y., Gu, Z., Liu, Z., and Dai, Z. Yi: Open foundati… find
  39. Yu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S., and Chun, B.-G. Orca: A distributed serving system for Transformer-Based generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp. 521–538, Carlsbad, CA, July 2022. USENIX Association. ISBN 978-1-939133-28-…
  40. Zadeh, A. H., Edo, I., Awad, O. M., and Moshovos, A. Gobo: Quantizing attention-based nlp models for low latency and energy efficient inference. In 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pp. 811–824. find
  41. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? CoRR, abs/1905.07830, 2019. URL http://arxiv.org/abs/1905.07830. ↗
  42. Zhang, L., Fei, W., Wu, W., He, Y., Lou, Z., and Zhou, H. Dual grained quantization: Efficient fine-grained quantization for llm. arXiv preprint arXiv:2310.04836, 2023. — [arXiv:2310.04836](https://arxiv.org/abs/2310.04836) find
  43. Zhao, Y., Lin, C.-Y., Zhu, K., Ye, Z., Chen, L., Zheng, S., Ceze, L., Krishnamurthy, A., Chen, T., and Kasikci, B. Atom: Low-bit quantization for efficient and accurate llm serving. In MLSys, 2023.
  44. Zheng, L., Yin, L., Xie, Z., Huang, J., Sun, C., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., and Sheng, Y. Efficiently programming large language models using sglang, 2023. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lin-2024-qserve-w4a8kv4-quantization-system-co/