Elamur Research
☰
home / papers / 2024 / QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks

QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2402.04396

Summary

Large language models are costly to deploy because their weights occupy a lot of memory. Earlier post-training quantization (PTQ) methods either lose too much quality at very low bitrates or rely on decoding that slows inference. QuIP# is a weight-only PTQ method aimed at 2–4 bits per weight, and it combines three components:

  • Outlier suppression: it applies a randomized Hadamard transform, which is cheaper than the Kronecker-based scheme in QuIP and has stronger theoretical guarantees.
  • Vector quantization: the transformed weights are close to Gaussian, so the method groups them and rounds them with BlockLDLQ using E8P. E8P is a compact codebook built on the E8 lattice, which is the densest packing of spheres in eight dimensions.
  • Fine-tuning: a tuning step across layers recovers additional accuracy.

On Llama models, QuIP# beats OmniQuant, QuIP and AQLM. With QuIP#, 3-bit models scale better than an idealized lossless 4-bit model, a result not seen before. The design also keeps inference fast on consumer GPUs. Taken together, this suggests that very low-bit models can be both accurate and practical to run.

Key points

  • On Llama 2 70B at 2 bits, Wikitext2 perplexity (context 4096) reaches 3.91. For comparison, FP16 scores 3.12, AQLM at about 2.07 bits scores 3.94, and the original QuIP scores 5.90.
  • The ablation on Llama 2 7B at 2 bits shows that each component matters. Perplexity worsens from 6.19 to 8.22 without fine-tuning and to 11.2 when lattice codebooks are also removed.
  • The E8P codebook covers 16 bits per 8-dimensional vector yet takes only about 1 KiB, so it fits in the GPU's L1 cache. Decoding takes fewer than 5 instructions per weight.
  • On an RTX 4090 using the FlashAttention Llama code, a 2-bit Llama 2 7B generates about 170 tok/s, and Llama 2 70B uses roughly 57% of peak memory bandwidth.
  • With the HuggingFace Llama code, 2-bit QuIP# runs Llama 2 7B at about 106 tok/s, versus 33 for FP16 and 21 for AQLM. The code is available in the Cornell-RelaxML/quip-sharp repository.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning · Computing & GPUs

Cited by

115 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 113 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.04396 · arXiv 2402.04396

@misc{tseng2024quip,
  title = {{QuIP\#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks}},
  author = {Tseng, Albert and Chee, Jerry and Sun, Qingyao and Kuleshov, Volodymyr and Sa, Christopher De},
  year = {2024},
  doi = {10.48550/arxiv.2402.04396},
  eprint = {2402.04396},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.04396}
}

References

35 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Almazrouei, E., Alobeidli, H., Alshamsi, A., Cappelli, A., Cojocaru, R., Debbah, M., Étienne Goffinet, Hesslow, D., Launay, J., Malartic, Q., Mazzotta, D., Noune, B., Pannier, B., and Penedo, G. The falcon series of open language models, 2023. find our card
  2. Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023. URL https:// openreview.net/forum?id=tcbBPnfwxS. find
  3. Chee, J., Renz, M., Damle, A., and Sa, C. D. Model preserving compression for neural networks. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum? id=gt-l9Hu2ndd. ↗
  4. Chee, J., Cai, Y., Kuleshov, V., and Sa, C. D. QuIP: 2-bit quantization of large language models with guarantees. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview. net/forum?id=xrk9g5vcXR. find
  5. Cochran, W., Cooley, J., Favin, D., Helms, H., Kaenel, R., Lang, W., Maling, G., Nelson, D., Rader, C., and Welch, P. What is the fast fourier transform? Proceedings of the IEEE, 55(10):1664–1674, 1967. doi: 10.1109/PROC. 1967.5957. — [doi:10.1109/proc](https://doi.org/10.1109/proc) find
  6. Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac’h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework… find
  7. Halko, N., Martinsson, P.-G., and Tropp, J. A. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM review, 53(2):217–288, 2011. find
  8. Hedayat, A. and Wallis, W. D. Hadamard Matrices and Their Applications. The Annals of Statistics, 6(6):1184 – 1238, 1978. doi: 10.1214/aos/1176344370. URL https: //doi.org/10.1214/aos/1176344370. — [doi:10.1214/aos/1176344370](https://doi.org/10.1214/aos/1176344370) find
  9. Computer, T. Redpajama: An open source recipe to reproduce llama training dataset, 2023. URL https://github.com/togethercomputer/ RedPajama-Data. ↗
  10. Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., de las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L., Lachaux, M.-A., Stock, P., Subramanian, S., Yang, S., Antoniak, S., Scao, T. L., Gervet, T., La… find
  11. Dao, T. FlashAttention-2: Faster attention with better parallelism and work partitioning. 2023. find
  12. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems, 2022. find
  13. Dettmers, T. and Zettlemoyer, L. The case for 4-bit precision: k-bit inference scaling laws. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning… find
  14. Juang, B.-H. and Gray, A. Multiple stage vector quantization for speech coding. In ICASSP ’82. IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 7, pp. 597–600, 1982. doi: 10.1109/ICASSP. 1982.1171604. — [doi:10.1109/icassp](https://doi.org/10.1109/icassp) find
  15. Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M., and Keutzer, K. Squeezellm: Dense-and-sparse quantization. arXiv, 2023. find our card
  16. Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization, 2017. find
  17. Dettmers, T., Svirschevski, R., Egiazarian, V., Kuznedelev, D., Frantar, E., Ashkboos, S., Borzunov, A., Hoefler, T., and Alistarh, D. Spqr: A sparse-quantized representation for near-lossless llm weight compression, 2023. find
  18. Kostina, V. and Verdú, S. Fixed-length lossy compression in the finite blocklength regime: Gaussian source. 2011 IEEE Information Theory Workshop, ITW 2011, 10 2011. doi: 10.1109/ITW.2011.6089501. — [doi:10.1109/itw.2011.6089501](https://doi.org/10.1109/itw.2011.6089501) find
  19. Egiazarian, V., Panferov, A., Kuznedelev, D., Frantar, E., Babenko, A., and Alistarh, D. Extreme compression of large language models via additive quantization, 2024. find
  20. Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., Gan, C., and Han, S. Awq: Activation-aware weight quantization for llm compression and acceleration, 2023. find
  21. Fino and Algazi. Unified matrix treatment of the fast walsh-hadamard transform. IEEE Transactions on Computers, C-25(11):1142–1146, 1976. doi: 10.1109/TC.1976. 1674569. — [doi:10.1109/tc.1976](https://doi.org/10.1109/tc.1976) find
  22. Liu, Z., Oguz, B., Zhao, C., Chang, E., Stock, P., Mehdad, Y., Shi, Y., Krishnamoorthi, R., and Chandra, V. Llm-qat: Data-free quantization aware training for large language models, 2023. find our card
  23. Lloyd, S. Least squares quantization in pcm. IEEE Transactions on Information Theory, 28(2):129–137, 1982. doi: 10.1109/TIT.1982.1056489. — [doi:10.1109/tit.1982.1056489](https://doi.org/10.1109/tit.1982.1056489) find
  24. Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation language models, 2023a. find
  25. Nagel, M., Amjad, R. A., Van Baalen, M., Louizos, C., and Blankevoort, T. Up or down? Adaptive rounding for post-training quantization. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp… find
  26. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hossei… find
  27. Nagel, M., Fournarakis, M., Bondarenko, Y., and Blankevoort, T. Overcoming oscillations in quantization-aware training. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceed… find
  28. Viazovska, M. The sphere packing problem in dimension 8. Annals of Mathematics, 185(3), May 2017. ISSN 0003-486X. doi: 10.4007/annals.2017.185.3. 7. URL http://dx.doi.org/10.4007/annals. 2017.185.3.7. — [doi:10.4007/annals.2017.185.3](https://doi.org/10.4007/annals.2017.185.3) ↗
  29. Nguyen, E., Poli, M., Faizi, M., Thomas, A., Birch-Sykes, C., Wornow, M., Patel, A., Rabideau, C., Massaroli, S., Bengio, Y., Ermon, S., Baccus, S. A., and Ré, C. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. 2023. find
  30. Xi, H., Li, C., Chen, J., and Zhu, J. Training transformers with 4-bit integers. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: //openreview.net/forum?id=H9hWlfMT6O. find
  31. Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., Rapin, J., Kozhevnikov, A., Evtimov, I., Bitton, J., Bhatt, M., Ferrer, C. C., Grattafiori, A., Xiong, W., Défossez, A., Copet, J., Azhar, F., Touvron, H., Martin, L., Usunier, N.,… find our card
  32. Shao, W., Chen, M., Zhang, Z., Xu, P., Zhao, L., Li, Z., Zhang, K., Gao, P., Qiao, Y., and Luo, P. Omniquant: Omnidirectionally calibrated quantization for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https:// openreview.net/forum?id=8Wuvhh0LY… find our card
  33. Sloane, N. Hadamard Matrices — neilsloane.com. http: //neilsloane.com/hadamard/. [Accessed 02-02-2024]. find
  34. Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models. In Workshop on Efficient Systems for Foundation Models @ ICML2023, 2023. URL https://openreview. net/forum?id=tz9JV2PRSv. find
  35. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., A. Concentration Inequalities for the Randomized Hadamard Transform and Fast Fourier Transform find