Elamur Research⌕
☰
home / papers / 2024 / Extreme Compression of Large Language Models via Additive Quantization

Extreme Compression of Large Language Models via Additive Quantization

Paper · 2024 · Vage Egiazarian, Andrei Panferov, Denis Kuznedelev et al.

Summary

The paper targets "extreme" compression of open LLMs, meaning roughly 2–3 bits per weight. At that level, earlier quantization schemes usually lose so much accuracy that a smaller model at 3–4 bits is the better deal. The authors borrow Multi-Codebook Quantization from approximate nearest neighbor search, specifically Additive Quantization (AQ). In AQ, groups of weights are encoded together as a sum of codewords drawn from several learned codebooks.

Their method, AQLM, changes AQ in two ways:

  • Output-aware objective: instead of reconstructing the weights themselves, it minimizes the error in each layer's outputs on calibration inputs.
  • Block-level tuning: codebooks and other parameters are refined jointly across a whole Transformer block, using only calibration data.

AQLM is evaluated on LLAMA 2 (7B, 13B, 70B) and Mixtral. It beats prior methods such as QuIP# across the 2–4 bit range, with the biggest gains at 2 bits. It is the first approach to be Pareto optimal in accuracy versus size below 3 bits. The authors also release GPU and CPU kernels that match or exceed FP16 speed while using far less memory. This matters for running large models on consumer hardware.

Key points

  • AQLM is evaluated on LLAMA 2 7B/13B/70B and Mixtral, with ablations over code width and number of codebooks.
  • Using one 16-bit codebook gives about 2.07 bits per parameter for LLAMA 2 70B, 2.19 for 13B and 2.29 for 7B. A 2×8-bit codebook configuration reaches a WikiText-2 perplexity of 6.57.
  • Memory footprint shrinks by up to 8x.
  • Kernel speedups on a single layer:
    • On an RTX 3090 GPU, layer matrix-vector products run about 1.2–1.3x faster than FP16, and up to about 3x with 2×8-bit codebooks on the 70B layer.
    • On an 8-core Intel i9 CPU, speedups reach up to about 4x over FP32.
  • End-to-end generation with HuggingFace integration reaches roughly 14 tokens/s on LLAMA 2 70B.
  • Code is released at github.com/Vahe1994/AQLM. Plugging in better fine-tuning methods from later work further improves the 2- and 3-bit models.

Interesting ideas

  • I1. Reuse retrieval tools for compression. Vector-search quantizers such as PQ, AQ and LSQ were built to compress embedding databases, but they transfer to neural network weights. This suggests surveying older fields for mature solutions to new compression problems.
  • I2. Optimize for what the layer produces. Fitting codes to minimize output error on real calibration activations, rather than raw weight error, focuses accuracy where the data actually goes. The same principle could guide pruning or distillation of task-specific models, for example Ukrainian NLP models.
  • I3. Learned codebooks instead of a fixed lattice. Directly optimizing the codebooks removed the need for the rotation step used by QuIP#. Where data is available, letting it shape the representation often beats hand-designed structures.
  • I4. Tune several small codebooks for hardware. Several 8-bit codebooks fit in cache and run faster, at a small accuracy cost. Treating the codebook layout as a dial between speed and quality is a useful pattern for deployment on modest hardware.
  • I5. Judge compression by the Pareto frontier. The paper compares methods by accuracy at equal model size in bytes, not at equal bit width. This shows whether a heavily compressed large model actually beats a lightly compressed smaller one, which is a fairer way to evaluate any compression method.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko and Dan Alistarh. 2024. Extreme Compression of Large Language Models via Additive Quantization. ICML 2024.

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 2401.06118

@inproceedings{egiazarian2024extreme,
  title = {{Extreme Compression of Large Language Models via Additive Quantization}},
  author = {Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko and Dan Alistarh},
  booktitle = {ICML 2024},
  year = {2024},
  eprint = {2401.06118},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.06118}
}

References

59 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Babenko, A. and Lempitsky, V. Additive quantization for extreme vector compression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 931–938, 2014. find
  2. Besag, J. On the statistical analysis of dirty pictures. Journal of the Royal Statistical Society Series B: Statistical Methodology, 48(3):259–279, 1986. find
  3. Biderman, S., Schoelkopf, H., Anthony, Q., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint arXiv:2304.01373, 2023. find
  4. Blalock, D. and Guttag, J. Multiplying matrices without multiplying. In International Conference on Machine Learning, pp. 992–1004. PMLR, 2021. find
  5. Burton, D., Shore, J., and Buck, J. A generalization of isolated word recognition using vector quantization. In https://github.com/Vahe1994/AQLM/https://www.reddit.com/r/LocalLLaMA/7 https://github.com/bigscience-workshop/petals/ICASSP ’83. IEEE International Conference on Acoustics, Speech, and Sig… — [doi:10.1109/icassp.1983.1171915](https://doi.org/10.1109/icassp.1983.1171915) ↗
  6. Chee, J., Cai, Y., Kuleshov, V., and Sa, C. D. Quip: 2-bit quantization of large language models with guarantees, 2023. find
  7. Chen, S., Wang, W., and Pan, S. J. Deep neural network quantization via layer-wise optimization using limited training data. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):3329–3336, Jul. 2019. doi: 10.1609/aaai.v33i01.33013329. URL https://ojs.aaai.org/index.php/AAAI/article/… — [doi:10.1609/aaai.v33i01.33013329](https://doi.org/10.1609/aaai.v33i01.33013329) ↗
  8. Chen, Y., Guan, T., and Wang, C. Approximate nearest neighbor search by residual vector quantization. Sensors, 10(12):11259–11273, 2010. find
  9. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.
  10. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. CoRR, abs/2110.14168, 2021. URL https://arxiv.org/abs/2110.14168. ↗
  11. Computer, T. Redpajama: an open dataset for training large language models, 2023. URL https://github.com/togethercomputer/RedPajama-Data. ↗
  12. Dettmers, T. and Zettlemoyer, L. The case for 4-bit precision: k-bit inference scaling laws. arXiv preprint arXiv:2212.09720, 2022.
  13. Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. LLM.int8(): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, 2022. find
  14. Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. QLoRA: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314, 2023a.
  15. Dettmers, T., Svirschevski, R., Egiazarian, V., Kuznedelev, D., Frantar, E., Ashkboos, S., Borzunov, A., Hoefler, T., and Alistarh, D. Spqr: A sparse-quantized representation for near-lossless llm weight compression. arXiv preprint arXiv:2306.03078, 2023b. find
  16. Fernández-Marqués, J., AbouElhamayed, A. F., Lane, N. D., and Abdelfattah, M. S. Are we there yet? product quantization and its hardware acceleration. ArXiv, abs/2305.18334, 2023. URL https://api.semanticscholar.org/CorpusID: 258967539. — [arXiv:2305.18334](https://arxiv.org/abs/2305.18334) ↗
  17. Frantar, E. and Alistarh, D. Qmoe: Practical sub-1-bit compression of trillion-parameter models. arXiv preprint arXiv:2310.16795, 2023. find
  18. Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022a. find
  19. Frantar, E., Singh, S. P., and Alistarh, D. Optimal Brain Compression: A framework for accurate post-training quantization and pruning. arXiv preprint arXiv:2208.11580, 2022b. Accepted to NeurIPS 2022, to appear. — [arXiv:2208.11580](https://arxiv.org/abs/2208.11580) find
  20. Gao, L., Tow, J., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., McDonell, K., Muennighoff, N., Phang, J., Reynolds, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, September 2021. URL https://doi.org/10.5281/zenodo… — [doi:10.5281/zenodo.5371628](https://doi.org/10.5281/zenodo.5371628) ↗
  21. Ge, T., He, K., Ke, Q., and Sun, J. Optimized product quantization. IEEE transactions on pattern analysis and machine intelligence, 36(4):744–755, 2013. find
  22. Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., and Keutzer, K. A survey of quantization methods for efficient neural network inference. arXiv preprint arXiv:2103.13630, 2021. — [arXiv:2103.13630](https://arxiv.org/abs/2103.13630) find
  23. Gray, R. Vector quantization. IEEE ASSP Magazine, 1(2): 4–29, 1984. doi: 10.1109/MASSP.1984.1162229. — [doi:10.1109/massp.1984.1162229](https://doi.org/10.1109/massp.1984.1162229) find
  24. Guo, R., Kumar, S., Choromanski, K., and Simcha, D. Quantization based fast inner product search. In Artificial intelligence and statistics, pp. 482–490. PMLR, 2016. find
  25. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. CoRR, abs/2009.03300, 2020. URL https://arxiv.org/abs/2009.03300. ↗
  26. Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network, 2015.
  27. Jegou, H., Douze, M., and Schmid, C. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence, 33(1):117–128, 2010. find
  28. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. find
  29. Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. find
  30. Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K. Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07629, 2023.
  31. Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. International Conference on Learning Representations (ICLR), 2015.
  32. Li, Z., Ni, B., Zhang, W., Yang, X., and Gao, W. Performance guaranteed network acceleration via high-order residual quantization, 2017. find
  33. Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., and Han, S. Awq: Activation-aware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978, 2023. find
  34. Malinovskii, V., Mazur, D., Ilin, I., Kuznedelev, D., Burlachenko, K., Yi, K., Alistarh, D., and Richtarik, P. Pv-tuning: Beyond straight-through estimation for extreme llm compression. arXiv preprint arXiv:2405.14852, 2024. — [arXiv:2405.14852](https://arxiv.org/abs/2405.14852) find
  35. Martinez, J., Clement, J., Hoos, H. H., and Little, J. J. Revisiting additive quantization. In Computer Vision– ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pp. 137–153. find
  36. Martinez, J., Zakhmi, S., Hoos, H. H., and Little, J. J. Lsq++: Lower running time and higher recall in multi-codebook quantization. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 491–506, 2018. find
  37. McCarter, C. and Dronen, N. Look-ups are not (yet) all you need for deep learning inference. ArXiv, abs/2207.05808, 2022. URL https://api.semanticscholar.org/CorpusID: 250491319. — [arXiv:2207.05808](https://arxiv.org/abs/2207.05808) ↗
  38. Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016.
  39. Nagel, M., Amjad, R. A., Van Baalen, M., Louizos, C., and Blankevoort, T. Up or down? Adaptive rounding for post-training quantization. In International Conference on Machine Learning (ICML), 2020. find
  40. Norouzi, M. and Fleet, D. J. Cartesian k-means. In Proceedings of the IEEE Conference on computer Vision and Pattern Recognition, pp. 3017–3024, 2013. find
  41. Ozan, E. C., Kiranyaz, S., and Gabbouj, M. Competitive quantization for approximate nearest neighbor search. IEEE Transactions on Knowledge and Data Engineering, 28(11):2884–2894, 2016. doi: 10.1109/TKDE.2016.2597834. — [doi:10.1109/tkde.2016.2597834](https://doi.org/10.1109/tkde.2016.2597834) find
  42. Park, G., Park, B., Kwon, S. J., Kim, B., Lee, Y., and Lee, D. nuQmm: Quantized matmul for efficient inference of large-scale generative language models. arXiv preprint arXiv:2206.09557, 2022.
  43. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. PyTorch: An imperative style, high-perfor… — PyTorch: An Imperative Style, High-Performance Deep Learning Library
  44. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21 (140):1–67, 2020.
  45. Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: an adversarial winograd schema challenge at scale. Commun. ACM, 64(9):99–106, 2021. doi: 10.1145/3474381. URL https://doi.org/10.1145/3474381. ↗
  46. Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon, F., Gallé, M., et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022. find
  47. Shazeer, N. Glu variants improve transformer, 2020. find
  48. Tata, S. and Patel, J. M. PiQA: An algebra for querying protein data sets. In International Conference on Scientific and Statistical Database Management, 2003. find
  49. TII UAE. The Falcon family of large language models. https://huggingface.co/tiiuae/falcon-40b, May 2023. ↗
  50. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
  51. Tseng, A., Chee, J., Sun, Q., Kuleshov, V., and Sa, C. D. Quip\#: Even better llm quantization with hadamard incoherence and lattice codebooks, 2024. — QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks
  52. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. arXiv preprint arXiv:1706.03762, 2017.
  53. Xiao, G., Lin, J., Seznec, M., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. arXiv preprint arXiv:2211.10438, 2022. find
  54. Yao, Z., Aminabadi, R. Y., Zhang, M., Wu, X., Li, C., and He, Y. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. arXiv preprint arXiv:2206.01861, 2022.
  55. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? In Korhonen, A., Traum, D. R., and Màrquez, L. (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August… find
  56. Zhang, B. and Sennrich, R. Root mean square layer normalization. CoRR, abs/1910.07467, 2019. URL http://arxiv.org/abs/1910.07467. ↗
  57. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
  58. Zhang, T., Du, C., and Wang, J. Composite quantization for approximate nearest neighbor search. In International Conference on Machine Learning, pp. 838–846. PMLR, 2014. find
  59. Zhou, S.-C., Wang, Y.-Z., Wen, H., He, Q.-Y., and Zou, Y.-H. Balanced quantization: An effective and efficient approach to quantized neural networks. Journal of Computer Science and Technology, 32(4):667–682, Jul 2017. ISSN 1860-4749. doi: 10.1007/s11390-017-1750-y. URL https://doi.org/10.1007/s1139… — [doi:10.1007/s11390-017-1750-y](https://doi.org/10.1007/s11390-017-1750-y) ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/egiazarian-2024-extreme-compression-large-language-models/