Elamur Research⌕
☰
home / papers / 2023 / OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models

OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models

Paper · 2023 · Changhun Lee, Jungyu Jin, Taesu Kim et al.

Summary

LLMs are expensive to serve. Weight-only quantization below 4 bits reduces this cost, but OPTQ (also called GPTQ) still loses noticeable quality at 3 bits, and smaller models suffer most. The authors observe that the few activation outliers known from int8 work also hurt weight-only quantization. The input channels that carry those outliers make the matching weight columns highly sensitive.

OWQ uses this observation. It ranks weight columns with a Hessian-based score, keeps a handful of these "weak columns" in fp16, and quantizes the remaining dense weights with a tuned OPTQ procedure. With about 3.1 bits per weight, OWQ reaches roughly the quality of 4-bit OPTQ, and the extra latency and memory are small.

The paper also proposes Weak Column Tuning (WCT), a parameter-efficient fine-tuning method. WCT updates only the fp16 weak columns and leaves the quantized part frozen. It matches full-precision LoRA while using a fraction of the memory, and it beats QLoRA in GPT-4-judged comparisons. This lets one compressed model serve both inference and task adaptation. Code is available at github.com/xvyaward/owq.

Key points

  • Perplexity gains over OPTQ: at about 3.01 effective bits, WikiText-2 perplexity improves over 3-bit OPTQ on both tested models.
    • OPT-6.7B: 12.76 → 11.22.
    • LLaMA-7B: 8.08 → 6.65.
  • Bit-width sweep on OPT-6.7B:
    • Even 3.005 bits already beats 3-bit OPTQ, with about 2.9% latency overhead and 0.21% memory overhead.
    • Gains level off as more columns are kept in fp16, while the overhead keeps growing.
  • Comparison with group-wise quantization: adding OPTQ-style grouping on top of OWQ helps little. Plain 3.01-bit OWQ needs only about 10% of the storage overhead of grouped OPTQ (group size 128) and gives similar or better perplexity and zero-shot accuracy.
  • WCT efficiency: WCT matches full-precision LoRA while using 24.4% of the memory at inference.
  • Fine-tuning evaluation: fine-tuned LLaMA-7B variants answered Vicuna Benchmark questions and were judged by GPT-4. WCT on OWQ won against QLoRA and against WCT built on simpler quantizers (minmax RTN and minmax OPTQ).
  • Unquantized parts: embedding and head weights stay in full precision.

Interesting ideas

  • I1. Use activation statistics to protect weights. In weight-only compression, decide which parameters to protect by looking at how strongly their inputs fire, not just at the weights themselves. The same reasoning could guide pruning or low-rank compression of other models.
  • I2. Score the error that matters downstream. The selection metric multiplies Hessian-based sensitivity by the weight perturbation. In the paper's ablation, scoring by weight error alone was clearly worse, which suggests that compression criteria should target the layer's output error.
  • I3. Mix precision at column granularity. Keeping whole columns in fp16 stays hardware-friendly and costs very little. A tiny budget (around 0.01 extra bits) captures most of the gain, which makes this a cheap add-on to almost any uniform quantizer.
  • I4. Fine-tune only the parameters you already kept in high precision. Reusing the protected columns as the trainable set means no separate adapter weights are needed. Because these columns are the most sensitive, a small rank is enough.
  • I5. Judge the frozen base model after adaptation. The paper compares quantizers after fine-tuning rather than before. This shows that the quality of the frozen base still matters once adapters are trained, which is a useful check for any QLoRA-style pipeline.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Pohang University of Science and TechnologyPohang · South Korea2 works · 354 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 1,344 worksEfficient models & compression · 77 works

Cite

Cite as Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim and Eunhyeok Park. 2023. OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models. arXiv:2306.02272.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2306.02272 · arXiv 2306.02272

@misc{lee2023owq,
  title = {{OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models}},
  author = {Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim and Eunhyeok Park},
  year = {2023},
  doi = {10.48550/arxiv.2306.02272},
  eprint = {2306.02272},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.02272}
}

References

36 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Beeching, E.; Fourrier, C.; Habib, N.; Han, S.; Lambert, N.; Rajani, N.; Sanseviero, O.; Tunstall, L.; and Wolf, T. 2023. Open LLM Leaderboard. https://huggingface.co/ spaces/HuggingFaceH4/open llm leaderboard. ↗
  2. Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020. Language models are few-shot learners. NeurIPS, 33: 1877–1901.
  3. Chiang, W.-L.; Li, Z.; Lin, Z.; Sheng, Y.; Wu, Z.; Zhang, H.; Zheng, L.; Zhuang, S.; Zhuang, Y.; Gonzalez, J. E.; Stoica, I.; and Xing, E. P. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. find
  4. Clark, P.; Cowhey, I.; Etzioni, O.; Khot, T.; Sabharwal, A.; Schoenick, C.; and Tafjord, O. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457. find
  5. Dettmers, T.; Lewis, M.; Belkada, Y.; and Zettlemoyer, L. 2022. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. arXiv:2208.07339. find
  6. Dettmers, T.; Pagnoni, A.; Holtzman, A.; and Zettlemoyer, L. 2023. QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314. find
  7. Dong, Z.; Yao, Z.; Arfeen, D.; Gholami, A.; Mahoney, M. W.; and Keutzer, K. 2020. Hawq-v2: Hessian aware trace-weighted quantization of neural networks. NeurIPS, 33: 18518–18529. find
  8. Dong, Z.; Yao, Z.; Gholami, A.; Mahoney, M. W.; and Keutzer, K. 2019. Hawq: Hessian aware quantization of neural networks with mixed-precision. In CVPR, 293–302. find
  9. Esser, S. K.; McKinstry, J. L.; Bablani, D.; Appuswamy, R.; and Modha, D. S. 2019. Learned step size quantization. arXiv:1902.08153. — [arXiv:1902.08153](https://arxiv.org/abs/1902.08153) find
  10. Frantar, E.; and Alistarh, D. 2022. Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning. In NeurIPS. find
  11. Frantar, E.; Ashkboos, S.; Hoefler, T.; and Alistarh, D. 2022. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. arXiv:2210.17323. find
  12. Frantar, E.; Ashkboos, S.; Hoefler, T.; and Alistarh, D. 2023. OPTQ: Accurate Quantization for Generative Pre-trained Transformers. In ICLR. find
  13. Gao, L.; Tow, J.; Abbasi, B.; Biderman, S.; Black, S.; DiPofi, A.; Foster, C.; Golding, L.; Hsu, J.; Le Noac’h, A.; Li, H.; McDonell, K.; Muennighoff, N.; Ociepa, C.; Phang, J.; Reynolds, L.; Schoelkopf, H.; Skowron, A.; Sutawika, L.; Tang, E.; Thite, A.; Wang, B.; Wang, K.; and Zou, A. 2023. A fram… find
  14. Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2021. Measuring Massive Multitask Language Understanding. arXiv:2009.03300.
  15. Horowitz, M. 2014. 1.1 computing’s energy problem (and what we can do about it). In ISSCC, 10–14. find
  16. IEEE. Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In ICLR.
  17. OpenReview.net. IST-DASLab. 2022. gptq. — OpenAssistant Conversations — Democratizing Large Language Model Alignment https://github.com/IST-DASLab/ gptq. Köpf, A.; Kilcher, Y.; von Rütte, D.; Anagnostidis, S.; Tam, Z.-R.; Stevens, K.; Barhoum, A.; Duc, N. M.; Stanley, O.; Nagyfi, R.; et al. 2023. OpenAssistant Conversations– Democratizing Large Language Model Alignment. ar… ↗
  18. Li, Y.; Gong, R.; Tan, X.; Yang, Y.; Hu, P.; Zhang, Q.; Yu, F.; Wang, W.; and Gu, S. 2021. BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction. In ICLR. find
  19. Marcus, M.; Kim, G.; Marcinkiewicz, M. A.; MacIntyre, R.; Bies, A.; Ferguson, M.; Katz, K.; and Schasberger, B. 1994. The Penn treebank: Annotating predicate argument structure. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994.
  20. Merity, S.; Xiong, C.; Bradbury, J.; and Socher, R. 2016. Pointer sentinel mixture models. arXiv:1609.07843.
  21. Nagel, M.; Amjad, R. A.; Van Baalen, M.; Louizos, C.; and Blankevoort, T. 2020. Up or down? adaptive rounding for post-training quantization. In ICML, 7197–7206. PMLR. Nagel, M.; Baalen, M. v.; Blankevoort, T.; and Welling, M. 2019. Data-free quantization through weight equalization and bias correct… find
  22. Nahshan, Y.; Chmiel, B.; Baskin, C.; Zheltonozhskii, E.; Banner, R.; Bronstein, A. M.; and Mendelson, A. 2021. Loss aware post-training quantization. Machine Learning, 110(11-12): 3245–3262. find
  23. OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774.
  24. Park, E.; Kim, D.; and Yoo, S. 2018. Energy-efficient neural network accelerator based on outlier-aware low-precision computation. In ISCA, 688–698. find
  25. IEEE. Park, E.; Yoo, S.; and Vajda, P. 2018. Value-aware quantization for training and inference of neural networks. In ECCV, 580–595. find
  26. Park, G.; Park, B.; Kim, M.; Lee, S.; Kim, J.; Kwon, B.; Kwon, S. J.; Kim, B.; Lee, Y.; and Lee, D. 2023. LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models. arXiv:2206.09557. find
  27. Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al. 2019. Pytorch: An imperative style, high-performance deep learning library. NeurIPS, 32.
  28. Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; Sutskever, I.; et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8): 9. find
  29. Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1): 5485–5551. Scao, T. L.; Fan, A.; Akiki, C.; Pavlick, E.; I…
  30. Wei, X.; Gong, R.; Li, Y.; Liu, X.; and Yu, F. 2022a. QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantization. In ICLR. find
  31. Wei, X.; Zhang, Y.; Zhang, X.; Gong, R.; Zhang, S.; Zhang, Q.; Yu, F.; and Liu, X. 2022b. Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models. In Advances in Neural Information Processing Systems. find
  32. Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; et al. 2019. Huggingface’s transformers: State-of-the-art natural language processing. arXiv:1910.03771.
  33. Xiao, G.; Lin, J.; Seznec, M.; Demouth, J.; and Han, S. 2022. Smoothquant: Accurate and efficient post-training quantization for large language models. arXiv:2211.10438. find
  34. Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019. HellaSwag: Can a Machine Really Finish Your Sentence? arXiv:1905.07830. find
  35. Zhang, S.; Roller, S.; Goyal, N.; Artetxe, M.; Chen, M.; Chen, S.; Dewan, C.; Diab, M.; Li, X.; Lin, X. V.; et al. 2022. Opt: Open pre-trained transformer language models. arXiv:2205.01068.
  36. Zhou, S.; Wu, Y.; Ni, Z.; Zhou, X.; Wen, H.; and Zou, Y. 2016. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv:1606.06160. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lee-2023-owq-outlier-aware-weight-quantization/