Elamur Research⌕
☰
home / papers / 2026 / Layer-Wise High-Impact Parameter Ratio Optimization in Post-Training Quantization for Large Language Models

Layer-Wise High-Impact Parameter Ratio Optimization in Post-Training Quantization for Large Language Models

Paper · 2026 · Cuong Pham, Anh Dung Hoang, Cuong C. Nguyen et al.

Summary

Post-training quantization (PTQ) of LLMs works well at 4 bits but degrades badly at around 2–3 bits. A major cause is a small set of high-impact weights. Earlier fixes such as AWQ, CherryQ and SqueezeLLM protect these weights with one fixed ratio for every layer. Some of them also use element-wise or non-uniform schemes that are awkward to run on hardware. The authors measure channel importance with Fisher information on LLaMA-2-7B and find that sensitivity varies widely across layers. They then pose a quadratic optimization problem that picks a per-layer, channel-wise share of high-impact parameters and accounts for dependencies between layers within a block. The selected channels are kept at a moderate precision (e.g., 4-bit) rather than FP16, so more of them fit in the same memory budget. The other channels go to extremely low bit-widths. This split enables a hybrid pipeline: costly AdaRound handles only the important channels, and lightweight OmniQuant handles the rest. On LLaMA-2 models, the approach matches or beats GPTQ, AWQ, OmniQuant, SqueezeLLM and CBQ, especially at 2-bit.

Key points

  • Importance analysis: channel importance is the average Fisher information, computed on LLaMA-2-7B with 128 WikiText-2 calibration samples. High-impact channels cluster in a few dimensions and differ by layer, e.g., the key projection in block 2 versus the MLP down-projection in block 3.
  • 3-bit results (LLaMA-2-7B, W3A16g64, ~3.30 avg. bits): perplexity is 7.49 on C4 and 5.82 on WikiText-2, against 7.56 and 5.89 for CBQ. On LLaMA-2-13B the same setting gives 6.80 (C4) and 5.16 (WikiText-2).
  • 2-bit commonsense reasoning (W2A16g128): across PiQA, ArcE, ArcC, HellaSwag and WinoGrande, average accuracy is 49.04. This compares with 47.59 for OmniQuant, 38.53 for GPTQ and 64.87 for FP16.
  • Ablation (LLaMA-2-7B, W2A16): WikiText-2 perplexity falls from 37.27 with plain OmniQuant to 10.66 with a uniform ratio and 9.80 with the optimized ratio. Adding the hybrid scheme brings it to 9.40.
  • Cost and limitation: dependency analysis stays within blocks, so complexity is O(B·L_b(L_b−1)) and grows linearly with the number of blocks. The higher bit-width for high-impact parameters is still set manually.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Monash UniversityMelbourne · Australia60 works · 54,866 citations🎓University of SurreyGuildford · United Kingdom21 works · 677 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEfficient models & compression · 242 works

Cite

Cite as Cuong Pham, Anh Dung Hoang, Cuong C. Nguyen, Trung Le, Gustavo Carneiro and Thanh-Toan Do. 2026. Layer-Wise High-Impact Parameter Ratio Optimization in Post-Training Quantization for Large Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.2092

@inproceedings{pham2026layer,
  title = {{Layer-Wise High-Impact Parameter Ratio Optimization in Post-Training Quantization for Large Language Models}},
  author = {Cuong Pham, Anh Dung Hoang, Cuong C. Nguyen, Trung Le, Gustavo Carneiro and Thanh-Toan Do},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2092},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2092}
}

References

39 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. 2024. Quarot: Outlier-free 4-bit inference in rotated llms. In NeurIPS.
  2. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, and 1 others. 2020. Piqa: Reasoning about physical commonsense in natural language. In AAAI.
  3. Weihan Chen, Peisong Wang, and Jian Cheng. 2021. Towards mixed-precision quantization of neural networks via constrained optimization. In ICCV. find
  4. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv.
  5. Wanyun Cui and Qianle Wang. 2024. Cherry on top: Parameter heterogeneity and quantization in large language models. In NeurIPS. find
  6. Zihao Deng, Xin Wang, Sayeh Sharify, and Michael Orshansky. 2023. Mixed-precision quantization with cross-layer dependencies. arXiv preprint arXiv:2307.05657. — [arXiv:2307.05657](https://arxiv.org/abs/2307.05657) find
  7. Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. 2023. Q-Diffusion: Quantizing diffusion models. In ICCV. find
  8. Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. 2021. BRECQ: Pushing the limit of post-training quantization by block reconstruction. In ICLR. find
  9. Chen Lin, Bo Peng, Zheyang Li, Wenming Tan, Ye Ren, Jun Xiao, and Shiliang Pu. 2023. Bit-shrinking: Limiting instantaneous sharpness for improving post-training quantization. In CVPR. find
  10. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. In MLSys. find
  11. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. In NeurIPS. find
  12. Jiawei Liu, Lin Niu, Zhihang Yuan, Dawei Yang, Xinggang Wang, and Wenyu Liu. 2023a. Pd-quant: Post-training quantization based on prediction difference metric. In CVPR. find
  13. Xin Ding, Xiaoyu Liu, Zhijun Tu, Yun Zhang, Wei Li, Jie Hu, Hanting Chen, Yehui Tang, Zhiwei Xiong, Baoqun Yin, and Yunhe Wang. 2025. CBQ: Cross-block quantization for large language models. In ICLR. find
  14. Jing Liu, Ruihao Gong, Xiuying Wei, Zhiwei Dong, Jianfei Cai, and Bohan Zhuang. 2023b. Qllm: Accurate and efficient low-bitwidth quantization for large language models. arXiv. find
  15. Zhen Dong, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. 2019. Hawq: Hessian aware quantization of neural networks with mixed-precision. In ICCV. find
  16. Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. 2025. Spinquant: LLM quantization with learned rotations. In ICLR. find
  17. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. GPTQ: Accurate post-training quantization for generative pre-trained transformers. In ICLR. find
  18. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843.
  19. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In ICLR.
  20. Yushi Huang, Ruihao Gong, Jing Liu, Tianlong Chen, and Xianglong Liu. 2024. TFMQ-DM: Temporal feature maintenance quantization for diffusion models. In CVPR. find
  21. Yongkweon Jeon, Chungman Lee, and Ho-young Kim. 2023. Genie: Show me the data for quantization. In CVPR. find
  22. Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. 2024. SqueezLLM: Dense-and-sparse quantization. In ICML. find
  23. Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park. 2024. Owq: Outlier-aware weight quantization for efficient fine-tuning and inference of large language models. In AAAI.
  24. Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. 2020. Up or down? adaptive rounding for post-training quantization. In ICML. find
  25. Cuong Pham, Anh Dung Hoang, Cuong C Nguyen, Trung Le, Dinh Phung, Gustavo Carneiro, and Thanh-Toan Do. 2024. MetaAug: Meta-data augmentation for post-training quantization. In ECCV. find
  26. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR.
  27. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106. find
  28. Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. 2023. Post-training quantization on diffusion models. In CVPR. find
  29. Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. 2024. Omniquant: Omnidirectionally calibrated quantization for large language models. In ICLR.
  30. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv.
  31. Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. 2024. Quip\#: Even better llm quantization with hadamard incoherence and lattice codebooks. In ICML.
  32. Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. 2019. Haq: Hardware-aware automated quantization with mixed precision. In CVPR. find
  33. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022a. Emergent abilities of large language models. TMLR.
  34. Xiuying Wei, Ruihao Gong, Yuhang Li, Xianglong Liu, and Fengwei Yu. 2022b. Qdrop: randomly dropping quantization for extremely low-bit post-training quantization. In ICLR. find
  35. Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. 2023. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling. arXiv preprint arXiv:2304.09145. — [arXiv:2304.09145](https://arxiv.org/abs/2304.09145) find
  36. Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. 2022c. Outlier suppression: Pushing the limit of low-bit transformer language models. In NeurIPS. find
  37. Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. Smoothquant: Accurate and efficient post-training quantization for large language models. In ICML. find
  38. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In ACL. find
  39. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, and 1 others. 2022. Opt: Open pre-trained transformer language models. arXiv.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pham-2026-layer-wise-high-impact-parameter/