Elamur Research⌕
☰
home / papers / 2026 / Exploring Layer-wise Information Effectiveness for Post-Training Quantization in Small Language Models

Exploring Layer-wise Information Effectiveness for Post-Training Quantization in Small Language Models

Paper · 2026 · He Xiao, Qingyao Yang, Dirui Xie et al.

Summary

Compressing small language models (under 8B parameters) to about 2 bits per weight usually ruins their accuracy. Methods that recover accuracy tend to rely on irregular formats, codebooks or per-element mixed precision, and these complicate deployment on edge hardware. The authors propose LieQ, a post-training quantization framework that gives every weight in a layer the same bit-width but lets precision differ between layers, so standard multiplication kernels still work.

The central observation is that a layer's functional importance closely tracks a static geometric property of its weights. Layers whose singular-value entropy drops most during training, which the authors call the Compactness Shift, are the ones that break under quantization. LieQ uses this as a sensitivity proxy and ranks layers without gradients or perplexity probing. A closed-form rule then assigns 4 bits to the top-ranked layers and 2 bits to the rest, within a target average budget.

On Qwen3 and LLaMA models, LieQ closes most of the gap left by uniform 2-bit baselines. This makes very low-bit small models a realistic option for on-device use.

Key points

  • Qwen3-4B at ~2.05 bits: LieQ scores 71.89 on PIQA and 63.89 on MMLU. GPTQ at 2 bits gets 52.18 and 26.44, and FP16 gets 74.97 and 68.25.
  • LLaMA-3.2-3B at ~2.07 bits: LieQ reaches 75.41 on PIQA against 76.82 for FP16. All 2-bit baselines (GPTQ, AWQ, OmniQuant, SliM-LLM) stay around 52–55.
  • LLaMA-2-7B at 2.05 bits: LieQ beats AQLM at 2.29 bits on PIQA (77.48 vs 74.92) and ARC-e (68.1 vs 66.5).
  • End-to-end decoding on Alpaca:
    • LLaMA-2-7B goes from 39.73 to 50.67 tokens/s, with memory falling from 12.86 GB to 4.32 GB.
    • A 13B LLaMA-1 model, which runs out of memory at baseline, runs at 40.09 tokens/s in 7.83 GB.
  • Ablation and scaling: The authors varied the number of 4-bit layers from 1 to 16. Protecting even a single geometrically critical layer recovers a large share of accuracy. The benefit is largest in smaller models, where layer criticality is more concentrated.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🎓University of Hong KongPok Fu Lam · China89 works · 11,774 citations🎓Huazhong University of Science and TechnologyWuhan · China45 works · 6,345 citations🎓Tsinghua Shenzhen International Graduate SchoolShenzhen · China28 works · 3,035 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksEngineering · 133 worksElectrical and Electronic Engineering · 18 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEfficient models & compression · 242 works

Cite

Cite as He Xiao, Qingyao Yang, Dirui Xie, Wendong XU, Zunhai Su, Runming Yang, Haobo Liu, Wenyong Zhou, Zhengwu Liu and Ngai Wong. 2026. Exploring Layer-wise Information Effectiveness for Post-Training Quantization in Small Language Models. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.771

@inproceedings{xiao2026exploring,
  title = {{Exploring Layer-wise Information Effectiveness for Post-Training Quantization in Small Language Models}},
  author = {He Xiao, Qingyao Yang, Dirui Xie, Wendong XU, Zunhai Su, Runming Yang, Haobo Liu, Wenyong Zhou, Zhengwu Liu and Ngai Wong},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.771},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.771}
}

References

43 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. 2025. Systematic outliers in large language models. arXiv preprint arXiv:2502.06415. — [arXiv:2502.06415](https://arxiv.org/abs/2502.06415) find
  3. Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. 2024. Quarot: Outlier-free 4-bit inference in rotated llms. Preprint, arXiv:2404.00456.
  4. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, and 12 others. 202… find
  5. Alireza Behtash, Marijan Fofonjka, Ethan Baird, Tyler Mauer, Hossein Moghimifam, David Stout, and Joel Dennison. 2025. Universality of layer-level entropy-weighted quantization beyond model architecture and size. Preprint, arXiv:2503.04704. — [arXiv:2503.04704](https://arxiv.org/abs/2503.04704) find
  6. Peter Belcak, Greg Heinrich, Shizhe Diao, Yonggan Fu, Xin Dong, Saurav Muralidharan, Yingyan Celine Lin, and Pavlo Molchanov. 2025. Small language models are the future of agentic ai. arXiv preprint arXiv:2506.02153. find
  7. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, and 1 others. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439.
  8. Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. 2023. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36:4396– 4429. find
  9. Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, and Ping Luo. 2025. Efficientqat: Efficient quantization-aware training for large language models. Preprint, arXiv:2407.11062.
  10. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044.
  11. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.
  12. Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world’s first truly open instruction-tuned llm. find
  13. Tim Dettmers, Ruslan A. Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. 2024. SpQR: A sparse-quantized representation for near-lossless LLM weight compression. In The Twelfth International Conference on Learning R… find
  14. Zhen Dong, Zhewei Yao, Amir Gholami, Michael Mahoney, and Kurt Keutzer. 2019. Hawq: Hessian aware quantization of neural networks with mixed-precision. Preprint, arXiv:1905.03696. find
  15. Razvan-Gabriel Dumitru, Vikas Yadav, Rishabh Maheshwary, Paul-Ioan Clotan, Sathwik Tejaswi Madhusudhan, and Mihai Surdeanu. 2024. Layer-wise quantization: A pragmatic and effective method for quantizing llms beyond integer bit-levels. Preprint, arXiv:2406.17415. — [arXiv:2406.17415](https://arxiv.org/abs/2406.17415) find
  16. Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. 2024. Extreme compression of large language models via additive quantization. Preprint, arXiv:2401.06118. find
  17. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. GPTQ: Accurate post-training compression for generative pretrained transformers. arXiv preprint arXiv:2210.17323. find
  18. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, and et.al. 2024. The language model evaluation harness. find
  19. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, and Aiesha Letman. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  20. Ziyi Guan, Hantao Huang, Yupeng Su, Hong Huang, Ngai Wong, and Hao Yu. 2024. Aptq: Attention-aware post-training mixed-precision quantization for large language models. In Proceedings of the 61st ACM/IEEE Design Automation Conference, page 1–6. ACM. find
  21. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  22. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Preprint, arXiv:2009.03300.
  23. Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, and XIAOJUAN QI. 2024. BiLLM: Pushing the limit of post-training quantization for LLMs. In Forty-first International Conference on Machine Learning. find
  24. Wei Huang, Haotong Qin, Yangdong Liu, Yawei Li, Qinshuo Liu, Xianglong Liu, Luca Benini, Michele Magno, Shiming Zhang, and Xiaojuan Qi. 2025. Slim-llm: Salience-driven mixed-precision quantization for large language models. Preprint, arXiv:2405.14917. find
  25. Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. 2017. Quantization and training of neural networks for efficient integer-arithmetic-only inference. Preprint, arXiv:1712.05877. find
  26. Shiyao Li, Xuefei Ning, Ke Hong, Tengxuan Liu, Luning Wang, Xiuhong Li, Kai Zhong, Guohao Dai, Huazhong Yang, and Yu Wang. 2023. Llm-mq: Mixed-precision quantization for efficient llm deployment. In The Efficient Natural Language and Speech Processing Workshop with NeurIPS, volume 9, page 3. find
  27. Zhen Li, Yupeng Su, Songmiao Wang, Runming Yang, Congkai Xie, Aofan Liu, Ming Li, Jiannong Cao, Yuan Xie, Ngai Wong, and Hongxia Yang. 2025. In-fijanice: Joint analysis and in-situ correction engine for quantization-induced math degradation in large language models. Preprint, arXiv:2505.11574. find
  28. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. In MLSys. find
  29. Lianbo Ma, Jianlun Ma, Yuee Zhou, Guoyang Xie, Qiang He, and Zhichao Lu. 2025. Learning from loss landscape: Generalizable mixed-precision quantization via adaptive sharpness-aware gradient aligning. find
  30. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. Preprint, arXiv:1609.07843.
  31. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, and Gregory Chanan et.al. 2019. Pytorch: An imperative style, high-performance deep learning library. Preprint, arXiv:1912.01703.
  32. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67.
  33. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106. find
  34. Yuzhang Shang, Zhihang Yuan, Qiang Wu, and Zhen Dong. 2023. Pb-llm: Partially binarized large language models. Preprint, arXiv:2310.00034. — [arXiv:2310.00034](https://arxiv.org/abs/2310.00034) find
  35. Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. 2024. Omniquant: Omnidirectionally calibrated quantization for large language models. Preprint, arXiv:2308.13137.
  36. Zunhai Su, Qingyuan Li, Hao Zhang, YuLei Qian, Yuchen Xie, and Kehong Yuan. 2025. Unveiling super experts in mixture-of-experts large language models. arXiv preprint arXiv:2507.23279. — [arXiv:2507.23279](https://arxiv.org/abs/2507.23279) find
  37. Zunhai Su and Kehong Yuan. 2025. Kvsink: Understanding and enhancing the preservation of attention sinks in kv cache quantization for llms. arXiv preprint arXiv:2508.04257. — [arXiv:2508.04257](https://arxiv.org/abs/2508.04257) find
  38. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, and Nikolay Bashlykov. 2023. Llama 2: Open foundation and fine-tuned chat models. Preprint, arXiv:2307.09288.
  39. Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. 2024. Quip\#: Even better llm quantization with hadamard incoherence and lattice codebooks. Preprint, arXiv:2402.04396.
  40. Zhewei Yao, Zhen Dong, Zhangcheng Zheng, Amir Gholami, Jiali Yu, Eric Tan, Leyuan Wang, Qijing Huang, Yida Wang, Michael W. Mahoney, and Kurt Keutzer. 2021. Hawqv3: Dyadic neural network quantization. Preprint, arXiv:2011.10680. find
  41. Mengxia Yu, De Wang, Qi Shan, Colorado J Reed, and Alvin Wan. 2024. The super weight in large language models. arXiv preprint arXiv:2411.07191. — [arXiv:2411.07191](https://arxiv.org/abs/2411.07191) find
  42. Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang, Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. 2023. Rptq: Reorder-based post-training quantization for large language models. Preprint, arXiv:2304.01089. — [arXiv:2304.01089](https://arxiv.org/abs/2304.01089) find
  43. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xiao-2026-exploring-layer-wise-information-effectiveness/