Elamur Research⌕
☰
home / papers / 2026 / PE-QAT: Parameter-Efficient Quantization-Aware Training for Large Language Models

PE-QAT: Parameter-Efficient Quantization-Aware Training for Large Language Models

Paper · 2026 · Shresth Mishra

Summary

Running large language models on devices requires aggressive compression. However, quantizing both weights and activations to 4 bits (W4A4) usually hurts accuracy, mainly because a few activation channels contain extreme outliers. Full quantization-aware training (QAT) can recover that accuracy, but it costs too much at LLM scale. PE-QAT offers a cheaper alternative. It keeps the full-precision base model frozen and trains only LoRA adapters. During training, the merged weights are fake-quantized using Straight-Through Estimators, so the adapters learn to offset quantization error. To deal with outliers, the method also learns two sets of per-channel parameters through backpropagation: SmoothQuant-style smoothing factors and symmetric activation clipping thresholds. Both are initialized from 99.5th-percentile calibration statistics. On Llama-2-7B, PE-QAT nearly matches FP16 accuracy on zero-shot commonsense reasoning while updating only about 1% of parameters. This places it between cheap post-training quantization and expensive full QAT.

Key points

  • Setup: Per-channel symmetric W4A4 quantization is applied to the attention and MLP projections. The KV cache stays in FP16, and the current implementation uses fake quantization only.
  • Accuracy and cost: On Llama-2-7B, average zero-shot accuracy is 66.69 versus 66.80 for FP16, with 1.26% of parameters trained. This is about 0.19 points above SpinQuant-H, and PE-QAT does not need Hadamard rotations.
  • Perplexity: WikiText-2 perplexity is 7.58, compared with 5.47 for FP16 and 5.90 for SpinQuant-H. This beats OmniQuant (14.26), ZeroQAT (12.95), QA-LoRA (9.50) and SmoothQuant (83.12).
  • Ablations:
    • Removing learnable clipping raises perplexity by 3.00, while removing smoothing raises it by only 0.40.
    • Percentile-based initialization outperforms max-based initialization.
  • Limitations:
    • Experiments cover models only up to 13B parameters.
    • No end-to-end INT4 latency measurements are reported.
    • Per-channel activation quantization needs specialized kernels.
    • MMLU and GSM8K are not evaluated.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEfficient models & compression · 242 works

Cite

Cite as Shresth Mishra. 2026. PE-QAT: Parameter-Efficient Quantization-Aware Training for Large Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-srw.63

@inproceedings{mishra2026pe,
  title = {{PE-QAT: Parameter-Efficient Quantization-Aware Training for Large Language Models}},
  author = {Shresth Mishra},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-srw.63},
  url = {https://doi.org/10.18653/v1/2026.acl-srw.63}
}

References

25 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. 2024. Quarot: Outlier-free 4-bit inference in rotated llms. arXiv preprint arXiv:2404.00456.
  2. Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. Preprint, arXiv:1308.3432. find
  3. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, and 1 others. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7432–7439.
  4. In International Conference on Learning Representations. find
  5. Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. 2021. A survey of quantization methods for efficient neural network inference. arXiv preprint arXiv:2103.13630. — [arXiv:2103.13630](https://arxiv.org/abs/2103.13630) find
  6. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  7. Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. 2018. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni… find
  8. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  9. Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. 2018. Pact: Parameterized clipping activation for quantized neural networks. In International Conference on Machine Learning, pages 834–842. PMLR. find
  10. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:…
  11. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.
  12. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Llm.int8(): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:30318–30332. find
  13. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36.
  14. Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S Modha. 2020. Learned step size quantization. In International Conference on Learning Representations. find
  15. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. Gptq: Accurate post-training quantization for generative pre-trained transformers. find
  16. Hyesung Jeon, Yulhwa Kim, and Jae-Joon Kim. 2025. L4Q: Parameter efficient quantization-aware fine-tuning on large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2002–2024, Vienna, Austria. Association for Co…
  17. Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, Nikos Karampatziakis, Weizhu Chen, and Tuo Zhao. 2023. Loftq: Lora-fine-tuning-aware quantization for large language models. Preprint, arXiv:2310.08659. — [arXiv:2310.08659](https://arxiv.org/abs/2310.08659) find
  18. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for llm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87–100. find
  19. Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. 2023. Llm-qat: Data-free quantization aware training for large language models. Preprint, arXiv:2305.17888.
  20. Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Paul N Whatmough, Barlas Oguz, Yuhao Tang, Jinjie Zhou, Wonmin Cho, Raghuraman Krishnamoorthi, and 1 others. 2024. Spinquant: Llm quantization with learned rotations. arXiv preprint arXiv:2405.16406. — [arXiv:2405.16406](https://arxiv.org/abs/2405.16406) find
  21. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. Preprint, arXiv:1609.07843.
  22. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391. find
  23. Shresth Mishra, Daniel Cionca, and Raunak Pradhan. 2025. A novel framework for vehicle detection and dynamic light control via yolov8n on edge-deployed imx500. In 2025 International Conference on Machine Learning and Applications (ICMLA), pages 772–777. find
  24. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740. find
  25. Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. 2024. Omniquant: Omnidirectionally calibrated quantization for large language models. Preprint, arXiv:2308.13137.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mishra-2026-pe-qat-parameter-efficient-quantization/