Elamur Research⌕
☰
home / papers / 2026 / Through a Compressed Lens: Investigating The Impact of Quantization on Factual Knowledge Recall

Through a Compressed Lens: Investigating The Impact of Quantization on Factual Knowledge Recall

Paper · 2026 · Qianli Wang, Mingyang Wang, Nils Feldhus et al.

Summary

The paper asks how post-training weight quantization affects factual knowledge recall (FKR), meaning an LLM's ability to retrieve facts stored during pretraining. The authors tested Llama3-8B, Qwen2.5-7B and Qwen2.5-14B in full precision and in versions compressed with GPTQ (4- and 8-bit), AWQ and BitsAndBytes integer quantization (bib4, bib8). They used two kinds of analysis:

  • Knowledge memorization: they measured recall accuracy on LRE triplets, then used neuron and layer attribution (Yu and Ananiadou, 2024) to locate where information is lost.
  • Latent multi-hop reasoning: following Yang et al. (2024), they applied TwoHop-Fact with three measures: entity recall, consistency between two-hop and one-hop answers, and accuracy.

Quantization usually lowers FKR, and smaller models in a family suffer more. However, fewer bits do not reliably mean worse recall, and quantization occasionally improves it. BitsAndBytes kept the original behaviour best. Overall the losses were modest. For anyone deploying compressed models where factual reliability matters, this suggests quantization remains a reasonable choice.

Key points

  • Accuracy on LRE: AWQ, GPTQ4 and bib4 reduce accuracy for every model size. GPTQ8 and bib8 stay close to full precision. Relations the full-precision model has not already mastered degrade the most.
  • Where the damage occurs: Contributions of the top-300 feed-forward neurons drop mainly in the final layers of Qwen2.5-7B. In Llama3-8B the losses appear in middle-to-late layers instead, which points to architecture-specific ways of storing facts.
  • Multi-hop effects: Quantization hurts the first hop by up to 30.08%, while the second hop drops only 4.25% on average. Two-hop errors track bridge-entity recall closely (Spearman 0.93). Qwen2.5-7B's average decline is just 0.77%.
  • Model and method comparison: Of the three models, Llama3-8B is the least affected. bib4/8 best preserves FKR and sometimes enhances it, notably in Qwen2.5-14B.
  • Limitation: All experiments use English data only, so the authors leave multilingual FKR under quantization to future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬German Research Centre for Artificial IntelligenceKaiserslautern · Germany77 works · 29,479 citations🎓Technical University of MunichMunich · Germany55 works · 48,474 citations🎓Technische Universität BerlinBerlin · Germany22 works · 957 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Efficient models & compression · 242 works

Cite

Cite as Qianli Wang, Mingyang Wang, Nils Feldhus, Simon Ostermann, Yuan Cao, Hinrich Schuetze, Sebastian Möller and Vera Schmitt. 2026. Through a Compressed Lens: Investigating The Impact of Quantization on Factual Knowledge Recall. Proceedings of the 6th Workshop on Trustworthy NLP (TrustNLP 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.trustnlp-main.2

@inproceedings{wang2026through,
  title = {{Through a Compressed Lens: Investigating The Impact of Quantization on Factual Knowledge Recall}},
  author = {Qianli Wang, Mingyang Wang, Nils Feldhus, Simon Ostermann, Yuan Cao, Hinrich Schuetze, Sebastian Möller and Vera Schmitt},
  booktitle = {Proceedings of the 6th Workshop on Trustworthy NLP (TrustNLP 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.trustnlp-main.2},
  url = {https://doi.org/10.18653/v1/2026.trustnlp-main.2}
}

References

26 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Minyeong Choe, Haehyun Cho, Changho Seo, and Hyunil Kim. 2025. Do all autoregressive transformers remember facts the same way? a cross-architecture analysis of recall mechanisms. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 28494–28513, Suzhou, Chi… find
  2. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Llm.int8(): 8-bit matrix multiplication for transformers at scale. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. Curran Associates Inc. find
  3. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations. find
  4. In our experiments, we extensively compare full-precision models with different quantized versions in 4-bit and 8-bit formats. Lower-bit quantization, such as 1-bit or 2-bit, is not included in our study. find
  5. Gustavo Gonçalves and Emma Strubell. 2023. Understanding the effect of model compression on social bias in large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2663–2675, Singapore. Association for Computational Linguistics. find
  6. Moreover, the scope of our experiments is limited to post-training quantization techniques. Investigating the impact of weight-activation quantization, KV cache compression, or quantization-aware training techniques on factual knowledge recall is counted as future work. find
  7. R.M. Gray and D.L. Neuhoff. 1998. Quantization. IEEE Transactions on Information Theory, 44(6):2325–2383. find
  8. Author Contributions Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. 2024. Linearity of relation decoding in transformer language models. In The Twelfth International Conference on Learning Representations. find
  9. Author contributions are listed according to the CRediT taxonomy as follows: • QW: Writing, idea conceptualization, experiments and evaluations, analysis, visualization. • MW: Writing, experiments and evaluations, and analysis. • NF: Writing – review & editing, experiments and evaluations, and super… find
  10. Junyuan Hong, Jinhao Duan, Chenhui Zhang, Zhangheng Li, Chulin Xie, Kelsey Lieberman, James Diffenderfer, Brian R. Bartoldson, Ajay Kumar Jaiswal, Kaidi Xu, Bhavya Kailkhura, Dan Hendrycks, Dawn Song, Zhangyang Wang, and Bo Li. 2024. Decoding compressed trust: Scrutinizing the trustworthiness of eff… find
  11. Renren Jin, Jiangcun Du, Wuwei Huang, Wei Liu, Jian Luan, Bin Wang, and Deyi Xiong. 2024. A comprehensive evaluation of quantization strategies for large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12186–12215, Bangkok, Thailand. Association for Com… find
  12. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu.… find
  13. Elisabeth Kirsten, Ivan Habernal, Vedant Nanda, and Muhammad Bilal Zafar. 2025. The impact of inference acceleration on bias of LLMs. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1:… find
  14. Krithika Ramesh, Arnav Chavan, Shrey Pandit, and Sunayana Sitaram. 2023. A comparative study on the impact of model compression techniques on fairness in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15762–… find
  15. Qun Li, Yuan Meng, Chen Tang, Jiacheng Jiang, and Zhi Wang. 2024. Investigating the impact of quantization on adversarial robustness. In 5th Workshop on practical ML for limited/low resource settings. find
  16. Manpreet Singh and Hassan Sajjad. 2025. Interpreting the effects of quantization on LLMs. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, pages 2267–2281, Mum… find
  17. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. In Proceedings of Machine Learning and Systems, volume 6, pages 87–100. find
  18. Qianli Wang, Nils Feldhus, Pepa Atanasova, Fedor Splitt, Simon Ostermann, Sebastian Möller, and Vera Schmitt. 2026. Can large language models still explain themselves? investigating the impact of quantization on self-explanations. Preprint, arXiv:2601.00282. — [arXiv:2601.00282](https://arxiv.org/abs/2601.00282) find
  19. Peiyu Liu, Zikang Liu, Ze-Feng Gao, Dawei Gao, Wayne Xin Zhao, Yaliang Li, Bolin Ding, and Ji-Rong Wen. 2024. Do emergent abilities exist in quantized large language models: An empirical study. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources… find
  20. Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. Smoothquant: accurate and efficient post-training quantization for large language models. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org. find
  21. Kelly Marchisio, Saurabh Dash, Hongyu Chen, Dennis Aumiller, Ahmet Üstün, Sara Hooker, and Sebastian Ruder. 2024. How does quantization affect multilingual LLMs? In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 15928–15947, Miami, Florida, USA. Association for Computat… find
  22. Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. 2024. Do large language models latently perform multi-hop reasoning? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10210–10229, Bangkok, Thailand.… find
  23. Satya Sai Srinath Namburi, Makesh Sreedhar, Srinath Srinivasan, and Frederic Sala. 2023. The cost of compression: Investigating the impact of compression on parametric knowledge in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5255–5273, Singapore.… find
  24. Zeping Yu and Sophia Ananiadou. 2024. Neuron-level knowledge attribution in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3267–3280, Miami, Florida, USA. Association for Computational Linguistics. find
  25. Sein Park, Yeongsang Jang, and Eunhyeok Park. 2022. Symmetry regularization and saturating nonlinearity for robust quantization. In Computer Vision – ECCV 2022, pages 206–222, Cham. Springer Nature Switzerland. find
  26. Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. 2024. A survey on model compression for large language models. Transactions of the Association for Computational Linguistics, 12:1556–1577.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-through-compressed-lens-investigating-impact/