Elamur Research⌕
☰
home / papers / 2026 / From 2:4 to 8:16 sparsity patterns in LLMs for Outliers and Weights with Variance Correction

From 2:4 to 8:16 sparsity patterns in LLMs for Outliers and Weights with Variance Correction

Paper · 2026 · Egor Maximov, Yulia Kuzkina, Egor Shvetsov et al.

Summary

The paper examines why N:M semi-structured pruning of LLMs lags behind quantization. The authors argue that the hardware-supported 2:4 pattern is too rigid and tends to destroy outlier weights. They test the wider 8:16 pattern, which costs 0.875 bits/element of metadata versus 0.75 for 2:4, to see whether it can reach the Performance Threshold.

Their pipeline has four steps:

  • SmoothQuant-inspired weight equalization, used only when computing importance scores
  • RIA-based pruning, with salient weights kept in structured 4:256, 8:256 or 16:256 patterns
  • a post-pruning Variance Correction (VC) that rescales the remaining weights back to the dense variance
  • blockwise EBFT fine-tuning

Experiments cover LLaMA2 7B/13B, LLaMA3 8B and Mistral 7B. Evaluation uses WikiText and C4 perplexity plus zero-shot ARC, WinoGrande, HellaSwag and PIQA.

With 8:16 sparsity, LLaMa-2-13B performs on par with dense LLaMa-2-7B. Storing outliers in structured form beats unstructured storage of a similar size. The results support building future hardware for 8:16 sparsity, although no such hardware exists yet and speedups are projected rather than measured.

Key points

  • LLaMA3-8B on WikiText-2 with RIA: perplexity is 22.53 at 2:4 and 10.64 at 8:16. Adding VC lowers these to 16.66 and 9.95. The 16:32 pattern helps only slightly more (9.51 with VC) and would require much more complex circuitry.
  • The outlier patterns 4:256, 8:256 and 16:256 keep roughly 1.5%, 3.1% and 6.25% of weights. For magnitude-pruned 2:4 LLaMA2-7B, adding 4:256 outliers lowers perplexity from 37.96 to 23.06.
  • The best configuration is 16:256 outliers plus EBFT:
    • LLaMA3 reaches 8.24 (2:4) and 7.40 (8:16), against a dense 6.24.
    • Mistral reaches 6.27 and 5.82, against a dense 5.32.
    • The paper's conclusion quotes 6.27 for Mistral as the 8:16 result, which conflicts with its results section.
  • Zero-shot average accuracy for LLaMA3 falls from 69.23 to 61.75 under 2:4 and to 64.89 under 8:16.
  • VC substantially improves LLaMA3 but degraded Mistral's generation, so it was left out for that model. Mistral proved more robust to pruning overall.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🤝AIRI - Artificial Intelligence Research InstituteMoscow · Russia5 works · 29 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Egor Maximov, Yulia Kuzkina, Egor Shvetsov, Azamat Kanametov, Aleksandr Prutko, Maxim Zhelnin and Aleksei Goncharov. 2026. From 2:4 to 8:16 sparsity patterns in LLMs for Outliers and Weights with Variance Correction. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-industry.66

@inproceedings{maximov20262,
  title = {{From 2:4 to 8:16 sparsity patterns in LLMs for Outliers and Weights with Variance Correction}},
  author = {Egor Maximov, Yulia Kuzkina, Egor Shvetsov, Azamat Kanametov, Aleksandr Prutko, Maxim Zhelnin and Aleksei Goncharov},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-industry.66},
  url = {https://doi.org/10.18653/v1/2026.acl-industry.66}
}

References

31 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, and 1 others. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439.
  2. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.
  3. Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:30318– 30332. find
  4. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023a. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088–10115.
  5. Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. 2023b. Spqr: A sparse-quantized representation for near-lossless llm weight compression. arXiv preprint arXiv:2306.03078. find
  6. Elias Frantar and Dan Alistarh. 2023. Sparsegpt: massive language models can be accurately pruned in one-shot. In Proceedings of the 40th International Conference on Machine Learning, pages 10323–10337. find
  7. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323. find
  8. Elias Frantar, Utku Evci, Wonpyo Park, Neil Houlsby, and Dan Alistarh. 2025. Compression scaling laws: Unifying sparsity and quantization. arXiv preprint arXiv:2502.16440. — [arXiv:2502.16440](https://arxiv.org/abs/2502.16440) find
  9. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv e-prints, pages arXiv–2407.
  10. Song Guo, Fan Wu, Lei Zhang, Xiawu Zheng, Shengchuan Zhang, Fei Chao, Yiyu Shi, and Rongrong Ji. 2024. Ebft: Effective and block-wise fine-tuning for sparse llms. arXiv preprint arXiv:2402.12419. find
  11. Simla Burcu Harma, Ayan Chakraborty, Elizaveta Kostenok, Danila Mishin, Dongho Ha, Babak Falsafi, Martin Jaggi, Ming Liu, Yunho Oh, Suvinay Subramanian, and 1 others. 2024. Effective interplay between sparsity and quantization: From theory to practice. arXiv preprint arXiv:2405.20935. — [arXiv:2405.20935](https://arxiv.org/abs/2405.20935) find
  12. Yuezhou Hu, Kang Zhao, Weiyu Huang, Jianfei Chen, and Jun Zhu. 2024. Accelerating transformer pre-training with 2: 4 sparsity. arXiv preprint arXiv:2404.01847. find
  13. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, and 1 others. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. find
  14. Artyom Kharinaev, Viktor Moskvoretskii, Egor Shvetsov, Kseniia Studenikina, Bykov Mikhail, and Evgeny Burnaev. 2025. Investigating the impact of quantization methods on the safety and reliability of large language models. arXiv preprint arXiv:2502.15799. — [arXiv:2502.15799](https://arxiv.org/abs/2502.15799) find
  15. Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park. 2024. Owq: Outlier-aware weight quantization for efficient fine-tuning and inference of large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 13355–13364.
  16. Zhuo Li, Hengyi Li, and Lin Meng. 2023. Model compression for deep neural networks: A survey. Computers, 12(3):60. find
  17. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843.
  18. Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. 2019. Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1325–1334. find
  19. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67.
  20. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106. find
  21. Christoph Schulte, Sven Wagner, Armin Runge, Dimitrios Bariamis, and Barbara Hammer. 2023. Best of both, structured and unstructured sparsity in neural networks. In Proceedings of the 3rd Workshop on Machine Learning and Systems, pages 104–108. find
  22. Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. 2023. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations. find
  23. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  24. Vinod Veeramachaneni. 2025. Large language models: A comprehensive survey on architectures, applications, and challenges. Advanced Innovations in Computer Programming Languages, 7(1):20–39. find
  25. Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087–38099. PMLR. find
  26. Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yu Hsieh, Yaqing Wang, Yiling Jia, Mykola Pechenizkiy, Yi Liang, Zhangyang Wang, and Shiwei Liu. 2023. Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity. In Conference on Parsimony and Learning (Recent Spotlight Tra… find
  27. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830. find
  28. Yingtao Zhang, Haoli Bai, Haokun Lin, Jialin Zhao, Lu Hou, and Carlo Vittorio Cannistraci. 2024. Plug-and-play: An efficient post-training pruning method for large language models. In The Twelfth International Conference on Learning Representations. find
  29. Yuxin Zhang, Mingbao Lin, Zhihang Lin, Yiting Luo, Ke Li, Fei Chao, Yongjian Wu, and Rongrong Ji. 2022. Learning best combination for efficient n: M sparsity. Advances in Neural Information Processing Systems, 35:941–953. find
  30. Kang Zhao, Tao Yuan, Han Bao, Zhenfeng Su, Chang Gao, Zhaofeng Sun, Zichen Liang, Liping Jing, and Jianfei Chen. 2024. Beyond 2: 4: exploring v: N: M sparsity for efficient transformer inference on gpus. arXiv preprint arXiv:2410.16135. find
  31. Maxim Zhelnin, Viktor Moskvoretskii, Egor Shvetsov, Egor Venediktov, Mariya Krylova, Aleksandr Zuev, and Evgeny Burnaev. 2024. Gift-sw: Gaussian noise injected fine-tuning of salient weights for llms. arXiv preprint arXiv:2408.15300. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/maximov-2026-2-4-8-16-sparsity/