Elamur Research⌕
☰
home / papers / 2026 / Bit-by-Bit: Progressive QAT Strategy with Outlier Channel Splitting for Stable Low-Bit LLMs

Bit-by-Bit: Progressive QAT Strategy with Outlier Channel Splitting for Stable Low-Bit LLMs

Paper · 2026 · Binxing Xu, Hao Gu, Lujun Li et al.

Summary

Quantization-aware training (QAT) of LLMs at extreme precisions, such as 2-bit weights and activations, often breaks down. The loss landscape becomes rugged, loss spikes occur, outlier channels inflate quantization noise, and errors build up in deeper Transformer blocks. Bit-by-Bit addresses this with a progressive QAT scheme that lowers precision in stages. Weights go first (e.g., 8→4→2 bit), then activations. Each stage is trained with a block-wise reconstruction objective, so the higher-precision phase provides a good starting point for the next. Because integer grids of different bit-widths are nested, a single training run produces one model deployable at several precisions. The method also splits outlier channels in a rounding-aware way: the split works as an identity transform on quantized outputs while reducing rounding error. Microscaling groups with E4M3 scales handle activation ranges. Custom CUDA kernels for W2A16 and W2A2 make the approach practical. Across Llama-2/3 and Mistral, it beats BitDistiller, EfficientQAT and ParetoQ under equal budgets. This makes cheap, stable ultra-low-bit deployment more realistic.

Key points

  • On LLaMA-2 7B with W2A2, WikiText2 perplexity is 7.72 versus 5.47 for FP16, an increase of only 2.25. The method needs about 3600× fewer training tokens than ParetoQ.
  • Once-for-any-precision on Mistral-7B: one run serves w8a16, w4a16 and w2a16.
    • At w2a16 it reaches a 65.37 task average and 10.73 C4 PPL.
    • This compares with OmniQuant at 50.99 PPL (trained separately per bit-width) and MatQuant at 13.05 PPL.
  • Ablations on LLaMA3.2-1B (w2a16):
    • Block-wise loss, progressive training and outlier channel splitting (OCS) each add gains.
    • For detecting outlier channels, the metric ‖x‖₂·max|w| beats kurtosis, weight-max and activation-max alternatives.
    • OCS raises memory only from 0.33GB to 0.36GB.
    • Group size 128 lowers task accuracy from 45.18 to 38.60.
  • The custom 2-bit GEMV kernels reach up to 11× speedup over BF16, and OCS adds negligible overhead. End-to-end decoding on Llama 3-8B runs at 76 tokens/s, versus 49 tokens/s for the Transformers baseline.
  • Stated limitations: total QAT time is somewhat longer than with one-shot methods, and KV-cache quantization is left for future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hong Kong University of Science and TechnologyHong Kong · China170 works · 26,341 citations🎓Zhejiang UniversityHangzhou · China224 works · 13,587 citations🎓City University of Hong KongHong Kong · China59 works · 3,480 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksEfficient models & compression · 297 works

Cite

Cite as Binxing Xu, Hao Gu, Lujun Li, Hao Wang, Bei Liu, Jiacheng Liu, Qiyuan Zhu, Xintong Yang, Chao Li, Sirui Han and Yike Guo. 2026. Bit-by-Bit: Progressive QAT Strategy with Outlier Channel Splitting for Stable Low-Bit LLMs. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1020

@inproceedings{xu2026bit,
  title = {{Bit-by-Bit: Progressive QAT Strategy with Outlier Channel Splitting for Stable Low-Bit LLMs}},
  author = {Binxing Xu, Hao Gu, Lujun Li, Hao Wang, Bei Liu, Jiacheng Liu, Qiyuan Zhu, Xintong Yang, Chao Li, Sirui Han and Yike Guo},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1020},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1020}
}

References

51 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. 2024. Quarot: Outlier-free 4-bit inference in rotated llms. Advances in Neural Information Processing Systems, 37:100213–100240.
  2. Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. 2019. Once-for-all: Train one network and specialize it for efficient deployment. arXiv preprint arXiv:1908.09791.
  3. Hong Chen, Chengtao Lv, Liang Ding, Haotong Qin, Xiabin Zhou, Yifu Ding, Xuebo Liu, Min Zhang, Jinyang Guo, Xianglong Liu, and 1 others. 2024a. Db-llm: Accurate dual-binarization for efficient llms. arXiv preprint arXiv:2402.11960. find
  4. Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, and Ping Luo. 2024b. Efficientqat: Efficient quantization-aware training for large language models. arXiv preprint arXiv:2407.11062.
  5. Pengyu Cheng, Jiacheng Wang, Tianle Chen, Bei Liu, Xiaofeng Hou, and Jiacheng Liu. 2026. Desirekv: Decoupling sensitivity and importance for reasoning-aware kv cache compression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 20518–20526. find
  6. Euntae Choi, Sumin Song, Woosang Lim, and Sungjoo Yoo. 2025. Rotate, clip, and partition: Towards w2a4kv4 quantization by integrating rotation and learnable non-uniform quantizer. arXiv preprint arXiv:2502.15779. find
  7. Lawrence T DeCarlo. 1997. On the meaning and use of kurtosis. Psychological methods, 2(3):292. find
  8. DeepSeek. 2025. Deepgemm: High-performance gemm implementation. Accessed: April 16, 2026. find
  9. Peijie Dong, Lujun Li, Yuedong Zhong, Dayou Du, Ruibo Fan, Yuhan Chen, Zhenheng Tang, Qiang Wang, Wei Xue, Yike Guo, and 1 others. 2024. Stbllm: Breaking the 1-bit barrier with structured binary llms. arXiv preprint arXiv:2408.01803. find
  10. Dayou Du, Yijia Zhang, Shijie Cao, Jiaqi Guo, Ting Cao, Xiaowen Chu, and Ningyi Xu. 2024. Bitdistiller: Unleashing the potential of sub-4-bit llms via self-distillation. arXiv preprint arXiv:2402.10631. find
  11. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv e-prints, pages arXiv–2407.
  12. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323. find
  13. Hao Gu, Lujun Li, Zheyu Wang, Bei Liu, Qiyuan Zhu, Sirui Han, and Yike Guo. 2025a. Btc-llm: Efficient sub-1-bit llm quantization via learnable transformation and binary codebook. arXiv preprint arXiv:2506.12040. find
  14. Hao Gu, Wei Li, Lujun Li, Qiyuan Zhu, Mark Lee, Shengjie Sun, Wei Xue, and Yike Guo. 2025b. Delta decompression for moe-based llms compression. arXiv preprint arXiv:2502.17298. — [arXiv:2502.17298](https://arxiv.org/abs/2502.17298) find
  15. Hao Gu, Hao Wang, Jiacheng Liu, Lujun Li, Qiyuan Zhu, Bei Liu, Binxing Xu, Lei Wang, Xintong Yang, Sida Lin, and 1 others. 2026. Qarl: Rollout-aligned quantization-aware rl for fast and stable training under training–inference mismatch. arXiv preprint arXiv:2604.07853. find
  16. Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, and Xiaojuan Qi. 2024. Billm: Pushing the limit of post-training quantization for llms. arXiv preprint arXiv:2402.04291. find
  17. Ayush Kaushal, Tejas Vaidhya, Arnab Kumar Mondal, Tejas Pandey, Aaryan Bhagat, and Irina Rish. 2024. Spectra: Surprising effectiveness of pretraining ternary language models at scale. arXiv preprint arXiv:2407.12327. find
  18. HyunJin Kim, Jungwoo Shin, and Alberto A Del Barrio. 2022. Ctmq: Cyclic training of convolutional neural networks with multiple quantization steps. arXiv preprint arXiv:2206.12794. — [arXiv:2206.12794](https://arxiv.org/abs/2206.12794) find
  19. Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. 2023. Squeezellm: Dense-and-sparse quantization. arXiv:2306.07629. arXiv preprint Lujun Li, Qiyuan Zhu, Jiacheng Wang, Xiaoyu Qin, Wei Li, Hao Gu, Sirui Han, and Yike Guo. 2026. Sub-moe: E…
  20. Wei Li, Lujun Li, You-Liang Huang, Mark G Lee, Shengjie Sun, Wei Xue, and Yike Guo. 2025. Structured mixture-of-experts llms compression via singular value decomposition. find
  21. Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. 2021. Brecq: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint arXiv:2102.05426. — [arXiv:2102.05426](https://arxiv.org/abs/2102.05426) find
  22. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of machine learning and systems, 6:87–100. find
  23. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024a. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. find
  24. Jiacheng Liu, Peng Tang, Wenfeng Wang, Yuhang Ren, Xiaofeng Hou, Pheng Ann Heng, Minyi Guo, and Chao Li. 2026. A survey on inference optimization techniques for mixture of experts models. ACM Computing Surveys, 58(10):1–37. find
  25. Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, and 1 others. 2025a. Muon is scalable for llm training. arXiv preprint arXiv:2502.16982. find
  26. Yifei Liu, Jicheng Wen, Yang Wang, Shengyu Ye, Li Lyna Zhang, Ting Cao, Cheng Li, and Mao Yang. 2024b. Vptq: Extreme low-bit vector post-training quantization for large language models. arXiv preprint arXiv:2409.17066. find
  27. Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. 2023. Llm-qat: Data-free quantization aware training for large language models. arXiv preprint arXiv:2305.17888.
  28. Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. 2024c. Spinquant: Llm quantization with learned rotations. arXiv preprint arXiv:2405.16406. — [arXiv:2405.16406](https://arxiv.org/abs/2405.16406) find
  29. Zechun Liu, Changsheng Zhao, Hanxian Huang, Sijia Chen, Jing Zhang, Jiawei Zhao, Scott Roy, Lisa Jin, Yunyang Xiong, Yangyang Shi, and 1 others. 2025b. Paretoq: Scaling laws in extremely low-bit llm quantization. arXiv preprint arXiv:2502.02631. — [arXiv:2502.02631](https://arxiv.org/abs/2502.02631) find
  30. Liqun Ma, Mingjie Sun, and Zhiqiang Shen. 2024. Fbi-llm: Scaling up fully binarized llms from scratch via autoregressive distillation. arXiv preprint arXiv:2407.07093.
  31. Shuming Ma, Hongyu Wang, Shaohan Huang, Xingxing Zhang, Ying Hu, Ting Song, Yan Xia, and Furu Wei. 2025. Bitnet b1. 58 2b4t technical report. arXiv preprint arXiv:2504.12285. find
  32. Vladimir Malinovskii, Denis Mazur, Ivan Ilin, Denis Kuznedelev, Konstantin Burlachenko, Kai Yi, Dan Alistarh, and Peter Richtarik. 2024. Pv-tuning: Beyond straight-through estimation for extreme llm compression. Advances in Neural Information Processing Systems, 37:5074–5121. find
  33. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In ICLR.
  34. Pranav Nair, Puranjay Datta, Jeff Dean, Prateek Jain, and Aditya Kusupati. 2025. Matryoshka quantization. arXiv preprint arXiv:2502.06786. — [arXiv:2502.06786](https://arxiv.org/abs/2502.06786) find
  35. Aniruddha Nrusimha, Mayank Mishra, Naigang Wang, Dan Alistarh, Rameswar Panda, and Yoon Kim. 2024. Mitigating the impact of outlier channels for language model quantization with activation regularization. arXiv preprint arXiv:2404.03605. find
  36. NVIDIA. 2025. Introducing NVFP4 for efficient and accurate low-precision inference. find
  37. OpenAI. 2025. Gpt-5 system model card. find
  38. Andrei Panferov, Jiale Chen, Soroush Tabesh, Roberto L Castro, Mahdi Nikdan, and Dan Alistarh. 2025. Quest: Stable training of llms with 1-bit weights and activations. arXiv preprint arXiv:2502.05003. find
  39. Jungwoo Park, Taewhoo Lee, Chanwoong Yoon, Hyeon Hwang, and Jaewoo Kang. 2025. Outlier-safe pre-training for robust 4-bit quantization of large language models. arXiv preprint arXiv:2506.19697. find
  40. Yeonhong Park, Jake Hyun, SangLyul Cho, Bonggeun Sim, and Jae W Lee. 2024. Any-precision llm: Low-cost deployment of multiple, different-sized llms. arXiv preprint arXiv:2402.10517. — [arXiv:2402.10517](https://arxiv.org/abs/2402.10517) find
  41. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR.
  42. Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Cornea, Eric Dellinger, Kristof Denolf, and 1 others. 2023. Microscaling data formats for deep learning. arXiv preprint arXiv:2310.10537.
  43. Yuzhang Shang, Zhihang Yuan, Qiang Wu, and Zhen Dong. 2023. Pb-llm: Partially binarized large language models. arXiv preprint arXiv:2310.00034. — [arXiv:2310.00034](https://arxiv.org/abs/2310.00034) find
  44. Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. 2023. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308.13137.
  45. Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. 2024. Massive activations in large language models. arXiv preprint arXiv:2402.17762.
  46. Qitao Tan, Xiaoying Song, Jin Lu, Guoming Li, Jun Liu, Lingzi Hong, Caiwen Ding, Jundong Li, Xiaoming Zhai, Shaoyi Huang, and 1 others. 2025. Zeroqat: Your quantization-aware training but efficient. arXiv preprint arXiv:2509.00031. find
  47. MiniCPM Team, Chaojun Xiao, Yuxuan Li, Xu Han, Yuzhuo Bai, Jie Cai, Haotian Chen, Wentong Chen, Xin Cong, Ganqu Cui, and 1 others. 2025. Minicpm4: Ultra-efficient llms on end devices. arXiv preprint arXiv:2506.07900.
  48. Hongyu Wang, Shuming Ma, and Furu Wei. 2025. Bitnet v2: Native 4-bit activations with hadamard transformation for 1-bit llms. arXiv preprint arXiv:2504.18415. — [arXiv:2504.18415](https://arxiv.org/abs/2504.18415) find
  49. Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023a. SmoothQuant: Accurate and efficient post-training quantization for large language models. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Resear… find
  50. Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023b. Smoothquant: Accurate and efficient post-training quantization for large language models. In International conference on machine learning, pages 38087– 38099. PMLR. find
  51. Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Chris De Sa, and Zhiru Zhang. 2019. Improving neural network quantization without retraining using outlier channel splitting. In International conference on machine learning, pages 7543–7552. PMLR. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2026-bit-bit-progressive-qat-strategy/