Elamur Research⌕
☰
home / papers / 2024 / FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillation

FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillation

Paper · 2024 · Liqun Ma, Mingjie Sun and Zhiqiang Shen

Summary

Binarizing LLMs down to pure 1-bit weights could greatly cut memory and compute. Earlier approaches fall short of that limit. Some keep salient weights at higher precision (PB-LLM, BiLLM), and BitNet b1.58 uses ternary values. Others start from a pretrained full-precision model, which locks in its size and vocabulary and leaves little data to relearn lost knowledge. The authors present FBI-LLM, a recipe for pretraining fully binarized transformer LLMs from random initialization. Every linear layer except the causal head becomes an FBI-Linear module, which pairs sign-binarized weights with learnable per-column scale and shift vectors and is trained via the straight-through estimator. Embeddings and normalization layers stay in full precision. The only training objective is autoregressive distillation: at each token position, the binary student imitates the output distribution of a full-precision teacher. The authors trained models with 130M, 1.3B and 7B parameters on a data budget comparable to ordinary pretraining. These models approach their full-precision counterparts and beat earlier binarized LLMs on perplexity and downstream tasks. The result suggests that 1-bit LLMs can be built natively, which may encourage dedicated hardware.

Key points

  • Architecture: FBI-Linear stores a {−1, 1} weight matrix plus column-wise α and β parameters. α starts as each column's mean, and β starts as that column's mean absolute deviation from the mean.
  • Training: a single autoregressive distillation loss against a full-precision teacher. Because only the loss changes, the method plugs into standard LLM pretraining pipelines.
  • Results: at comparable model sizes, FBI-LLM reaches similar or lower Wikitext2 perplexity than BiLLM, OneBit and BitNet b1.58. Generation examples show fluent output that follows prompts better than BitNet b1.58.
  • Analysis: flip-flop ratio and gradient-norm analysis of training checkpoints shows no major benefit from inheriting pretrained weights compared with training from scratch.
  • Limitations and release: activations are not binarized, and current hardware gives no real speedup yet. Distillation also adds training cost. Code is released at github.com/LiqunMa/FBI-LLM, models at huggingface.co/LiqunMa, and the training data is released as well.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Liqun Ma, Mingjie Sun and Zhiqiang Shen. 2024. FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillation. arXiv:2407.07093.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2407.07093 · arXiv 2407.07093

@misc{ma2024fbi,
  title = {{FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillation}},
  author = {Liqun Ma, Mingjie Sun and Zhiqiang Shen},
  year = {2024},
  doi = {10.48550/arxiv.2407.07093},
  eprint = {2407.07093},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.07093}
}

References

38 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei. The era of 1-bit llms: All large language models are in 1.58 bits. arXiv preprint arXiv:2402.17764, 2024. find
  2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  3. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
  4. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  5. Yuzhang Shang, Zhihang Yuan, and Zhen Dong. Pb-llm: Partially binarized large language models. In The Twelfth International Conference on Learning Representations, 2023. find
  6. Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning. ArXiv, abs/2208.11580, 2022. — [arXiv:2208.11580](https://arxiv.org/abs/2208.11580) find
  7. Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. Onebit: Towards extremely low-bit large language models. arXiv preprint arXiv:2402.11295, 2024. — [arXiv:2402.11295](https://arxiv.org/abs/2402.11295) find
  8. Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. Advances in neural information processing systems, 28, 2015. find
  9. Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks. Advances in neural information processing systems, 29, 2016. find
  10. Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference on computer vision, pages 525–542. Springer, 2016. find
  11. Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160, 2016. find
  12. Asit Mishra, Eriko Nurvitadhi, Jeffrey J Cook, and Debbie Marr. Wrpn: Wide reduced-precision networks. arXiv preprint arXiv:1709.01134, 2017. — [arXiv:1709.01134](https://arxiv.org/abs/1709.01134) find
  13. Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural networks. arXiv preprint arXiv:1711.11294, 2017. — [arXiv:1711.11294](https://arxiv.org/abs/1711.11294) find
  14. Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantization. arXiv preprint arXiv:1802.05668, 2018. — [arXiv:1802.05668](https://arxiv.org/abs/1802.05668) find
  15. Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013. find
  16. Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang-Ting Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16, pages 143–159. Springer, 2020. find
  17. Sajad Darabi, Mouloud Belbahri, Matthieu Courbariaux, and Vahid Partovi Nia. BNN+: Improved binary network training, 2019. find
  18. Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, and Xiaojuan Qi. Billm: Pushing the limit of post-training quantization for llms. arXiv preprint arXiv:2402.04291, 2024. find
  19. Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, and Furu Wei. Bitnet: Scaling 1-bit transformers for large language models. arXiv preprint arXiv:2310.11453, 2023. find
  20. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018. find
  21. Jangho Kim, Yash Bhalgat, Jinwon Lee, Chirag Patel, and Nojun Kwak. Qkd: Quantization-aware knowledge distillation. arXiv preprint arXiv:1911.12491, 2019. — [arXiv:1911.12491](https://arxiv.org/abs/1911.12491) find
  22. Yoonho Boo, Sungho Shin, Jungwook Choi, and Wonyong Sung. Stochastic precision ensemble: self-knowledge distillation for quantized deep neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6794–6802, 2021. find
  23. Cuong Pham, Tuan Hoang, and Thanh-Toan Do. Collaborative multi-teacher knowledge distillation for learning low bit-width deep neural networks. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6435–6443, 2023. find
  24. Kartikeya Bhardwaj, Nilesh Prasad Pandey, Sweta Priyadarshi, Kyunggeun Lee, Jun Ma, and Harris Teague. Oh! we freeze: Improving quantized knowledge distillation via signal propagation analysis for large language models. arXiv preprint arXiv:2403.18159, 2024. find
  25. Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, et al. Llm360: Towards fully transparent open-source llms. arXiv preprint arXiv:2312.06550, 2023. find
  26. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116… find
  27. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161, 2023. find
  28. Together Computer. Redpajama: An open source recipe to reproduce llama training dataset, 2023. find
  29. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
  30. Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385, 2024.
  31. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. ArXiv, abs/1905.10044, 2019.
  32. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020.
  33. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Annual Meeting of the Association for Computational Linguistics, 2019. find
  34. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021. find
  35. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.05457, 2018.
  36. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Conference on Empirical Methods in Natural Language Processing, 2018. find
  37. Zechun Liu, Zhiqiang Shen, Shichao Li, Koen Helwegen, Dong Huang, and Kwang-Ting Cheng. How do adam and training strategies help bnns optimization. In International conference on machine learning, pages 6936–6946. PMLR, 2021. find
  38. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1– 113, 2023.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ma-2024-fbi-llm-scaling-up-fully/