Elamur Research⌕
☰
home / papers / 2024 / Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization

Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization

Paper · 2024 · Mohammad Samragh, Iman Mirzadeh, Keivan Alizadeh Vahid et al.

Summary

Training large language models from random weights is slow and costly, while small models are cheap but less accurate. The authors, from Apple, propose HyperCloning, which builds a wider model directly from a smaller pretrained one. The number of layers stays the same, but the hidden dimension grows n-fold. Every hidden vector of the large model is a stacked copy of the small model's vector, and the copied weight blocks are scaled by 1/n. As a result, the enlarged network gives exactly the same logits as its source before training begins. Only initialization changes; the training loop stays standard.

The method was tested on three open model families: OPT, Pythia and OLMO, with source checkpoints downloaded from HuggingFace. All hyperparameters and the data order matched those of randomly initialized baselines. HyperCloning reached the baselines' final accuracy 2.2–4× faster and ended with higher accuracy. This shows that existing small checkpoints can substantially cut the GPU budget needed to pre-train bigger models.

Key points

  • Three source→target pairs were tested: OPT-350M→OPT-1.3B, Pythia-460M→Pythia-1.4B and OLMO-1B→OLMO-2.9B. Accuracy was averaged over 10 tasks using the Harness evaluation framework.
  • The OLMO source model had already seen 2.4T tokens. Its cloned 2.9B successor surpassed the random-init baseline after roughly 250B training tokens.
  • Cloned models showed catastrophic forgetting early in training, most visibly for OLMO. Continued training recovered from it, and the method still beat random initialization.
  • Duplicated weights initially have a cosine similarity of 1, and this similarity decays during training, likely helped by dropout. A function-preserving noise term that deliberately breaks the symmetry sped up convergence. Diagonal-style expansion (Shen et al.) converged more slowly.
  • For a 5.3B OPT target, 2-fold cloning from OPT-1.3B outperformed 4-fold cloning from OPT-350M. Both outperformed random initialization.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢AppleCupertino · United States20 works · 2,146 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Mohammad Samragh, Iman Mirzadeh, Keivan Alizadeh Vahid, Fartash Faghri, Minsik Cho, Moin Nabi, Devang Naik and Mehrdad Farajtabar. 2024. Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization. arXiv:2409.12903.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2409.12903 · arXiv 2409.12903

@misc{samragh2024scaling,
  title = {{Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization}},
  author = {Mohammad Samragh, Iman Mirzadeh, Keivan Alizadeh Vahid, Fartash Faghri, Minsik Cho, Moin Nabi, Devang Naik and Mehrdad Farajtabar},
  year = {2024},
  doi = {10.48550/arxiv.2409.12903},
  eprint = {2409.12903},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2409.12903}
}

References

26 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
  2. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021. find
  3. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.
  4. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on M… find
  5. Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, and Pablo Villalobos. Compute trends across three eras of machine learning. In 2022 International Joint Conference on Neural Networks (IJCNN), pages 1–8. find
  6. Ben Cottier, Robi Rahman, Loredana Fattorini, Nestor Maslej, and David Owen. The rising costs of training frontier ai models. arXiv preprint arXiv:2405.21015, 2024. — [arXiv:2405.21015](https://arxiv.org/abs/2405.21015) find
  7. Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the International… find
  8. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
  9. Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. arXiv preprint arXiv:1511.05641, 2015. — [arXiv:1511.05641](https://arxiv.org/abs/1511.05641) find
  10. Wenyu Du, Tongxu Luo, Zihan Qiu, Zeyu Huang, Yikang Shen, Reynold Cheng, Yike Guo, and Jie Fu. Stacking your transformers: A closer look at model growth for efficient llm pre-training. arXiv preprint arXiv:2405.15319, 2024. — [arXiv:2405.15319](https://arxiv.org/abs/2405.15319) find
  11. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models, 2022. URL https://arxiv.org/abs/2205.01068, 3:19–0, 2023. ↗
  12. Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Olmo: Accelerating the science of language models. arXiv preprint arXiv:2402.00838, 2024. find
  13. Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tieyan Liu. Efficient training of bert by progressively stacking. In International conference on machine learning, pages 2337–2346. PMLR, 2019. find
  14. Mohammad Samragh, Mehrdad Farajtabar, Sachin Mehta, Raviteja Vemulapalli, Fartash Faghri, Devang Naik, Oncel Tuzel, and Mohammad Rastegari. Weight subcloning: direct initialization of transformers using larger pretrained ones. arXiv preprint arXiv:2312.09299, 2023. — [arXiv:2312.09299](https://arxiv.org/abs/2312.09299) find
  15. Cheng Yang, Shengnan Wang, Chao Yang, Yuechuan Li, Ru He, and Jingqiao Zhang. Progressively stacking 2.0: A multi-stage layerwise training method for bert training speedup. arXiv preprint arXiv:2011.13635, 2020. — [arXiv:2011.13635](https://arxiv.org/abs/2011.13635) find
  16. Stefani Karp, Nikunj Saunshi, Sobhan Miryoosefi, Sashank J Reddi, and Sanjiv Kumar. Landscape-aware growing: The power of a little lag. arXiv preprint arXiv:2406.02469, 2024. — [arXiv:2406.02469](https://arxiv.org/abs/2406.02469) find
  17. Xiang Li, Yiqun Yao, Xin Jiang, Xuezhi Fang, Xuying Meng, Siqi Fan, Peng Han, Jing Li, Li Du, Bowen Qin, et al. Flm-101b: An open llm and how to train it with 100 k budget. arXiv preprint arXiv:2309.03852, 2023. — [arXiv:2309.03852](https://arxiv.org/abs/2309.03852) find
  18. Peihao Wang, Rameswar Panda, Lucas Torroba Hennigen, Philip Greengard, Leonid Karlinsky, Rogerio Feris, David Daniel Cox, Zhangyang Wang, and Yoon Kim. Learning to grow pretrained models for efficient transformer training. arXiv preprint arXiv:2303.00980, 2023a. find
  19. Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402.13116, 2024. — [arXiv:2402.13116](https://arxiv.org/abs/2402.13116) find
  20. Ming Zhong, Chenxin An, Weizhu Chen, Jiawei Han, and Pengcheng He. Seeking neural nuggets: Knowledge transfer in large language models from a parametric perspective. arXiv preprint arXiv:2310.11451, 2023. — [arXiv:2310.11451](https://arxiv.org/abs/2310.11451) find
  21. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Ani… find
  22. Yite Wang, Jiahao Su, Hanlin Lu, Cong Xie, Tianyi Liu, Jianbo Yuan, Haibin Lin, Ruoyu Sun, and Hongxia Yang. Lemon: Lossless model expansion. arXiv preprint arXiv:2310.07999, 2023b. — [arXiv:2310.07999](https://arxiv.org/abs/2310.07999) find
  23. Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256. JMLR Workshop and Conference Proceedings, 2010. find
  24. Cheng Chen, Yichun Yin, Lifeng Shang, Xin Jiang, Yujia Qin, Fengyu Wang, Zhi Wang, Xiao Chen, Zhiyuan Liu, and Qun Liu. bert2bert: Towards reusable pretrained language models. arXiv preprint arXiv:2110.07143, 2021. find
  25. Sheng Shen, Pete Walsh, Kurt Keutzer, Jesse Dodge, Matthew Peters, and Iz Beltagy. Staged training for transformer language models. In International Conference on Machine Learning, pages 19893–19908. PMLR, 2022. find
  26. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/samragh-2024-scaling-smart-accelerating-large-language/