Elamur Research⌕
☰
home / papers / 2025 / 大语言模型预训练系统关键技术综述

大语言模型预训练系统关键技术综述

Summary

This Chinese-language survey from Renmin University of China (Journal of Software) examines the systems engineering behind pre-training large language models. The core problem is that 10B–100B+ parameter models, trained for days to months on hundreds or thousands of accelerators, no longer fit the single-GPU or single-node computing model. The training pipeline is also prone to memory overflow and hardware or software faults.

The authors first explain autoregressive training and the Transformer workload. They then organise open-source pre-training frameworks into a four-stage evolution and propose a research framework with three axes:

  • Scalability: parallelism, recomputation/offloading, and MoE.
  • Performance: efficient attention, mixed precision, quantization-aware training, and communication optimisation.
  • Reliability: defect analysis, checkpointing, and elastic training.

For each technique, the survey compares strengths, weaknesses and suitable scenarios. It closes with open challenges: scarce compute, fast-moving research and engineering, and growing system complexity. Proposed directions include supporting multimodal models and using LLMs to help design and debug their own training systems. The survey is a useful map for teams choosing a training stack.

Key points

  • The four generations of systems are:
    • LM libraries: Transformers, LightSeq, MindNLP.
    • Single-strategy systems: Horovod, PyTorch DDP, GPipe, PyTorch FSDP.
    • Hybrid-parallel systems: Megatron-LM, DeepSpeed.
    • Automatic parallel-strategy search: Alpa.
  • A reference table of models gives sample training scales:
    • LLaMA: 1.4T tokens on 2,048 A100-80G GPUs.
    • Llama 2: 2T tokens.
    • PaLM: 780B tokens on 6,144 TPU-v4 chips.
    • GPT-3 175B used a 3.2M-token batch.
  • For checkpointing, Gemini measures cost as average wasted time, roughly checkpoint time + 1/(2f) + recovery time.
  • MegaScale checkpoints in two steps: a fast copy from GPU to pinned host memory, then an asynchronous upload to HDFS.
  • Elastic training is split into two approaches:
    • Deployment-layer control (TorchElastic, DLRover) understands the model and parallelism but is tied to specific systems.
    • Runtime instrumentation (Singularity, via CUDA/driver hooks) works with any framework but may back up redundant state.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Natural language processing

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.13328/j.cnki.jos.007438

@article{2025,
  title = {{大语言模型预训练系统关键技术综述}},
  author = {高彦杰 and 陈跃国},
  journal = {软件学报 (Journal of Software)},
  year = {2025},
  doi = {10.13328/j.cnki.jos.007438},
  url = {https://doi.org/10.13328/j.cnki.jos.007438}
}

References

109 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I. Attention is all you need. In: Proc. of the 31st Int’l Conf. on Neural Information Processing Systems. Long Beach: ACM, 2017. 6000–6010. find
  2. Brown TB, Mann B, Ryder N, et al. Language models are few-shot learners. In: Proc. of the 34th Int’l Conf. on Neural Information Processing Systems. Vancouver: ACM, 2020. 159. find
  3. Devlin J, Chang MW, Lee K, Toutanova K. BERT: Pre-Training of deep bidirectional Transformers for language understanding. In: Proc. of the 2019 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Minneapolis: ACL, 2019. 4171–4186. [doi:… find
  4. Henighan T, Kaplan J, Katz M, Chen M, Hesse C, Jackson J, Jun H, Brown TB, Dhariwal P, Gray S, Hallacy C, Mann B, Radford A, Ramesh A, Ryder N, Ziegler DM, Schulman J, Amodei D, McCandlish S. Scaling laws for autoregressive generative modeling. arXiv:2010.14701, 2020. find
  5. Le Scao T, Fan A, Akiki C, et al. BLOOM: A 176B-parameter open-access multilingual language model. arXiv:2211.05100, 2022. find
  6. Touvron H, Lavril T, Izacard G, Martinet X, Lachaux MA, Lacroix T, Rozière B, Goyal N, Hambro E, Azhar F, Rodriguez A, Joulin A, Grave E, Lample G. LLaMA: Open and efficient foundation language models. arXiv:2302.13971, 2023. find
  7. Touvron H, Martin L, Stone K, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288, 2023. find
  8. Black S, Biderman S, Hallahan E, Anthony Q, Gao L, Golding L, He H, Leahy C, McDonell K, Phang J, Pieler M, Prashanth US, Purohit S, Reynolds L, Tow J, Wang B, Weinbach S. GPT-NeoX-20B: An open-source autoregressive language model. In: Proc. of BigScience Episode\#5 —Workshop on Challenges & Perspect… find our card
  9. Zhang SS, Roller S, Goyal N, Artetxe M, Chen MY, Chen SH, Dewan C, Diab M, Li X, Lin XV, Mihaylov T, Ott M, Shleifer S, Shuster K, Simig D, Koura PS, Sridhar A, Wang TL, Zettlemoyer L. OPT: Open pre-trained Transformer language models. arXiv:2205.01068, 2022. find
  10. Zeng AH, Liu X, Du ZX, et al. GLM-130B: An open bilingual pre-trained model. arXiv:2210.02414, 2023 find our card
  11. Rae JW, Borgeaud S, Cai T, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv:2112.11446, 2021. find
  12. Thoppilan R, De Freitas D, Hall J, et al. LaMDA: Language models for dialog applications. arXiv:2201.08239, 2022. find our card
  13. Du N, Huang YP, Dai AM, et al. GLaM: Efficient scaling of language models with mixture-of-experts. In: Int’l Conf. on Machine Learning. Baltimore: ICML, 2022. 5547–5569. find our card
  14. Chowdhery A, Narang S, Devlin J, et al. PaLM: Scaling language modeling with pathways. The Journal of Machine Learning Research, 2023, 24(1): 240. find
  15. Anil R, Dai AM, Firat O, et al. PaLM 2 technical report. arXiv:2305.10403, 2023. find our card
  16. Zeng W, Ren XZ, Su T, et al. PanGu-α: Large-scale autoregressive pretrained Chinese language models with auto-parallel computation. arXiv:2104.12369, 2021. find
  17. Su H, Zhou X, Yu HJ, Shen XY, Chen YW, Zhu ZL, Yu Y, Zhou J. WeLM: A well-read pre-trained language model for Chinese. arXiv:2209.10372, 2022. — [arXiv:2209.10372](https://arxiv.org/abs/2209.10372) find
  18. Chung HW, Hou L, Longpre S, et al. Scaling instruction-finetuned language models. The Journal of Machine Learning Research, 2024, 25(1): 70. find
  19. The Mosaic Research Team. Introducing MPT-7B: A new standard for open-source, commercially usable LLMs. 2023. https://www.databricks.com/blog/mpt-7b ↗
  20. Ma ZX, Zhai JD, Han WT, Chen WG, Zheng WM. Challenges and measures for efficient training of trillion-parameter pre-trained models. ZTE Technology Journal, 2022, 28(2): 51–58 (in Chinese with English abstract). [doi: 10.12142/ZTETJ.202202008] — [doi:10.12142/ztetj.202202008](https://doi.org/10.12142/ztetj.202202008) find
  21. Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, Cistac P, Rault T, Louf R, Funtowicz M, Davison J, Shleifer S, von Platen P, Ma C, Jernite Y, Plu J, Xu CW, Le Scao T, Gugger S, Drame M, Lhoest Q, Rush AM. Transformers: State-of-the-art natural language processing. In: Proc. of the 2020 Conf.… find our card
  22. Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin ZM, Gimelshein N, Antiga L, Desmaison A, Köpf A, Yang E, DeVito Z, Raison M, Tejani A, Chilamkurthy S, Steiner B, Fang L, Bai JJ, Chintala S. PyTorch: An imperative style, high-performance deep learning library. In: Proc. of t… find our card
  23. Sergeev A, Del Balso M. Horovod: Fast and easy distributed deep learning in TensorFlow. arXiv:1802.05799, 2018. find
  24. Huang YP, Cheng YL, Bapna A, Firat O, Chen MX, Chen DH, Lee H, Ngiam J, Le QV, Wu YH, Chen ZF. GPipe: Efficient training of giant neural networks using pipeline parallelism. In: Proc. of the 33rd Int’l Conf. on Neural Information Processing Systems. Vancouver: ACM, 2019. 10. find
  25. Li S, Zhao YL, Varma R, Salpekar O, Noordhuis P, Li T, Paszke A, Smith J, Vaughan B, Damania P, Chintala S. PyTorch distributed: Experiences on accelerating data parallel training. Proc. of the VLDB Endowment, 2020, 13(12): 3005–3018. [doi: 10.14778/3415478.3415530] find our card
  26. Shoeybi M, Patwary M, Puri R, LeGresley P, Casper J, Catanzaro B. Megatron-LM: Training multi-billion parameter language models using model parallelism. arXiv:1909.08053, 2019. find
  27. Rasley J, Rajbhandari S, Ruwase O, He YX. DeepSpeed: System optimizations enable training deep learning models with over 100 billion parameters. In: Proc. of the 26th ACM SIGKDD Int’l Conf. on Knowledge Discovery & Data Mining. ACM, 2020. 3505–3506. [doi: 10.1145/3394486.3406703] — [doi:10.1145/3394486.3406703](https://doi.org/10.1145/3394486.3406703) find
  28. Zheng LM, Li ZH, Zhang H, Zhuang YH, Chen ZF, Huang YP, Wang YD, Xu YZ, Zhuo DY, Xing EP, Gonzalez JE, Stoica I. Alpa: Automating inter- and intra-operator parallelism for distributed deep learning. In: Proc. of the 16th USENIX Symp. on Operating Systems Design and Implementation. Carlsbad: USENIX,… find
  29. Abadi M, Barham P, Chen JM, Chen ZF, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M, Kudlur M, Levenberg J, Monga R, Moore S, Murray DG, Steiner B, Tucker PA, Vasudevan V, Warden P, Wicke M, Yu Y, Zheng XQ. TensorFlow: A system for large-scale machine learning. In: Proc. of the 12th USENIX… find
  30. Google. MaxText. GitHub [Internet]. 2023. https://github.com/google/maxtext/ ↗
  31. Bradbury J, Frostig R, Hawkins P, Johnson MJ, Leary C, Maclaurin D, Necula G, Paszke A, VanderPlas J, Wanderman-Milne S, Zhang Q. JAX: Composable transformations of Python+NumPy programs. 2018. http://github.com/google/jax ↗
  32. Li DC, Shao RL, Xie AZ, Xing EP, Gonzalez JE, Stoica I, Ma XZ, Zhang H. LightSeq: Sequence level parallelism for distributed training of long context Transformers. arXiv:2310.03294, 2023. [doi: 10.48550/arXiv.2310.03294] find
  33. MindNLP Contributors. MindNLP: Easy-to-use and high-performance NLP and LLM framework based on MindSpore [Internet]. 2022. https://github.com/mindlab-ai/mindnlp ↗
  34. Zhao YL, Gu A, Varma R, Luo L, Huang CC, Xu M, Wright L, Shojanazeri H, Ott M, Shleifer S, Desmaison A, Balioglu C, Damania P, Nguyen B, Chauhan G, Hao YC, Mathews A, Li S. PyTorch FSDP: Experiences on scaling fully sharded data parallel. Proc. of the VLDB Endowment, 2023, 16(12): 3848–3860. [doi: 1… find
  35. Nie XN, Liu Y, Fu FC, Xue JB, Jiao D, Miao XP, Tao YY, Cui B. Angel-PTM: A scalable and economical large-scale pre-training system in tencent. Proc. of the VLDB Endowment, 2023, 16(12): 3781–3794. [doi: 10.14778/3611540.3611564] find
  36. Baines M, Bhosale S, Caggiano V, Goyal N, Goyal S, Ott M, Lefaudeux B, Liptchinsky V, Rabbat M, Sheiffer S, Sridhar A. FairScale: A general purpose modular PyTorch library for high performance and large scale training. 2020. https://github.com/facebookresearch/fairscale ↗
  37. Mosaic ML Team. Composer. 2021. https://github.com/mosaicml/composer/ ↗
  38. Databricks. Databricks Mosaic: pioneering AI & open-source research. 2024. https://www.databricks.com/mosaic ↗
  39. Smith S, Patwary M, Norick B, LeGresley P, Rajbhandari S, Casper J, Liu Z, Prabhumoye S, Zerveas G, Korthikanti V, Zhang E, Child R, Aminabadi RY, Bernauer J, Song X, Shoeybi M, He YX, Houston M, Tiwary S, Catanzaro B. Using DeepSpeed and megatron to train megatron-turing NLG 530B, a large-scale gen… find our card
  40. Rajbhandari S, Rasley J, Ruwase O, He YX. ZeRO: Memory optimizations toward training trillion parameter models. In: Proc. of the Int’l Conf. for High Performance Computing, Networking, Storage and Analysis. Atlanta: ACM, 2020. 20. find
  41. Ren J, Rajbhandari S, Aminabadi RY, Ruwase O, Yang SY, Zhang MJ, Li D, He YX. ZeRO-offload: Democratizing billion-scale model training. In: Proc. of the 2021 USENIX Annual Technical Conf. USENIX, 2021. 551–564. find our card
  42. Aicheng Technology, Alibaba Group. Megatron-LLaMA [Internet]. 2023. https://github.com/alibaba/Megatron-LLaMA ↗
  43. Li SG, Liu HX, Bian ZD, Fang JR, Huang HC, Liu YL, Wang BX, You Y. Colossal-AI: A unified deep learning system for large-scale parallel training. In: Proc. of the 52nd Int’l Conf. on Parallel Processing. Salt Lake City: ACM, 2023. 766–775. [doi: 10.1145/3605573.3605613] find
  44. Jiang ZH, Lin HB, Zhong YM, et al. MegaScale: Scaling large language model training to more than 10 000 GPUs. In: Proc. of the 21st USENIX Symp. on Networked Systems Design and Implementation. Santa Clara: USENIX, 2024. 745–760. find
  45. PaddleNLP Contributors. PaddleNLP: An easy-to-use and high performance NLP library. 2024. https://github.com/PaddlePaddle/PaddleNLP ↗
  46. Lin ZQ, Miao YS, Zhang QL, Yang F, Zhu Y, Li C, Maleki S, Cao X, Shang N, Yang YL, Xu WJ, Yang M, Zhang LT, Zhou LD. nnScaler: Constraint-guided parallelization plan generation for deep learning training. In: Proc. of the 18th USENIX Symp. on Operating Systems Design and Implementation. Santa Clara:… find
  47. Patarasuk P, Yuan X. Bandwidth optimal All-Reduce algorithms for clusters of workstations. Journal of Parallel and Distrib Computing, 2009, 69(2): 117–124. [doi: 10.1016/j.jpdc.2008.09.002] — [doi:10.1016/j.jpdc.2008.09.002](https://doi.org/10.1016/j.jpdc.2008.09.002) find
  48. Ott M, Edunov S, Baevski A, Fan A, Gross S, Ng N, Grangier D, Auli M. fairseq: A fast, extensible toolkit for sequence modeling. In: Proc. of the 2019 Conf. of the North American Chapter of the Association for Computational Linguistics. Minneapolis: ACL, 2019. 48–53. [doi: 10.18653/v1/N19-4009] find our card
  49. Ma SM, Wang HY, Huang SH, Wang WH, Chi ZW, Dong L, Benhaim A, Patra B, Chaudhary V, Song X, Wei FR. TorchScale: Transformers at scale. arXiv:2211.13184, 2022. — [arXiv:2211.13184](https://arxiv.org/abs/2211.13184) find
  50. Narayanan D, Harlap A, Phanishayee A, Seshadri V, Devanur NR, Ganger GR, Gibbons PB, Zaharia M. PipeDream: Generalized pipeline parallelism for DNN training. In: Proc. of the 27th ACM Symp. on Operating Systems Principles. Huntsville: ACM, 2019. 1–15. [doi: 10.1145/3341301.3359646] — [doi:10.1145/3341301.3359646](https://doi.org/10.1145/3341301.3359646) find
  51. Narayanan D, Shoeybi M, Casper J, LeGresley P, Patwary M, Korthikanti V, Vainbrand D, Kashinkunti P, Bernauer J, Catanzaro B, Phanishayee A, Zaharia M. Efficient large-scale language model training on GPU clusters using megatron-LM. In: Proc. of the 2021 Int’l Conf. for High Performance Computing, N… find
  52. Li SG, Hoefler T. Chimera: Efficiently training large-scale neural networks with bidirectional pipelines. In: Proc. of the 2021 Int’l Conf. for High Performance Computing, Networking, Storage and Analysis. St. Louis: ACM, 2021. 27. [doi: 10.1145/3458817.3476145] find
  53. Choi S, Koo I, Ahn J, Jeon M, Kwon Y. EnvPipe: Performance-preserving DNN training framework for saving energy. In: Proc. of the 2023 USENIX Annual Technical Conf. Boston: USENIX, 2023. 851–864. find
  54. Osawa K, Li SG, Hoefler T. PipeFisher: Efficient training of large language models using pipelining and Fisher information matrices. In: Proc. of the 6th Conf. on Machine Learning and Systems. Miami: MLSys, 2023. 708–727. find
  55. Lu WY, Yan GH, Li JJ, Gong SJ, Han YH, Li XW. FlexFlow: A flexible dataflow accelerator architecture for convolutional neural networks. In: Proc. of the 2017 IEEE Int’l Symp. on High Performance Computer Architecture (HPCA). Austin: IEEE, 2017. 553–564. IEEE. [doi: 10.1109/HPCA.2017.29] — [doi:10.1109/hpca.2017.29](https://doi.org/10.1109/hpca.2017.29) find
  56. Beaumont O, Eyraud-Dubois L, Herrmann J, Joly A, Shilova A. Optimal re-materialization strategies for heterogeneous chains: How to train deep neural networks with limited memory. ACM Trans. on Mathematical Software, 2024, 50(2): 10. [doi: 10.1145/3648633] — [doi:10.1145/3648633](https://doi.org/10.1145/3648633) find
  57. Korthikanti VA, Casper J, Lym S, McAfee L, Andersch M, Shoeybi M, Catanzaro B. Reducing activation recomputation in large Transformer models. In: Proc. of the 6th Conf. on Machine Learning and Systems. Miami: MLSys, 2023. find our card
  58. Rajbhandari S, Ruwase O, Rasley J, Smith S, He YX. ZeRO-infinity: Breaking the GPU memory wall for extreme scale deep learning. In: Proc. of the 2021 Int’l Conf. for High Performance Computing, Networking, Storage and Analysis. St. Louis: ACM, 2021. 59. [doi: 10.1145/3458817.3476205] find
  59. Yuan TL, Liu YL, Ye XC, Zhang SL, Tan JC, Chen B, Song CR, Zhang D. Accelerating the training of large language models using efficient activation rematerialization and optimal hybrid parallelism. In: Proc. of the 2024 USENIX Annual Technical Conf. Santa Clara: USENIX, 2024. 545–561. find
  60. Yuksel SE, Wilson JN, Gader PD. Twenty years of mixture of experts. IEEE Trans. on Neural Networks and Learning Systems, 2012, 23(8): 1177–1193. [doi: 10.1109/TNNLS.2012.2200299] — [doi:10.1109/tnnls.2012.2200299](https://doi.org/10.1109/tnnls.2012.2200299) find
  61. Hwang C, Cui W, Xiong YF, Yang ZY, Liu Z, Hu H, Wang ZL, Salas R, Jose J, Ram P, Chau H, Cheng P, Yang F, Yang M, Xiong YQ. Tutel: Adaptive mixture-of-experts at scale. In: Proc. of the 6th Conf. on Machine Learning and Systems. Miami: MLSys, 2023. find
  62. Li JM, Jiang YM, Zhu YB, Wang C, Xu H. Accelerating distributed MoE training and inference with lina. In: Proc. of the 2023 USENIX Annual Technical Conf. Boston: USENIX, 2023. 945–959. find
  63. Zhai MS, He JA, Ma ZX, Zong Z, Zhang RQ, Zhai JD. SmartMoE: Efficiently training sparsely-activated models through combining offline and online parallelization. In: Proc. of the 2023 USENIX Annual Technical Conf. Boston: USENIX, 2023. 961–975. find
  64. Ivanov A, Dryden N, Ben-Nun T, Li SG, Hoefler T. Data movement is all you need: A case study on optimizing Transformers. In: Proc. of the 4th Conf. on Machine Learning and Systems. MLSys, 2021. 711–732. find
  65. Williams S, Waterman A, Patterson D. Roofline: An insightful visual performance model for multicore architectures. Communications of the ACM, 2009, 52(4): 65–76. [doi: 10.1145/1498765.1498785] find
  66. Dao T, Fu DY, Ermon S, Rudra A, Ré C. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In: Proc. of the 36th Int’l Conf. on Neural Information Processing Systems. New Orleans: ACM, 2022. 1189. find
  67. Dao T. FlashAttention-2: Faster attention with better parallelism and work partitioning. In: Proc. of the 12th Int’l Conf. on Learning Representations. Vienna: ICLR, 2024. find
  68. Beltagy I, Peters ME, Cohan A. Longformer: The long-document Transformer. arXiv:2004.05150, 2020. find
  69. Kitaev N, Kaiser Ł, Levskaya A. Reformer: The efficient Transformer. In: Proc. of the 8th Int’l Conf. on Learning Representations. Addis Ababa: ICLR, 2020. find our card
  70. Wang SN, Li BZ, Khabsa M, Fang H, Ma H. Linformer: Self-attention with linear complexity. arXiv:2006.04768, 2020. find
  71. Zhu C, Ping W, Xiao CW, Shoeybi M, Goldstein T, Anandkumar A, Catanzaro B. Long-short Transformer: Efficient Transformers for language and vision. In: Proc. of the 35th Int’l Conf. on Neural Information Processing Systems. ACM, 2021. 17723–17736. find
  72. Micikevicius P, Narang S, Alben J, Diamos G, Elsen E, Garcia D, Ginsburg B, Houston M, Kuchaiev O, Venkatesh G, Wu H. Mixed precision training. arXiv:1710.03740. 2017. find
  73. Liu ZC, Oguz B, Zhao CS, Chang E, Stock P, Mehdad Y, Shi YY, Krishnamoorthi R, Chandra V. LLM-QAT: Data-Free quantization aware training for large language models. In: Proc. of the Findings of the Association for Computational Linguistics. Bangkok: ACL, 2024. 467–484. [doi: 10.18653/v1/2024.findings… find our card
  74. Arshia FZ, Keyvanrad MA, Sadidpour SS, Mohammadi SMR. PeQA: A massive Persian question-answering and chatbot dataset. In: Proc. of the 12th Int’l Conf. on Computer and Knowledge Engineering (ICCKE). Mashhad: IEEE, 2022. 392–397. [doi: 10.1109/ICCKE57176.2022.9960071] — [doi:10.1109/iccke57176.2022.9960071](https://doi.org/10.1109/iccke57176.2022.9960071) find
  75. Dettmers T, Pagnoni A, Holtzman A, Zettlemoyer L. QLoRA: Efficient finetuning of quantized LLMs. In: Proc. of the 37th Int’l Conf. on Neural Information Processing Systems. New Orleans: ACM, 2023. 441. find
  76. Wang WY, Khazraee M, Zhong ZZ, Ghobadi M, Jia ZH, Mudigere D, Zhang Y, Kewitsch A. TopoOpt: Co-Optimizing network topology and parallelization strategy for distributed training jobs. In: Proc. of the 20th USENIX Symp. on Networked Systems Design and Implementation. Boston: USENIX, 2023. 739–767. find
  77. Cowan M, Maleki S, Musuvathi M, Saarikivi O, Xiong YF. MSCCLang: Microsoft collective communication language. In: Proc. of the 28th ACM Int’l Conf. on Architectural Support for Programming Languages and Operating Systems. Vancouver: ACM, 2023. 502–514. [doi: 10.1145/3575693.3575724] — [doi:10.1145/3575693.3575724](https://doi.org/10.1145/3575693.3575724) find
  78. Cho M, Finkler U, Kung DS, Hunter HC. BlueConnect: Decomposing all-reduce for deep learning on heterogeneous network hierarchy. In: Proc. of the 2nd Conf. on Machine Learning and Systems. Stanford: SysML, 2019. 241–251. find
  79. Luo L, West P, Krishnamurthy A, Ceze L, Nelson J. PLink: Discovering and exploiting locality for accelerated distributed training on the public cloud. In: Proc. of the 3rd Conf. on Machine Learning and Systems. Austin: SysML, 2020. 82–97. find
  80. Wang GH, Venkataraman S, Phanishayee A, Thelin J, Devanur NR, Stoica I. Blink: Fast and generic collectives for distributed ml. In: Proc. of the 3rd Conf. on Machine Learning and Systems. Austin: SysML, 2020. 172–186. find
  81. Cai ZX, Liu ZY, Maleki S, Musuvathi M, Mytkowicz T, Nelson J, Saarikivi O. Synthesizing optimal collective algorithms. In: Proc. of the 26th ACM SIGPLAN Symp. on Principles and Practice of Parallel Programming. ACM, 2021. 62–75. [doi: 10.1145/3437801.3441620] — [doi:10.1145/3437801.3441620](https://doi.org/10.1145/3437801.3441620) find
  82. Zhuang HP, Wang Y, Liu QL, Lin ZP. Fully decoupled neural network learning using delayed gradients. IEEE Trans. on Neural Networks and Learning Systems, 2022, 33(10): 6013–6020. [doi: 10.1109/TNNLS.2021.3069883] — [doi:10.1109/tnnls.2021.3069883](https://doi.org/10.1109/tnnls.2021.3069883) find
  83. Hashemi SH, Abdu Jyothi S, Campbell RH. TicTac: Accelerating distributed deep learning with communication scheduling. In: Proc. of the 2nd Conf. on Machine Learning and Systems. Stanford: SysML, 2019. 418–430. find
  84. Romero J, Yin JQ, Laanait N, Xie B, Young MT, Treichler S, Starchenko V, Borisevich AY, Sergeev A, Matheson MA. Accelerating collective communication in data parallel training across deep learning frameworks. In: Proc. of the 19th USENIX Symp. on Networked Systems Design and Implementation. Renton:… find
  85. Faghri F, Tabrizian I, Markov I, Alistarh D, Roy DM, Ramezani-Kebrya A. Adaptive gradient quantization for data-parallel SGD. In: Proc. of the 34th Int’l Conf. on Neural Information Processing Systems. Vancouver: ACM, 2020. 267. find
  86. Bian S, Li DC, Wang HY, Xing EP, Venkataraman S. Does compressing activations help model parallel training? In: Proc. of the 7th Annual Conf. on Machine Learning and Systems. Santa Clara: MLSys, 2024. 239–252. find
  87. Bayatpour M, Sarkauskas N, Subramoni H, Hashmi JM, Panda DK. BluesMPI: Efficient MPI non-blocking alltoall offloading designs on modern BlueField smart NICs. In: Proc. of the 36th Int’l Conf. on High Performance Computing. Virtual Event: Springer, 2021. 18–37. [doi: 10.1007/978-3-030-78713-4_2] — [doi:10.1007/978-3-030-78713-4_2](https://doi.org/10.1007/978-3-030-78713-4_2) find
  88. Dong JB, Wang SC, Feng F, Cao Z, Pan H, Tang LB, Li PC, Li H, Ran QY, Guo YQ, Gao SY, Long X, Zhang J, Li Y, Xia ZS, Song LYH, Zhang YY, Pan P, Wang GH, Jiang XW. ACCL: Architecting highly scalable distributed training systems with highly efficient collective communication library. IEEE Micro, 2021,… — [doi:10.1109/mm.2021.3091475](https://doi.org/10.1109/mm.2021.3091475) find
  89. Jiang YM, Zhu YB, Lan C, Yi BR, Cui Y, Guo CX. A unified architecture for accelerating distributed DNN training in heterogeneous GPU/CPU clusters. In: Proc. of the 14th USENIX Symp. on Operating Systems Design and Implementation. USENIX, 2020. 463–479. find
  90. Sapio A, Canini M, Ho CY, Nelson J, Kalnis P, Kim C, Krishnamurthy A, Moshref M, Ports DRK, Richtárik P. Scaling distributed machine learning with in-network aggregation. In: Proc. of the 18th USENIX Symp. on Networked Systems Design and Implementation. USENIX, 2021. 785–808. find
  91. Lao C, Le YF, Mahajan K, Chen YX, Wu WF, Akella A, Swift MM. ATP: In-network aggregation for multi-tenant learning. In: Proc. of the 18th USENIX Symp. on Networked Systems Design and Implementation. USENIX, 2021. 741–761. find
  92. Zhang R, Xiao WC, Zhang HY, Liu Y, Lin HX, Yang M. An empirical study on program failures of deep learning jobs. In: Proc. of the 42nd Int’l Conf. on Software Engineering. Seoul: ACM, 2020. 1159–1170. [doi: 10.1145/3377811.3380362] — [doi:10.1145/3377811.3380362](https://doi.org/10.1145/3377811.3380362) find
  93. Gao YJ, Shi XX, Lin HX, Zhang HY, Wu H, Li R, Yang M. An empirical study on quality issues of deep learning platform. In: Proc. of the 45th Int’l Conf. on Software Engineering: Software Engineering in Practice. Melbourne: IEEE, 2023. 455–466. [doi: 10.1109/ICSE-SEIP58684.2023.00052] — [doi:10.1109/icse-seip58684.2023.00052](https://doi.org/10.1109/icse-seip58684.2023.00052) find
  94. Gao YJ, Li ZX, Lin HX, Zhang HY, Wu M, Yang M. Refty: Refinement types for valid deep learning models. In: Proc. of the 44th Int’l Conf. on Software Engineering. Pittsburgh: ACM, 2022. 1843–1855. [doi: 10.1145/3510003.3510077] — [doi:10.1145/3510003.3510077](https://doi.org/10.1145/3510003.3510077) find
  95. Gao YJ, Gu XY, Zhang HY, Lin HX, Yang M. Runtime performance prediction for deep learning models with graph neural network. In: Proc. of the 45th IEEE/ACM Int’l Conf. on Software Engineering: Software Engineering in Practice. Melbourne: IEEE, 2023. 368–380. [doi: 10.1109/ICSE-SEIP58684.2023.00039] — [doi:10.1109/icse-seip58684.2023.00039](https://doi.org/10.1109/icse-seip58684.2023.00039) find
  96. Mei HQ, Qu HZ, Sun JW, Gao YJ, Lin HX, Sun GZ. GPU occupancy prediction of deep learning models using graph neural network. In: Proc. of the 25th IEEE Int’l Conf. on Cluster Computing. Santa Fe: IEEE, 2023. 318–329. [doi: 10.1109/CLUSTER52292.2023.00034] — [doi:10.1109/cluster52292.2023.00034](https://doi.org/10.1109/cluster52292.2023.00034) find
  97. Zhu HY, Phanishayee A, Pekhimenko G. Daydream: Accurately estimating the efficacy of optimizations for DNN training. In: Proc. of the 2020 USENIX Annual Technical Conf. USENIX, 2020. 337–352. find
  98. OpenAI. Scaling Kubernetes to 7 500 nodes. 2021. https://openai.com/index/scaling-kubernetes-to-7500-nodes/ ↗
  99. Xiong YF, Jiang YT, Yang ZY, Qu L, Zhao GS, Liu SG, Zhong D, Pinzur B, Zhang J, Wang Y, Jose J, Pourreza H, Baxter J, Datta K, Ram P, Melton L, Chau J, Cheng P, Xiong YQ, Zhou LD. SuperBench: Improving cloud AI infrastructure reliability with proactive validation. In: Proc. of the 2024 USENIX Annual… find
  100. Rojas E, Kahira AN, Meneses E, Gomez LB, Badia RM. A study of checkpointing in large scale training of deep neural networks. arXiv:2012.00825, 2020. — [arXiv:2012.00825](https://arxiv.org/abs/2012.00825) find
  101. Wang Z, Jia Z, Zheng S, Zhang Z, Fu XW, Ng TSE, Wang YD. Gemini: Fast failure recovery in distributed training with in-memory checkpoints. In: Proc. of the 29th Symp. on Operating Systems Principles. Koblenz: ACM, 2023. 364–381. [doi: 10.1145/3600006.3613145] — [doi:10.1145/3600006.3613145](https://doi.org/10.1145/3600006.3613145) find
  102. Nicolae B, Li JL, Wozniak JM, Bosilca G, Dorier M, Cappello F. DeepFreeze: Towards scalable asynchronous checkpointing of deep learning models. In: Proc. of the 20th IEEE/ACM Int’l Symp. on Cluster, Cloud and Internet Computing. Melbourne: IEEE, 2020. 172–181. [doi: 10.1109/CCGrid49817.2020.00-76] — [doi:10.1109/ccgrid49817.2020.00-76](https://doi.org/10.1109/ccgrid49817.2020.00-76) find
  103. Mohan J, Phanishayee A, Chidambaram V. CheckFreq: Frequent, fine-grained DNN checkpointing. In: Proc. of the 19th USENIX Conf. on File and Storage Technologies. USENIX, 2021. 203–216. find
  104. Eisenman A, Matam KK, Ingram S, Mudigere D, Krishnamoorthi R, Nair K, Smelyanskiy M, Annavaram M. Check-N-Run: A checkpointing system for training deep learning recommendation models. In: Proc. of the 19th USENIX Symp. on Networked Systems Design and Implementation. Renton: USENIX, 2022. 929–943. find
  105. Jang I, Yang ZN, Zhang Z, Jin X, Chowdhury M. Oobleck: Resilient distributed training of large models using pipeline templates. In: Proc. of the 29th Symp. on Operating Systems Principles. Koblenz: ACM, 2023. 382–395. [doi: 10.1145/3600006.3613152] — [doi:10.1145/3600006.3613152](https://doi.org/10.1145/3600006.3613152) find
  106. PyTorch. Elastic. 2024. https://github.com/pytorch/elastic ↗
  107. Wang QL, Sang B, Zhang HT, Tang MJ, Zhang K. DLRover: An elastic deep training extension with auto job resource recommendation. arXiv:2304.01468, 2023. find
  108. Xiao WC, Ren SR, Li Y, Zhang Y, Hou PY, Li Z, Feng YH, Lin W, Jia YQ. AntMan: Dynamic scaling on GPU clusters for deep learning. In: Proc. of the 14th USENIX Symp. on Operating Systems Design and Implementation. USENIX, 2020. 533–548. find
  109. Shukla D, Sivathanu M, Viswanatha S, et al. Singularity: Planet-scale, preemptive and elastic scheduling of AI workloads. arXiv:2202.07848, 2022. 附中文参考文献 [20] 马子轩, 翟季冬, 韩文弢, 陈文光, 郑纬民. 高效训练百万亿参数预训练模型的系统挑战和对策. 中兴通讯技术, 2022, 28(2): 51–58. [doi: 10.12142/ZTETJ.202202008] 作者简介 高彦杰, 博士生, CCF 专业会员, 主要研究领域为… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gao-2025-key-technologies-large/