Elamur Research⌕
☰
home / papers / 2024 / Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning

Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning

Paper · 2024 · Wei An, Xiao Bi, Guanting Chen et al.

Summary

Compute and bandwidth needs for deep learning, especially LLMs, are rising much faster than hardware improves. Fast chips and interconnects make large clusters costly to build and run. DeepSeek-AI presents Fire-Flyer AI-HPC, a combined hardware and software design for building such infrastructure cheaply.

The hardware is Fire-Flyer 2, a cluster of 10,000 PCIe A100 GPUs. A two-layer Fat-Tree network carries both storage and compute traffic and is split into two zones. PCIe nodes are weaker than DGX-style systems, so the authors make up the gap in software:

  • HFReduce runs allreduce asynchronously on the CPU so communication overlaps with computation.
  • HaiScale adapts the usual parallelism schemes to the PCIe setup.
  • 3FS removes storage I/O bottlenecks.
  • HAI-Platform handles scheduling, fault handling and recovery.
  • Network tuning keeps the shared fabric free of congestion.

The resulting cluster performs close to DGX-A100 at roughly half the cost and 40% lower energy use. The paper matters as a practical, documented recipe for affordable large-scale training, backed by a year of real hardware-failure statistics.

Key points

  • Fire-Flyer 2 uses 10,000 PCIe A100 GPUs and approaches DGX-A100 performance at about 50% of the cost and 40% less energy.
  • HFReduce, a CPU-based asynchronous allreduce, outperforms NCCL on this PCIe architecture.
  • The software stack includes HaiScale (DP, PP, TP, EP, FSDP, ZeRO), the 3FS distributed file system and HAI-Platform, which is open-sourced at github.com/HFAiLab/hai-platform.
  • Stability tooling includes a checkpoint manager for disaster recovery and a validator utility for detecting faulty hardware.
  • One year of GPU Xid error data is reported:
    • NVLink errors (xid 74): 42.57%
    • xid 43: 33.48%
    • xid 31: 19.18%
  • A proposed next-generation design for MoE LLM training uses multiple NICs and a Multi-Plane network.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢DeepSeekHangzhou · China9 works · 7,942 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksInformation Systems · 1,437 worksHardware and Architecture · 81 worksComputer Networks and Communications · 64 works

🧭 Topics 1Research subjects and languages the work is about.GPU & parallel computing · 19 works

Cite

Cite as Wei An, Xiao Bi, Guanting Chen, Shanhuang Chen, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Wenjun Gao, Kang Guan, Jianzhong Guo, Yongqiang Guo, Zhe Fu, Ying He, Panpan Huang, Jiashi Li, Wenfeng Liang, Xiaodong Liu, Xin Liu, Yiyuan Liu, Yuxuan Liu, Shanghao Lu, Xuan Lu, Xiaotao Nie, Tian Pei, Junjie Qiu, Hui Qu, Zehui Ren, Zhangli Sha, Xuecheng Su, Xiaowen Sun, Yixuan Tan, Minghui Tang, Shiyu Wang, Yaohui Wang, Yongji Wang, Ziwei Xie, Yiliang Xiong, Yanhong Xu, Shengfeng Ye, Shuiping Yu, Yukun Zha, Liyue Zhang, Haowei Zhang, Mingchuan Zhang, Wentao Zhang, Yichao Zhang, Chenggang Zhao, Yao Zhao, Shangyan Zhou, Shunfeng Zhou and Yuheng Zou. 2024. Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning. arXiv:2408.14158.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2408.14158 · arXiv 2408.14158

@misc{an2024fire,
  title = {{Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning}},
  author = {Wei An, Xiao Bi, Guanting Chen, Shanhuang Chen, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Wenjun Gao, Kang Guan, Jianzhong Guo, Yongqiang Guo, Zhe Fu, Ying He, Panpan Huang, Jiashi Li, Wenfeng Liang, Xiaodong Liu, Xin Liu, Yiyuan Liu, Yuxuan Liu, Shanghao Lu, Xuan Lu, Xiaotao Nie, Tian Pei, Junjie Qiu, Hui Qu, Zehui Ren, Zhangli Sha, Xuecheng Su, Xiaowen Sun, Yixuan Tan, Minghui Tang, Shiyu Wang, Yaohui Wang, Yongji Wang, Ziwei Xie, Yiliang Xiong, Yanhong Xu, Shengfeng Ye, Shuiping Yu, Yukun Zha, Liyue Zhang, Haowei Zhang, Mingchuan Zhang, Wentao Zhang, Yichao Zhang, Chenggang Zhao, Yao Zhao, Shangyan Zhou, Shunfeng Zhou and Yuheng Zou},
  year = {2024},
  doi = {10.48550/arxiv.2408.14158},
  eprint = {2408.14158},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2408.14158}
}

References

101 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” nature, vol. 521, no. 7553, pp. 436–444, 2015. I find
  2. R. R. Schaller, “Moore’s law: past, present and future,” IEEE spectrum, vol. 34, no. 6, pp. 52–59, 1997. I find
  3. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020. I
  4. Anthropic, “Introducing claude,” 2023. [Online]. Available: https://www.anthropic.com/news/introducing-claude I ↗
  5. Google, “An important next step on our ai journey,” 2023. [Online]. Available: https://blog.google/technology/ai/bard-google-ai-search-updates/ I ↗
  6. OpenAI, “Chatgpt: Optimizing language models for dialogue,” 2022. [Online]. Available: https://openai.com/blog/chatgpt I ↗
  7. J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023. I, II-A
  8. NVIDIA, “Nvidia dgx platform the best of nvidia ai—all in one place.” 2022. [Online]. Available: https://www.nvidia.com/en-us/data-center/dgx-platform/ I, III-C ↗
  9. C. E. Leiserson, “Fat-trees: Universal networks for hardware-efficient supercomputing,” IEEE Transactions on Computers, vol. C-34, no. 10, pp. 892–901, Oct 1985. I, III-B find
  10. NVIDIA, “Nvidia collective communications library (nccl): Optimized primitives for collective multi-gpu communication,” 2017. [Online]. Available: https://github.com/NVIDIA/nccl I, III-B ↗
  11. Y. Huang, Y. Cheng, D. Chen, H. Lee, J. Ngiam, Q. V. Le, and Z. Chen, “Gpipe: Efficient training of giant neural networks using pipeline parallelism,” CoRR, vol. abs/1811.06965, 2018. [Online]. Available: http://arxiv.org/abs/1811.06965 I, II-B1, V-B ↗
  12. D. Narayanan, A. Harlap, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, P. B. Gibbons, and M. Zaharia, “Pipedream: generalized pipeline parallelism for dnn training,” in Proceedings of the 27th ACM Symposium on Operating Systems Principles, ser. SOSP ’19. New York, NY, USA: Association fo… — [doi:10.1145/3341301.3359646](https://doi.org/10.1145/3341301.3359646) find
  13. M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism,” Mar. 2020, arXiv:1909.08053 [cs]. [Online]. Available: http://arxiv.org/abs/1909.08053 I, II-B1 ↗
  14. D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, A. Phanishayee, and M. Zaharia, “Efficient large-scale language model training on gpu clusters using megatron-lm,” in Proceedings of the International Conference fo… find
  15. S. Singh, O. Ruwase, A. A. Awan, S. Rajbhandari, Y. He, and A. Bhatele, “A hybrid tensor-expert-data parallelism approach to optimize mixture-of-experts training,” in Proceedings of the 37th ACM International Conference on Supercomputing, ser. ICS ’23. New York, NY, USA: Association for Computing Ma… find
  16. S. Rajbhandari, C. Li, Z. Yao, M. Zhang, R. Aminabadi, A. Awan, J. Rasley, and Y. He, “Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale,” Proceedings of Machine Learning Research, vol. 162, pp. 18 332–18 346, 2022, publisher Copyright: Copyright ©…
  17. C. Hwang, W. Cui, Y. Xiong, Z. Yang, Z. Liu, H. Hu, Z. Wang, R. Salas, J. Jose, P. Ram et al., “Tutel: Adaptive mixture-of-experts at scale,” Proceedings of Machine Learning and Systems, vol. 5, 2023. I, II-B1, V-B find
  18. Y. Zhao, A. Gu, R. Varma, L. Luo, C.-C. Huang, M. Xu, L. Wright, H. Shojanazeri, M. Ott, S. Shleifer et al., “Pytorch fsdp: experiences on scaling fully sharded data parallel,” arXiv preprint arXiv:2304.11277, 2023. I, V-B3 find
  19. S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He, “Zero: Memory optimizations toward training trillion parameter models,” in SC20: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 2020, pp. 1–16. I, II-B1, II-B1, V-B3 find
  20. HFAiLab, “Hai platform: A high-performance deep learning training platform with task-level gpu compute time-sharing scheduling,” 2023. [Online]. Available: https://github.com/HFAiLab/hai-platform I ↗
  21. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Advances in neural information processing systems, vol. 25, 2012. II-A
  22. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778. II-A, III
  23. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017. II-A
  24. A. W. Senior, R. Evans, J. Jumper, J. Kirkpatrick, L. Sifre, T. Green, C. Qin, A. Žı́dek, A. W. Nelson, A. Bridgland et al., “Improved protein structure prediction using potentials from deep learning,” Nature, vol. 577, no. 7792, pp. 706–710, 2020. II-A find
  25. D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel et al., “A general reinforcement learning algorithm that masters chess, shogi, and go through self-play,” Science, vol. 362, no. 6419, pp. 1140–1144, 2018. II-A find
  26. L. Floridi and M. Chiriatti, “Gpt-3: Its nature, scope, limits, and consequences,” Minds and Machines, vol. 30, pp. 681–694, 2020. II-A find
  27. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann et al., “Palm: Scaling language modeling with pathways,” Journal of Machine Learning Research, vol. 24, no. 240, pp. 1–113, 2023. II-A
  28. R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton, “Adaptive mixtures of local experts,” Neural computation, vol. 3, no. 1, pp. 79–87, 1991. II-A find
  29. M. I. Jordan and R. A. Jacobs, “Hierarchical mixtures of experts and the em algorithm,” Neural computation, vol. 6, no. 2, pp. 181–214, 1994. II-A find
  30. N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” arXiv preprint arXiv:1701.06538, 2017. II-A find
  31. T. Brooks, B. Peebles, C. Holmes, W. DePue, Y. Guo, L. Jing, D. Schnurr, J. Taylor, T. Luhman, E. Luhman, C. Ng, R. Wang, and A. Ramesh, “Video generation models as world simulators,” 2024. [Online]. Available: https://openai.com/research/video-generation-models-as-world-simulators II-A ↗
  32. A. Gholami, Z. Yao, S. Kim, C. Hooper, M. W. Mahoney, and K. Keutzer, “AI and Memory Wall,” Mar. 2024, arXiv:2403.14123 [cs]. [Online]. Available: http://arxiv.org/abs/2403.14123 II-A — [arXiv:2403.14123](https://arxiv.org/abs/2403.14123) ↗
  33. P. Qi, X. Wan, G. Huang, and M. Lin, “Zero bubble pipeline parallelism,” arXiv preprint arXiv:2401.10241, 2023. II-B1 find
  34. V. A. Korthikanti, J. Casper, S. Lym, L. McAfee, M. Andersch, M. Shoeybi, and B. Catanzaro, “Reducing activation recomputation in large transformer models,” in Proceedings of the Sixth Conference on Machine Learning and Systems, MLSys 2023, Miami, FL, USA, June 4-8, 2023, D. Song, M. Carbin, and T.…
  35. Z. Sun, H. Cao, Y. Wang, G. Feng, S. Chen, H. Wang, and W. Chen, “Adapipe: Optimizing pipeline parallelism with adaptive recomputation and partitioning,” in Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, ser. A… — [doi:10.1145/3620666.3651359](https://doi.org/10.1145/3620666.3651359) find
  36. X. Hou, Y. Yuan, S. Ma, R. Xu, B. Wang, T. Li, W. Jiang, L. Wu, and J. Zhang, “Optimizing the parallelism of communication and computation in distributed training platform,” in Algorithms and Architectures for Parallel Processing: 23rd International Conference, ICA3PP 2023, Tianjin, China, October 2… — [doi:10.1007/978-981-97-0834-5](https://doi.org/10.1007/978-981-97-0834-5) find
  37. J. Dongarra, “REPORT ON THE TIANHE-2A SYSTEM,” 2017. II-C1 find
  38. D. Stanzione, B. Barth, N. Gaffney, K. Gaither, C. Hempel, T. Minyard, S. Mehringer, E. Wernert, H. Tufo, D. Panda, and P. Teller, “Stampede 2: The evolution of an xsede supercomputer,” in Proceedings of the Practice and Experience in Advanced Research Computing 2017 on Sustainability, Success and I… — [doi:10.1145/3093338.3093385](https://doi.org/10.1145/3093338.3093385) find
  39. H. Fu, J. Liao, J. Yang, L. Wang, Z. Song, X. Huang, C. Yang, W. Xue, F. Liu, F. Qiao, W. Zhao, X. Yin, C. Hou, C. Zhang, W. Ge, J. Zhang, Y. Wang, C. Zhou, and G. Yang, “The Sunway TaihuLight supercomputer: system and applications,” Science China Information Sciences, vol. 59, no. 7, p. 072001, Jul… — [doi:10.1007/s11432-016-5588-7](https://doi.org/10.1007/s11432-016-5588-7) find
  40. T. Shimizu, “Supercomputer Fugaku: Co-designed with application developers/researchers,” in 2020 IEEE Asian Solid-State Circuits Conference (A-SSCC). Hiroshima, Japan: IEEE, Nov. 2020, pp. 1– 4. [Online]. Available: https://ieeexplore.ieee.org/document/9336127/II-C1 ↗
  41. D. Schneider, “The Exascale Era is Upon Us: The Frontier supercomputer may be the first to reach 1,000,000,000,000,000,000 operations per second,” IEEE Spectrum, vol. 59, no. 1, pp. 34–35, Jan. 2022. [Online]. Available: https://ieeexplore.ieee.org/document/9676353/ II-C2 ↗
  42. A. N. Laboratory, “Argonne’s aurora supercomputer,” 2023. [Online]. Available: https://www.alcf.anl.gov/aurora II-C2 ↗
  43. C. B. Stunkel, R. L. Graham, G. Shainer, M. Kagan, S. S. Sharkawi, B. Rosenburg, and G. A. Chochia, “The high-speed networks of the Summit and Sierra supercomputers,” IBM Journal of Research and Development, vol. 64, no. 3/4, pp. 3:1–3:10, May 2020. [Online]. Available: https://ieeexplore.ieee.org/d… ↗
  44. J. Gu, G. Eisenhauer, S. Klasky, N. Podhorszki, R. Wang, and K. Wu, “Exploring large all-flash storage system with scientific simulation,” in Proceedings of the 34th International Conference on Scientific and Statistical Database Management, ser. SSDBM ’22. New York, NY, USA: Association for Computi… — [doi:10.1145/3538712.3538734](https://doi.org/10.1145/3538712.3538734) find
  45. D. Mudigere, Y. Hao, J. Huang, Z. Jia, A. Tulloch, S. Sridharan, X. Liu, M. Ozdal, J. Nie, J. Park, L. Luo, J. A. Yang, L. Gao, D. Ivchenko, A. Basant, Y. Hu, J. Yang, E. K. Ardestani, X. Wang, R. Komuravelli, C.-H. Chu, S. Yilmaz, H. Li, J. Qian, Z. Feng, Y. Ma, J. Yang, E. Wen, H. Li, L. Yang, C.… — [arXiv:2104.05158](https://arxiv.org/abs/2104.05158) find
  46. A. Gangidi, R. Miao, S. Zheng, S. J. Bondu, G. Goes, H. Morsy, R. Puri, M. Riftadi, A. J. Shetty, J. Yang, S. Zhang, M. J. Fernandez, S. Gandham, and H. Zeng, “Rdma over ethernet for distributed training at meta scale,” in Proceedings of the ACM SIGCOMM 2024 Conference, ser. ACM SIGCOMM ’24. New Yor… — [doi:10.1145/3651890.3672233](https://doi.org/10.1145/3651890.3672233) find
  47. Y. Jiang, Y. Zhu, C. Lan, B. Yi, Y. Cui, and C. Guo, “A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters.” II-C3 find
  48. Z. Jiang, H. Lin, Y. Zhong, Q. Huang, Y. Chen, Z. Zhang, Y. Peng, X. Li, C. Xie, S. Nong, Y. Jia, S. He, H. Chen, Z. Bai, Q. Hou, S. Yan, D. Zhou, Y. Sheng, Z. Jiang, H. Xu, H. Wei, Z. Zhang, P. Nie, L. Zou, S. Zhao, L. Xiang, Z. Liu, Z. Li, X. Jia, J. Ye, X. Jin, and X. Liu, “MegaScale: Scaling Lar… find
  49. K. Qian, Y. Xi, J. Cao, J. Gao, Y. Xu, Y. Guan, B. Fu, X. Shi, F. Zhu, R. Miao, C. Wang, P. Wang, P. Zhang, X. Zeng, E. Ruan, Z. Yao, E. Zhai, and D. Cai, “Alibaba hpn: A data center network for large language model training,” in Proceedings of the ACM SIGCOMM 2024 Conference, ser. ACM SIGCOMM ’24.… — [doi:10.1145/3651890.3672265](https://doi.org/10.1145/3651890.3672265) find
  50. NVIDIA, “Nvidia announces dgx h100 systems – world’s most advanced enterprise ai infrastructure,” 2023. [Online]. Available: https://nvidianews.nvidia.com/news/nvidia-announces-dgx-h100-systems-worlds-most-advanced-enterprise-ai-infrastructure II-C3 ↗
  51. N. Jouppi, G. Kurian, S. Li, P. Ma, R. Nagarajan, L. Nai, N. Patil, S. Subramanian, A. Swing, B. Towles, C. Young, X. Zhou, Z. Zhou, and D. A. Patterson, “TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings,” in Proceedings of the 50th Annual I…
  52. C. Zhang, B. Sun, X. Yu, Z. Xie, W. Zheng, K. Iskra, P. Beckman, and D. Tao, “Benchmarking and In-depth Performance Study of Large Language Models on Habana Gaudi Processors,” Sep. 2023, arXiv:2309.16976 [cs]. [Online]. Available: http://arxiv.org/abs/2309.16976 II-C4 — [arXiv:2309.16976](https://arxiv.org/abs/2309.16976) ↗
  53. E. Talpes, D. Williams, and D. D. Sarma, “Dojo: The microarchitecture of tesla’s exa-scale computer,” in 2022 IEEE Hot Chips 34 Symposium (HCS), 2022, pp. 1–28. II-C4 find
  54. E. Talpes, D. D. Sarma, D. Williams, S. Arora, T. Kunjan, B. Floering, A. Jalote, C. Hsiong, C. Poorna, V. Samant, J. Sicilia, A. K. Nivarti, R. Ramachandran, T. Fischer, B. Herzberg, B. McGee, G. Venkataramanan, and P. Banon, “The microarchitecture of dojo, tesla’s exa-scale computer,” IEEE Micro,… find
  55. H. Liao, J. Tu, J. Xia, and X. Zhou, “Davinci: A scalable architecture for neural network computing,” in 2019 IEEE Hot Chips 31 Symposium (HCS), 2019, pp. 1–44. II-C4 find
  56. H. Liao, J. Tu, J. Xia, H. Liu, X. Zhou, H. Yuan, and Y. Hu, “Ascend: a scalable and unified architecture for ubiquitous deep neural network computing : Industry track paper,” in 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), 2021, pp. 789–801. II-C4 find
  57. D. Milojicic, P. Faraboschi, N. Dube, and D. Roweth, “Future of hpc: Diversifying heterogeneity,” in 2021 Design, Automation Test in Europe Conference Exhibition (DATE), 2021, pp. 276–281. II-D find
  58. Y. Su, J. Zhou, J. Ying, M. Zhou, and B. Zhou, “Computing infrastructure construction and optimization for high-performance computing and artificial intelligence,” CCF Transactions on High Performance Computing, vol. 3, no. 4, pp. 331–343, Dec. 2021. [Online]. Available: https://doi.org/10.1007/s425… — [doi:10.1007/s42514-021-00080-x](https://doi.org/10.1007/s42514-021-00080-x) ↗
  59. E. Hassan, N. El-Rashidy, and F. M. Talaa, “Review: Mask R-CNN Models,” Nile Journal of Communication and Computer Science, vol. 3, no. 1, pp. 17–27, May 2022. [Online]. Available: https://njccs.journals.ekb.eg/article 280047.html III ↗
  60. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in North American Chapter of the Association for Computational Linguistics, 2019. [Online]. Available: https://api.semanticscholar.org/CorpusID:52967399 III ↗
  61. K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” arXiv:2111.06377, 2021. III
  62. J. Kim, W. J. Dally, S. Scott, and D. Abts, “Technology-driven, highly-scalable dragonfly topology,” in 2008 International Symposium on Computer Architecture, 2008, pp. 77–88. III-B find
  63. G. Feng, D. Dong, and Y. Lu, “Optimized mpi collective algorithms for dragonfly topology,” in Proceedings of the 36th ACM International Conference on Supercomputing, ser. ICS ’22. New York, NY, USA: Association for Computing Machinery, 2022. [Online]. Available: https://doi.org/10.1145/3524059.35323… — [doi:10.1145/3524059.3532380](https://doi.org/10.1145/3524059.3532380) ↗
  64. H. Subramoni, P. Lai, M. Luo, and D. K. Panda, “Rdma over ethernet — a preliminary study,” in 2009 IEEE International Conference on Cluster Computing and Workshops, 2009, pp. 1–9. III-B find
  65. P. Sanders, J. Speck, and J. Träff, “Two-tree algorithms for full bandwidth broadcast, reduction and scan,” Parallel Computing, vol. 35, pp. 581–594, 12 2009. III-B, IV, 1, IV-D1 find
  66. R. Shi, S. Potluri, K. Hamidouche, J. Perkins, M. Li, D. Rossetti, and D. K. D. K. Panda, “Designing efficient small message transfer mechanism for inter-node mpi communication on infiniband gpu clusters,” in 2014 21st International Conference on High Performance Computing (HiPC), 2014, pp. 1–10. 1 find
  67. P. Foundation, “Tensors and dynamic neural networks in python with strong gpu acceleration,” 2016. [Online]. Available: https://github.com/pytorch/pytorch V-A ↗
  68. S. Liu and W. Deng, “Very deep convolutional neural network based image classification using small training sample size,” in 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR), 2015, pp. 730–734. V-A find
  69. M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, “Megatron-lm: Training multi-billion parameter language models using model parallelism,” arXiv preprint arXiv:1909.08053, 2019. V-B find
  70. J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He, “Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters,” in Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, ser. KDD ’20. New York, NY, USA: Association… — [doi:10.1145/3394486.3406703](https://doi.org/10.1145/3394486.3406703) find
  71. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “Llama: Open and efficient foundation language models,” ArXiv, vol. abs/2302.13971, 2023. [Online]. Available: https://api.semantic… ↗
  72. DeepSeek-AI, A. Liu, B. Feng, B. Wang, B. Wang, B. Liu, C. Zhao, C. Dengr, C. Ruan, D. Dai, D. Guo, D. Yang, D. Chen, D. Ji, E. Li, F. Lin, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Xu, H. Yang, H. Zhang, H. Ding, H. Xin, H. Gao, H. Li, H. Qu, J. L. Cai, J. Liang, J. Guo, J. Ni, J. Li, J. Chen, J… find
  73. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” 2019. V-B3 find
  74. S.-A. Reinemo, T. Skeie, T. Sodring, O. Lysne, and O. Trudbakken, “An overview of qos capabilities in infiniband, advanced switching interconnect, and ethernet,” IEEE Communications Magazine, vol. 44, no. 7, pp. 32–38, 2006. VI-A1, VI-A1 find
  75. D. Crupnicoff, S. Das, and E. Zahavi, “Deploying Quality of Service and Congestion Control in InfiniBand-based Data Center Networks.” [Online]. Available: https://network.nvidia.com/sites/default/files/related-docs/whitepapers/deploying qos wp 10 19 2005.pdf VI-A1, VI-A1 ↗
  76. M. Scharf and S. Kiesel, “Nxg03-5: Head-of-line blocking in tcp and sctp: Analysis and measurements,” in IEEE Globecom 2006, 2006, pp. 1–5. VI-A1 find
  77. R. Budruk, D. Anderson, and E. Solari, PCI Express System Architecture. Pearson Education, 2003. VI-A3 find
  78. Z. Liran, H. David, and M. Barbara, “Wekafs architecture white paper,” 2021. [Online]. Available: https://www.weka.io/wp-content/uploads/files/2017/12/Architectural WhitePaper-W02R6WP201812-1.pdf VI-B1 ↗
  79. Z. Liang, J. Lombardi, M. Chaarawi, and M. Hennecke, “Daos: A scale-out high performance storage stack for storage class memory,” in Supercomputing Frontiers, D. K. Panda, Ed. Cham: Springer International Publishing, 2020, pp. 40–54. VI-B1 find
  80. M. Hennecke, “Understanding daos storage performance scalability,” in Proceedings of the HPC Asia 2023 Workshops, ser. HPCAsia ’23 Workshops. New York, NY, USA: Association for Computing Machinery, 2023, p. 1–14. [Online]. Available: https://doi.org/10.1145/3581576.3581577 VI-B1 — [doi:10.1145/3581576.3581577](https://doi.org/10.1145/3581576.3581577) ↗
  81. H. Frank and B. Sven, “Wekafs architecture white paper,” 2018. [Online]. Available: https://www.beegfs.io/docs/whitepapers/Introduction to BeeGFS by ThinkParQ.pdf VI-B1 ↗
  82. J. Terrace and M. J. Freedman, “Object storage on CRAQ: High-Throughput chain replication for Read-Mostly workloads,” in 2009 USENIX Annual Technical Conference (USENIX ATC 09). San Diego, CA: USENIX Association, Jun. 2009. [Online]. Available: https://www.usenix.org/conference/usenix-09/object-stor… ↗
  83. E. B. Nightingale, J. Elson, J. Fan, O. Hofmann, J. Howell, and Y. Suzue, “Flat datacenter storage,” in Proceedings of the 10th USENIX Conference on Operating Systems Design and Implementation, ser. OSDI’12. USA: USENIX Association, 2012, p. 1–15. VI-B3 find
  84. DeepSeek-AI, “Deepseek api introduces context caching on disk, cutting prices by an order of magnitude,” 2024. [Online]. Available: https://platform.deepseek.com/api-docs/news/news0802 VI-B4 ↗
  85. K. Liu, Z. Jiang, J. Zhang, H. Wei, X. Zhong, L. Tan, T. Pan, and T. Huang, “Hostping: Diagnosing intra-host network bottlenecks in RDMA servers,” in 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23). Boston, MA: USENIX Association, Apr. 2023, pp. 15–29. [Online]. Availa… find
  86. Y. He, M. Hutton, S. Chan, R. De Gruijl, R. Govindaraju, N. Patil, and Y. Li, “Understanding and Mitigating Hardware Failures in Deep Learning Training Systems,” in Proceedings of the 50th Annual International Symposium on Computer Architecture. Orlando FL USA: ACM, Jun. 2023, pp. 1–16. [Online]. Av… — [doi:10.1145/3579371.3589105](https://doi.org/10.1145/3579371.3589105) find
  87. S. Wang, G. Zhang, J. Wei, Y. Wang, J. Wu, and Q. Luo, “Understanding Silent Data Corruptions in a Large Production CPU Population,” in Proceedings of the 29th Symposium on Operating Systems Principles. Koblenz Germany: ACM, Oct. 2023, pp. 216–230. [Online]. Available: https://dl.acm.org/doi/10.1145… — [doi:10.1145/3600006.3613149](https://doi.org/10.1145/3600006.3613149) ↗
  88. NVIDIA, “This document explains what xid messages are, and is intended to assist system administrators, developers, and faes in understanding the meaning behind these messages as an aid in analyzing and resolving gpu-related problems.” [Online]. Available: https://docs.nvidia.com/deploy/xid-errors/… ↗
  89. “Priority flow control : Build reliable layer 2 infrastructure,” 2015. [Online]. Available: https://api.semanticscholar.org/CorpusID: 42645413 VIII-A ↗
  90. S. Yan, G. Min, and I. Awan, “Performance analysis of credit-based flow control in infiniband interconnection networks,” Journal of Interconnection Networks, vol. 07, no. 04, pp. 535–548, 2006. [Online]. Available: https://doi.org/10.1142/S0219265906001843 VIII-A — [doi:10.1142/s0219265906001843](https://doi.org/10.1142/s0219265906001843) ↗
  91. E. Nepolo and G.-A. Lusilao Zodi, “A predictive ecmp routing protocol for fat-tree enabled data centre networks,” in 2021 15th International Conference on Ubiquitous Information Management and Communication (IMCOM), 2021, pp. 1–8. VIII-A find
  92. J. Rocher-González, E. G. Gran, S.-A. Reinemo, T. Skeie, J. Escudero-Sahuquillo, P. J. Garcı́a, and F. J. Q. Flor, “Adaptive routing in infiniband hardware,” in 2022 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing (CCGrid), 2022, pp. 463–472. VIII-A find
  93. Y. Zhu, H. Eran, D. Firestone, C. Guo, M. Lipshteyn, Y. Liron, J. Padhye, S. Raindel, M. H. Yahia, and M. Zhang, “Congestion control for large-scale rdma deployments,” SIGCOMM Comput. Commun. Rev., vol. 45, no. 4, p. 523–536, aug 2015. [Online]. Available: https://doi.org/10.1145/2829988.2787484 VII… — [doi:10.1145/2829988.2787484](https://doi.org/10.1145/2829988.2787484) ↗
  94. Y. Li, R. Miao, H. H. Liu, Y. Zhuang, F. Feng, L. Tang, Z. Cao, M. Zhang, F. Kelly, M. Alizadeh, and M. Yu, “Hpcc: high precision congestion control,” in Proceedings of the ACM Special Interest Group on Data Communication, ser. SIGCOMM ’19. New York, NY, USA: Association for Computing Machinery, 201… — [doi:10.1145/3341302.3342085](https://doi.org/10.1145/3341302.3342085) find
  95. R. Mittal, V. T. Lam, N. Dukkipati, E. R. Blem, H. M. G. Wassel, M. Ghobadi, A. Vahdat, Y. Wang, D. Wetherall, and D. Zats, “Timely: Rtt-based congestion control for the datacenter,” Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication, 2015. [Online]. Available: ht… find
  96. Q. Hu, Z. Ye, Z. Wang, G. Wang, M. Zhang, Q. Chen, P. Sun, D. Lin, X. Wang, Y. Luo et al., “Characterization of large language model development in the datacenter,” in 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), 2024, pp. 709– 729. VIII-D find
  97. DeepSeek-AI, X. Bi, D. Chen, G. Chen, S. Chen, D. Dai, C. Deng, H. Ding, K. Dong, Q. Du, Z. Fu, H. Gao, K. Gao, W. Gao, R. Ge, K. Guan, D. Guo, J. Guo, G. Hao, Z. Hao, Y. He, W. Hu, P. Huang, E. Li, G. Li, J. Li, Y. Li, Y. K. Li, W. Liang, F. Lin, A. X. Liu, B. Liu, W. Liu, X. Liu, X. Liu, Y. Liu, H… find
  98. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo, “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models,” Apr. 2024, arXiv:2402.03300 [cs]. [Online]. Available: http://arxiv.org/abs/2402.03300 X ↗
  99. H. Lu, W. Liu, B. Zhang, B. Wang, K. Dong, B. Liu, J. Sun, T. Ren, Z. Li, H. Yang, Y. Sun, C. Deng, H. Xu, Z. Xie, and C. Ruan, “DeepSeek-VL: Towards Real-World Vision-Language Understanding,” Mar. 2024, arXiv:2403.05525 [cs]. [Online]. Available: http://arxiv.org/abs/2403.05525 X ↗
  100. DeepSeek-AI, Q. Zhu, D. Guo, Z. Shao, D. Yang, P. Wang, R. Xu, Y. Wu, Y. Li, H. Gao, S. Ma, W. Zeng, X. Bi, Z. Gu, H. Xu, D. Dai, K. Dong, L. Zhang, Y. Piao, Z. Gou, Z. Xie, Z. Hao, B. Wang, J. Song, D. Chen, X. Xie, K. Guan, Y. You, A. Liu, Q. Du, W. Gao, X. Lu, Q. Chen, Y. Wang, C. Deng, J. Li, C.… — DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence
  101. H. Xin, Z. Z. Ren, J. Song, Z. Shao, W. Zhao, H. Wang, B. Liu, L. Zhang, X. Lu, Q. Du, W. Gao, Q. Zhu, D. Yang, Z. Gou, Z. F. Wu, F. Luo, and C. Ruan, “DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Search,” Aug. 2024, arXiv:2408.08152 [cs].…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/an-2024-fire-flyer-ai-hpc/