Elamur Research
☰
home / papers / 2020 / PyTorch Distributed: Experiences on Accelerating Data Parallel Training

PyTorch Distributed: Experiences on Accelerating Data Parallel Training

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 4.0 (arXiv)" does not allow redistributing the text itself. Original — arxiv.org/pdf/2006.15704

Summary

The paper describes how PyTorch implements DistributedDataParallel (DDP), its module for spreading training across many GPUs and machines. Each worker holds a full model copy, processes its own slice of data, and gradients are averaged every iteration so replicas stay identical. The authors set three goals: results matching single-machine training, a wrapper that needs almost no changes to user code yet can still hook into training internals, and high speed despite the coupling between computation and communication. They chose gradient synchronization over parameter averaging because averaging can diverge from local training, for example with momentum. Averaging also cannot overlap work. DDP groups gradients into buckets, starts AllReduce while the backward pass is still running, and can skip synchronization on some iterations. It runs on the c10d library with NCCL, Gloo and MPI backends. Tests used ResNet50 and BERT on a dedicated 32-GPU cluster and on 256 shared GPUs. With NCCL and sensible settings, DDP scaled almost linearly. This matters because DDP is a widely used industrial tool for scaling deep learning.

Key points

  • The design is described for PyTorch v1.5; the acceleration techniques first shipped in v1.1.
  • Communication was the largest source of training latency, and its share grew with model size.
  • Tuning bucket size alone could more than double speed.
  • Skipping synchronization on some iterations reduced average communication cost without noticeably slowing convergence.
  • In a Facebook workload study (05/11/20–06/05/20), more than 60% of production GPU hours used DDP, covering speech, vision, translation and other tasks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Computing & GPUs

Cited by

270 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 266 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2006.15704 · arXiv 2006.15704

@misc{li2020pytorch,
  title = {{PyTorch Distributed: Experiences on Accelerating Data Parallel Training}},
  author = {Li, Shen and Zhao, Yanli and Varma, Rohan and Salpekar, Omkar and Noordhuis, Pieter and Li, Teng and Paszke, Adam and Smith, Jeff and Vaughan, Brian and Damania, Pritam and Chintala, Soumith},
  year = {2020},
  doi = {10.48550/arxiv.2006.15704},
  eprint = {2006.15704},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2006.15704}
}

References

40 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gloo: a collective communications library. https://github.com/facebookincubator/gloo, 2019. ↗
  2. NVIDIA Collective Communications Library (NCCL). https://developer.nvidia.com/nccl, 2019. ↗
  3. NVLINK AND NVSWITCH: The Building Blocks of Advanced Multi-GPU Communication. https: //www.nvidia.com/en-us/data-center/nvlink/, 2019. find
  4. Open MPI: A High Performance Message Passing Library. https://www.open-mpi.org/, 2019. ↗
  5. Pybind11: Seamless operability between C++11 and Python. https://pybind11.readthedocs.io/, 2019. ↗
  6. PyTorch Distributed RPC Framework. https://pytorch.org/docs/master/rpc.html, 2019. ↗
  7. PyTorch Module forward Function. https://pytorch.org/docs/stable/nn.html\#torch. nn.Module.forward, 2019. ↗
  8. SciPy: open-source software for mathematics, science, and engineering. https://docs.scipy.org/, 2019. ↗
  9. PyTorch DistributedDataParallel. https://pytorch.org/docs/stable/nn.html\#torch. nn.parallel.DistributedDataParallel, 2020. ↗
  10. TensorFlow Distributed Training MultiWorkerMirroredStrategy. https://www.tensorflow.org/guide/distributed\_training\#multiworkermirroredstrategy, 2020. ↗
  11. TensorFlow Distributed Training ParameterServerStrategy. https://www.tensorflow.org/guide/distributed\_training\#parameterserverstrategy, 2020. ↗
  12. Y. Bao, Y. Peng, Y. Chen, and C. Wu. Preemptive all-reduce scheduling for expediting distributed dnn training. In IEEE INFOCOM, 2020. find
  13. M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, et al. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316, 2016. — [arXiv:1604.07316](https://arxiv.org/abs/1604.07316) find
  14. M. Cho, U. Finkler, M. Serrano, D. Kung, and H. Hunter. Blueconnect: Decomposing all-reduce for deep learning on heterogeneous network hierarchy. IBM Journal of Research and Development, 63(6):1–1, 2019. find
  15. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. find
  16. M. Du, F. Li, G. Zheng, and V. Srikumar. Deeplog: Anomaly detection and diagnosis from system logs through deep learning. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pages 1285–1298, 2017. find
  17. A. Fan, E. Grave, and A. Joulin. Reducing transformer depth on demand with structured dropout. arXiv preprint arXiv:1909.11556, 2019. find our card
  18. X. Guo, S. Singh, H. Lee, R. L. Lewis, and X. Wang. Deep learning for real-time atari game play using offline monte-carlo tree search planning. In Advances in neural information processing systems, pages 3338–3346, 2014. find
  19. S. H. Hashemi, S. A. Jyothi, and R. H. Campbell. Tictac: Accelerating distributed deep learning with communication scheduling. arXiv preprint arXiv:1803.03288, 2018. — [arXiv:1803.03288](https://arxiv.org/abs/1803.03288) find
  20. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. find
  21. Y. Huang, Y. Cheng, A. Bapna, O. Firat, D. Chen, M. Chen, H. Lee, J. Ngiam, Q. V. Le, Y. Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in Neural Information Processing Systems, pages 103–112, 2019. find
  22. S. Jeaugey. Massively Scale Your Deep Learning Training with NCCL 2.4. https://devblogs.nvidia.com/massively-scale-deep-learning-training-nccl-2-4/, February 2019. ↗
  23. S. Kim, G.-I. Yu, H. Park, S. Cho, E. Jeong, H. Ha, S. Lee, J. S. Jeong, and B.-G. Chun. Parallax: Sparsity-aware data parallel training of deep neural networks. In Proceedings of the Fourteenth EuroSys Conference 2019, pages 1–15, 2019. find
  24. J. Kosaian, K. V. Rashmi, and S. Venkataraman. Parity models: Erasure-coded resilience for prediction serving systems. In Proceedings of the 27th ACM Symposium on Operating Systems Principles, SOSP 19, page 3046, New York, NY, USA, 2019. Association for Computing Machinery. find
  25. Y. LeCun, C. Cortes, and C. Burges. The MNIST Database. http://yann.lecun.com/exdb/mnist/, 1999. ↗
  26. Y. LeCun, D. Touresky, G. Hinton, and T. Sejnowski. A theoretical framework for back-propagation. In Proceedings of the 1988 connectionist models summer school, volume 1, pages 21–28. CMU, Pittsburgh, Pa: Morgan Kaufmann, 1988. find
  27. M. Li, D. G. Andersen, J. W. Park, A. J. Smola, A. Ahmed, V. Josifovski, J. Long, E. J. Shekita, and B.-Y. Su. Scaling distributed machine learning with the parameter server. In 11th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 14), pages 583–598, 2014. find
  28. H. Mao, M. Cheung, and J. She. Deepart: Learning joint representations of visual arts. In Proceedings of the 25th ACM international conference on Multimedia, pages 1183–1191, 2017. find
  29. D. Narayanan, A. Harlap, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, P. B. Gibbons, and M. Zaharia. Pipedream: generalized pipeline parallelism for dnn training. In Proceedings of the 27th ACM Symposium on Operating Systems Principles, pages 1–15, 2019. find
  30. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning… find our card
  31. Y. Peng, Y. Zhu, Y. Chen, Y. Bao, B. Yi, C. Lan, C. Wu, and C. Guo. A generic communication scheduler for distributed dnn training acceleration. In Proceedings of the 27th ACM Symposium on Operating Systems Principles, pages 16–29, 2019. find
  32. S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: Memory optimization towards training a trillion parameter models. arXiv preprint arXiv:1910.02054, 2019. find
  33. B. Ramsundar, P. Eastman, P. Walters, and V. Pande. Deep learning for the life sciences: applying deep learning to genomics, microscopy, drug discovery, and more. ” O’Reilly Media, Inc.”, 2019. find
  34. F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu. 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech dnns. In Fifteenth Annual Conference of the International Speech Communication Association, 2014. find
  35. A. Sergeev and M. D. Balso. Horovod: fast and easy distributed deep learning in TensorFlow. arXiv preprint arXiv:1802.05799, 2018. find
  36. N. Shazeer, Y. Cheng, N. Parmar, D. Tran, A. Vaswani, P. Koanantakool, P. Hawkins, H. Lee, M. Hong, C. Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414–10423, 2018. find our card
  37. P. Sun, Y. Wen, R. Han, W. Feng, and S. Yan. Gradientflow: Optimizing network performance for large-scale distributed dnn training. IEEE Transactions on Big Data, 2019. find
  38. A. Van den Oord, S. Dieleman, and B. Schrauwen. Deep content-based music recommendation. In Advances in neural information processing systems, pages 2643–2651, 2013. find
  39. G. Wang, S. Venkataraman, A. Phanishayee, J. Thelin, N. Devanur, and I. Stoica. Blink: Fast and generic collectives for distributed ml. arXiv preprint arXiv:1910.04940, 2019. find
  40. J. Wang, V. Tantia, N. Ballas, and M. Rabbat. Slowmo: Improving communication-efficient distributed sgd with slow momentum. arXiv preprint arXiv:1910.00643, 2019. — [arXiv:1910.00643](https://arxiv.org/abs/1910.00643) find