Elamur Research⌕
☰
home / papers / 2020 / DAPPLE: A Pipelined Data Parallel Approach for Training Large Models

DAPPLE: A Pipelined Data Parallel Approach for Training Large Models

Paper · 2020 · Shiqing Fan, Yi Rong, Chen Meng et al.

Summary

DAPPLE tackles the problem of training very large DNNs on GPU clusters whose interconnects vary, such as NVLink inside servers and slower Ethernet between them. Pure data parallelism breaks down when a model no longer fits on one device or when gradient synchronization dominates. Existing pipeline approaches have their own costs: asynchronous ones (PipeDream) raise convergence concerns and store several weight versions, while synchronous GPipe needs a lot of activation memory or about 20% extra recomputation. The authors from Alibaba and HKU build a synchronous framework with three parts. A profiler collects per-layer compute time, activation size and parameter size. A planner then searches hybrid strategies that split layers into stages, replicate stages data-parallel style, and place them on devices with awareness of memory and topology. Finally, a runtime interleaves forward and backward work so activations are freed early. Across image classification, translation and language modeling, DAPPLE beats DP, the PipeDream planner and GPipe in throughput while using less memory. This makes synchronous large-model training more practical on everyday industrial hardware.

Key points

  • Benchmarks: GNMT-16, BERT-48, XLNET-36, AmoebaNet-36 and VGG-19, tested on three cluster setups: hierarchical NVLink + Ethernet, 25 Gbps Ethernet and 10 Gbps Ethernet.
  • Reported gains:
    • up to 2.32× over the best DP baseline;
    • up to 3.23× over strategies from the PipeDream planner in synchronous settings;
    • 1.6× higher throughput than GPipe, with 12% less memory.
  • The scheduling method is independent of re-computation. It saves roughly another 20% of device memory even when recomputation is already turned on.
  • The efficiency model treats bubble overhead as P = (1+α)(S−1)/M and efficiency as 1/(1+P), where S is the number of stages, M the number of micro-batches and α the communication-to-computation ratio. More micro-batches and fewer stages give better utilization.
  • Activations crossing partition boundaries are tiny next to gradients, for example 8.8MB vs 2.8GB for BERT-48 and 26MB vs 1.1GB for GNMT-16. This explains why pipelining can cut traffic compared with full DP.

Interesting ideas

  • I1. Model communication as a pipeline stage. Folding communication into the analytical pipeline model gives a simple formula that predicts when a split pays off. The same trick could help capacity planning for any staged data-processing job.
  • I2. Compare traffic volumes before choosing a strategy. Measuring boundary-activation size against gradient size is a cheap first check for whether pipelining or plain DP suits a model and network. This is handy before committing to costly infrastructure.
  • I3. Profile first, then plan automatically. Collecting per-layer statistics and letting a planner search hybrid layouts, including stage replication, replaces manual tuning with a reproducible search. It also adapts when the hardware changes.
  • I4. Interleave forward and backward work. Running backward passes in a fixed order alongside forward ones lets memory for each micro-batch be released as early as possible. Peak memory drops without recomputing anything, and the idea stacks with other memory-saving techniques.
  • I5. Topology-aware placement. Assigning stages to devices with the interconnect hierarchy in mind matters on mixed NVLink/Ethernet clusters. It is a useful design rule for anyone training on heterogeneous or rented GPU setups.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Alibaba GroupHangzhou · China201 works · 17,172 citations🎓University of Hong KongPok Fu Lam · China118 works · 13,496 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 worksHardware and Architecture · 167 works

Cite

Cite as Shiqing Fan, Yi Rong, Chen Meng, Zongyan Cao, Siyu Wang, Zhen Zheng, Chuan Wu, Guoping Long, Jun Yang, Lixue Xia, Lansong Diao, Xiaoyong Liu and Wei Lin. 2020. DAPPLE: A Pipelined Data Parallel Approach for Training Large Models. arXiv:2007.01045.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2007.01045 · arXiv 2007.01045

@misc{fan2020dapple,
  title = {{DAPPLE: A Pipelined Data Parallel Approach for Training Large Models}},
  author = {Shiqing Fan, Yi Rong, Chen Meng, Zongyan Cao, Siyu Wang, Zhen Zheng, Chuan Wu, Guoping Long, Jun Yang, Lixue Xia, Lansong Diao, Xiaoyong Liu and Wei Lin},
  year = {2020},
  doi = {10.48550/arxiv.2007.01045},
  eprint = {2007.01045},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2007.01045}
}

References

59 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. Sutton, The Bitter Lesson, 2019, http://www.incompleteideas.net/IncIdeas/BitterLesson.html. ↗
  2. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and J. P. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” https://arxiv.org/abs/1910.10683, 2019. ↗
  3. J. Wang, P. Huang, H. Zhao, Z. Zhang, B. Zhao, and D. L. Lee, “Billion-scale commodity embedding for e-commerce recommendation in alibaba,” in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2018, pp. 839–848. find
  4. P. Covington, J. Adams, and E. Sargin, “Deep neural networks for youtube recommendations,” in Proceedings of the 10th ACM Conference on Recommender Systems, ACM, New York, NY, USA. ACM, 2016. find
  5. Gradients Accumulation-Tensorflow, 2019, https://github.com/tensorflow/tensorflow/pull/32576. ↗
  6. Gradients Accumulation-PyTorch, 2019, https://gist.github.com/thomwolf/ac7a7da6b1888c2eeac8ac8b9b05d3d3. ↗
  7. A New Lightweight, Modular, and Scalable Deep Learning Framework, 2016, https://caffe2.ai/. ↗
  8. A. Jayarajan, J. Wei, G. Gibson, A. Fedorova, and G. Pekhimenko, “Priority-based parameter propagation for distributed dnn training,” arXiv preprint arXiv:1905.03960, 2019. — [arXiv:1905.03960](https://arxiv.org/abs/1905.03960) find
  9. A. Sergeev and M. Del Balso, “Horovod: fast and easy distributed deep learning in tensorflow,” arXiv preprint arXiv:1802.05799, 2018.
  10. Y. Huang, Y. Cheng, A. Bapna, O. Firat, D. Chen, M. Chen, H. Lee, J. Ngiam, Q. V. Le, Y. Wu et al., “Gpipe: Efficient training of giant neural networks using pipeline parallelism,” in Advances in Neural Information Processing Systems, 2019, pp. 103–112. find
  11. J. Zhan and J. Zhang, “Pipe-torch: Pipeline-based distributed deep learning in a gpu cluster with heterogeneous networking,” in 2019 Seventh International Conference on Advanced Cloud and Big Data (CBD). IEEE, 2019, pp. 55–60. find
  12. D. Narayanan, A. Harlap, A. Phanishayee, V. Seshadri, N. R. Devanur, G. R. Ganger, P. B. Gibbons, and M. Zaharia, “Pipedream: generalized pipeline parallelism for dnn training,” in Proceedings of the 27th ACM Symposium on Operating Systems Principles. ACM, 2019, pp. 1–15. find
  13. T. Chen, B. Xu, C. Zhang, and C. Guestrin, “Training deep nets with sublinear memory cost,” arXiv preprint arXiv:1604.06174, 2016. find
  14. A. Harlap, D. Narayanan, A. Phanishayee, V. Seshadri, N. Devanur, G. Ganger, and P. Gibbons, “Pipedream: Fast and efficient pipeline parallel dnn training,” arXiv preprint arXiv:1806.03377, 2018.
  15. M. Wang, C. Meng, G. Long, C. Wu, J. Yang, W. Lin, and Y. Jia, “Characterizing deep learning training workloads on alibaba-pai,” arXiv preprint arXiv:1910.05930, 2019. find
  16. GpipeTalk, 2019, https://www.youtube.com/watch?v=9s2cum25Kkc. ↗
  17. P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He, “Accurate, large minibatch sgd: Training imagenet in 1 hour,” arXiv preprint arXiv:1706.02677, 2017. find
  18. G. Long, J. Yang, K. Zhu, and W. Lin, “Fusionstitching: Deep fusion and code generation for tensorflow computations on gpus,” https://arxiv.org/abs/1811.05213, 2018. ↗
  19. G. Long, J. Yang, and W. Lin, “Fusionstitching: Boosting execution efficiency of memory intensive computations for dl workloads,” https://arxiv.org/abs/1911.11576, 2019. ↗
  20. H. Zhang, Z. Zheng, S. Xu, W. Dai, Q. Ho, X. Liang, Z. Hu, J. Wei, P. Xi, and E. P. Xing, “Poseidon: An efficient communication architecture for distributed deep learning on GPU clusters,” in 2017 USENIX Annual Technical Conference (USENIX ATC 17). Santa Clara, CA: USENIX Association, Jul. 2017, pp.… find
  21. B. Yang, J. Zhang, J. Li, C. Ré, C. R. Aberger, and C. De Sa, “Pipemare: Asynchronous pipeline parallel dnn training,” arXiv preprint arXiv:1910.05124, 2019. — [arXiv:1910.05124](https://arxiv.org/abs/1910.05124) find
  22. NVLink, 2019, https://www.nvidia.com/en-us/data-center/nvlink/. ↗
  23. H. Lee, M. Jeong, C. Kim, S. Lim, I. Kim, W. Baek, and B. Yoon, “torchgpipe, A GPipe implementation in PyTorch,” https://github.com/kakaobrain/torchgpipe, 2019. ↗
  24. I. Bárány and V. S. Grinberg, “Block partitions of sequences,” Israel Journal of Mathematics, vol. 206, no. 1, pp. 155–164, 2015. find
  25. M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B.… find
  26. J. Demouth, “Cuda pro tip: Minimize the tail effect,” 2015. [Online]. Available: https://devblogs.nvidia.com/cuda-pro-tip-minimize-the-tail-effect/ ↗
  27. R. Sennrich, B. Haddow, and A. Birch, “Edinburgh neural machine translation systems for wmt 16,” arXiv preprint arXiv:1606.02891, 2016.
  28. P. Rajpurkar, R. Jia, and P. Liang, “Know what you don’t know: Unanswerable questions for squad,” arXiv preprint arXiv:1806.03822, 2018.
  29. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al., “Imagenet large scale visual recognition challenge,” International journal of computer vision, vol. 115, no. 3, pp. 211–252, 2015.
  30. Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey et al., “Google’s neural machine translation system: Bridging the gap between human and machine translation,” arXiv preprint arXiv:1609.08144, 2016. find
  31. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.
  32. Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. V. Le, “Xlnet: Generalized autoregressive pretraining for language understanding,” arXiv preprint arXiv:1906.08237, 2019.
  33. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  34. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
  35. S. A. R. Shah, W. Wu, Q. Lu, L. Zhang, S. Sasidharan, P. DeMar, C. Guok, J. Macauley, E. Pouyoul, J. Kim et al., “Amoebanet: An sdn-enabled network service for big data science,” Journal of Network and Computer Applications, vol. 119, pp. 70–82, 2018. find
  36. NCCL, 2019, https://developer.nvidia.com/nccl. ↗
  37. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  38. T. Tieleman and G. Hinton, “Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude,” COURSERA: Neural Networks for Machine Learning 4, 2012. find
  39. J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang et al., “Large scale distributed deep networks,” in Advances in neural information processing systems, 2012, pp. 1223–1231. find
  40. S. Pal, E. Ebrahimi, A. Zulfiqar, Y. Fu, V. Zhang, S. Migacz, D. Nellans, and P. Gupta, “Optimizing multi-gpu parallelization strategies for deep learning training,” IEEE Micro, vol. 39, no. 5, pp. 91–101, 2019. find
  41. Z. Jia, S. Lin, C. R. Qi, and A. Aiken, “Exploring the hidden dimension in accelerating convolutional neural networks,” 2018. find
  42. J. Geng, D. Li, and S. Wang, “Horizontal or vertical?: A hybrid approach to large-scale distributed machine learning,” in Proceedings of the 10th Workshop on Scientific Cloud Computing. ACM, 2019, pp. 1–4. find
  43. A. Krizhevsky, “One weird trick for parallelizing convolutional neural networks,” arXiv preprint arXiv:1404.5997, 2014.
  44. Baidu-allreduce, 2018, https://github.com/baidu-research/baidu-allreduce. ↗
  45. X. Jia, S. Song, W. He, Y. Wang, H. Rong, F. Zhou, L. Xie, Z. Guo, Y. Yang, L. Yu et al., “Highly scalable deep learning training system with mixed-precision: Training imagenet in four minutes,” arXiv preprint arXiv:1807.11205, 2018.
  46. N. Arivazhagan, A. Bapna, O. Firat, D. Lepikhin, M. Johnson, M. Krikun, M. X. Chen, Y. Cao, G. Foster, C. Cherry et al., “Massively multilingual neural machine translation in the wild: Findings and challenges,” arXiv preprint arXiv:1907.05019, 2019.
  47. Byteps, A high performance and generic framework for distributed DNN training, 2019, https://github.com/bytedance/byteps. ↗
  48. Y. You, J. Hseu, C. Ying, J. Demmel, K. Keutzer, and C.-J. Hsieh, “Large-batch training for lstm and beyond,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2019, pp. 1–16. find
  49. T. D. Le, T. Sekiyama, Y. Negishi, H. Imai, and K. Kawachiya, “Involving cpus into multi-gpu deep learning,” in Proceedings of the 2018 ACM/SPEC International Conference on Performance Engineering. ACM, 2018, pp. 56–67. find
  50. Z. Jia, M. Zaharia, and A. Aiken, “Beyond data and model parallelism for deep neural networks,” arXiv preprint arXiv:1807.05358, 2018.
  51. Z. Huo, B. Gu, Q. Yang, and H. Huang, “Decoupled parallel backpropagation with convergence guarantee,” arXiv preprint arXiv:1804.10574, 2018. — [arXiv:1804.10574](https://arxiv.org/abs/1804.10574) find
  52. J. Geng, D. Li, and S. Wang, “Rima: An rdma-accelerated model-parallelized solution to large-scale matrix factorization,” in 2019 IEEE 35th International Conference on Data Engineering (ICDE). IEEE, 2019, pp. 100–111. find
  53. C.-C. Chen, C.-L. Yang, and H.-Y. Cheng, “Efficient and robust parallel dnn training through model parallelism on multi-gpu platform,” arXiv preprint arXiv:1809.02839, 2018. find
  54. N. Dryden, N. Maruyama, T. Moon, T. Benson, M. Snir, and B. Van Essen, “Channel and filter parallelism for large-scale cnn training,” in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2019, pp. 1–20. find
  55. J. Geng, D. Li, and S. Wang, “Elasticpipe: An efficient and dynamic model-parallel solution to dnn training,” in Proceedings of the 10th Workshop on Scientific Cloud Computing. ACM, 2019, pp. 5–9. find
  56. L. Guan, W. Yin, D. Li, and X. Lu, “Xpipe: Efficient pipeline model parallelism for multi-gpu dnn training,” arXiv preprint arXiv:1911.04610, 2019. find
  57. A. Mirhoseini, H. Pham, Q. V. Le, B. Steiner, R. Larsen, Y. Zhou, N. Kumar, M. Norouzi, S. Bengio, and J. Dean, “Device placement optimization with reinforcement learning,” in Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017, pp. 2430–2439. find
  58. J. Gu, M. Chowdhury, K. G. Shin, Y. Zhu, M. Jeon, J. Qian, H. Liu, and C. Guo, “Tiresias: A {GPU} cluster manager for distributed deep learning,” in 16th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 19), 2019, pp. 485–500. find
  59. M. Wang, C.-c. Huang, and J. Li, “Supporting very large models using automatic dataflow graph partitioning,” in Proceedings of the Fourteenth EuroSys Conference 2019, 2019, pp. 1–17. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/fan-2020-dapple-pipelined-data-parallel-approach/