Elamur Research⌕
☰
home / papers / 2019 / Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads

Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads

Paper · 2019 · Myeongjae Jeon, Shivaram Venkataraman, Amar Phanishayee et al.

Summary

The paper examines how deep learning training behaves on a shared, production GPU cluster at Microsoft. Such clusters differ from classic big-data clusters in two ways: a GPU cannot easily be split among users, and multi-GPU jobs must have all their GPUs allocated at once (gang scheduling). The authors describe Philly, Microsoft's internal training service built on Apache YARN. They analyze a two-month trace of about 100,000 jobs from hundreds of users by combining scheduler logs, Ganglia utilization metrics and per-job logs. They focus on three issues:

  • how gang scheduling and locality requirements create queueing delay,
  • how job placement affects actual GPU utilization,
  • why jobs fail.

GPUs that are allocated are busy only about half the time on average. Roughly 30% of jobs end unsuccessfully. From these findings the authors derive design guidelines for future DNN cluster schedulers and publish the scheduler trace, which matters because public data on real ML training workloads is scarce.

Key points

  • Cluster setup: Philly runs on YARN with HDFS storage and Docker containers. Each production group gets a virtual cluster with a GPU quota, managed by the Fair Scheduler. Preemption starts only once about 90% of all GPUs are in use.
  • Locality-aware placement: The scheduler tries to put a job's GPUs on as few servers as possible, inside one RDMA domain (100-Gbps InfiniBand within a rack, Ethernet across racks). It waits a timeout of 2–3 minutes before relaxing these constraints.
  • Cluster growth: The cluster grew fivefold in one year, and servers moved from 2-GPU to 8-GPU configurations.
  • Low real utilization: Average hardware utilization of allocated GPUs is only around 52%. The authors attribute this to two causes:
    • jobs spread across servers pay synchronization costs,
    • jobs packed onto the same server interfere with each other.
  • Failures: About 30% of jobs are killed or fail.
    • Programming errors dominate and appear early in training.
    • Infrastructure failures, such as HDFS problems, tend to surface much later.

Interesting ideas

  • I1. Look past allocation numbers. A cluster where nearly every GPU is assigned can still waste much of its hardware. Tracking actual device utilization alongside allocation is a cheap check that applies to any shared compute pool.
  • I2. Prefer locality over a faster start. For long-running jobs, it can pay to wait longer in the queue in exchange for good placement, since poor placement slows the whole run. The same reasoning fits any batch system where startup delay is small relative to runtime.
  • I3. Smoke-test jobs before scaling. Run each new job briefly on a small dedicated pool, or even a single GPU, before giving it many shared GPUs. This catches code and configuration bugs cheaply and fits any expensive experiment pipeline, including NLP training runs.
  • I4. Retry based on failure type. Classify failures at runtime and adjust retries accordingly: stop retrying on bad inputs, keep retrying on network timeouts. This avoids wasting resources on reruns that are certain to fail.
  • I5. Join several log sources and release the trace. Linking scheduler, monitoring and per-job logs gives a full view of each job's lifecycle. Publishing the resulting anonymized trace lets others test scheduling ideas on real workloads.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States378 works · 161,630 citations🎓Beihang UniversityBeijing · China86 works · 9,097 citations🏢Microsoft Research (United Kingdom)Cambridge · United Kingdom11 works · 3,215 citations🎓University of Wisconsin SystemMadison · United States2 works · 179 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksInformation Systems · 1,730 worksHardware and Architecture · 167 worksComputer Networks and Communications · 100 works

🧭 Topics 1Research subjects and languages the work is about.GPU & parallel computing · 41 works

Cite

Cite as Myeongjae Jeon, Shivaram Venkataraman, Amar Phanishayee, Junjie Qian, Wencong Xiao and Fan Yang. 2019. Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads. arXiv:1901.05758.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1901.05758 · arXiv 1901.05758

@misc{jeon2019analysis,
  title = {{Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads}},
  author = {Myeongjae Jeon, Shivaram Venkataraman, Amar Phanishayee, Junjie Qian, Wencong Xiao and Fan Yang},
  year = {2019},
  doi = {10.48550/arxiv.1901.05758},
  eprint = {1901.05758},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1901.05758}
}

References

56 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Deep Learning for Siri’s Voice. https://machinelearning.apple.com/2017/08/06/siri-voices.html. ↗
  2. Hadoop: Fair Scheduler. https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/FairScheduler.html. ↗
  3. NVIDIA Management Library. https://developer.nvidia.com/nvidia-management-library-nvml. ↗
  4. Using Deep Learning to Create Professional-Level Photographs. https://research.googleblog.com/2017/07/using-deep-learning-to-create.html. ↗
  5. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. TensorFlow: A System for Large-Scale Machine Learning. In OSDI, 2016.
  6. Sameer Agarwal, Barzan Mozafari, Aurojit Panda, Henry Milner, Samuel Madden, and Ion Stoica. BlinkDB: Queries with Bounded Errors and Bounded Response Times on Very Large Data. In EuroSys, 2013. find
  7. Ganesh Ananthanarayanan, Sameer Agarwal, Srikanth Kandula, Albert Greenberg, Ion Stoica, Duke Harlan, and Ed Harris. Scarlett: Coping with Skewed Content Popularity in Mapreduce Clusters. In EuroSys, 2011. find
  8. Ganesh Ananthanarayanan, Michael Chien-Chun Hung, Xiaoqi Ren, Ion Stoica, Adam Wierman, and Minlan Yu. GRASS: Trimming Stragglers in Approximation Analytics. In NSDI, 2014. find
  9. Brian Babcock, Surajit Chaudhuri, and Gautam Das. Dynamic Sample Selection for Approximate Query Processing. In SIGMOD, 2003. find
  10. Luiz André Barroso, Jeffrey Dean, and Urs Hölzle. Web Search for a Planet: The Google Cluster Architecture. IEEE Micro, 23(2):22–28, March 2003. find
  11. Eric Boutin, Jaliya Ekanayake, Wei Lin, Bing Shi, Jingren Zhou, Zhengping Qian, Ming Wu, and Lidong Zhou. Apollo: Scalable and Coordinated Scheduling for Cloud-scale Computing. In OSDI, 2014. find
  12. Ronnie Chaiken, Bob Jenkins, Per-Åke Larson, Bill Ramsey, Darren Shakib, Simon Weaver, and Jingren Zhou. SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets. VLDB, 2008. find
  13. Quan Chen, Hailong Yang, Minyi Guo, Ram Srivatsa Kannan, Jason Mars, and Lingjia Tang. Prophet: Precise QoS Prediction on Non-Preemptive Accelerators to Improve Utilization in Warehouse-Scale Computers. In ASPLOS, 2017. find
  14. Quan Chen, Hailong Yang, Jason Mars, and Lingjia Tang. Baymax: QoS Awareness and Increased Utilization for Non-Preemptive Accelerators in Warehouse Scale Computers. In ASPLOS, 2016. find
  15. Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang. MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems. arXiv preprint arXiv:1512.01274, 2015.
  16. Tyson Condie, Neil Conway, Peter Alvaro, Joseph M. Hellerstein, John Gerth, Justin Talbot, Khaled Elmeleegy, and Russell Sears. Online Aggregation and Continuous Query Support in MapReduce. In SIGMOD, 2010. find
  17. Jeffrey Dean and Luiz André Barroso. The Tail at Scale. Commun. ACM, 56(2):74–80, February 2013. find
  18. Dror G Feitelson. Packing schemes for gang scheduling. In Workshop on Job Scheduling Strategies for Parallel Processing, pages 89–110. Springer, 1996. find
  19. Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep Learning, volume 1. MIT press Cambridge, 2016. find
  20. Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour. arXiv preprint arXiv:1706.02677, 2017. find
  21. Juncheng Gu, Kang G. Chowdhury, Mosharaf abd Shin, Yibo Zhu, Myeongjae Jeon, Junjie Qian, Hongqiang Liu, and Chuanxiong Guo. Tiresias: A GPU Cluster Manager for Distributed Deep Learning. In NSDI, 2019. find
  22. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR, 2016.
  23. Joseph M. Hellerstein, Peter J. Haas, and Helen J. Wang. Online Aggregation. In SIGMOD, 1997. find
  24. ImageNet, 2016. http://image-net.org. ↗
  25. Michael Isard, Vijayan Prabhakaran, Jon Currey, Udi Wieder, Kunal Talwar, and Andrew Goldberg. Quincy: Fair Scheduling for Distributed Computing Clusters. In SOSP, 2009. find
  26. Myeongjae Jeon, Shivaram Venkataraman, Amar Phanishayee, Junjie Qian, Wencong Xiao, and Fan Yang. Multi-tenant GPU Clusters for Deep Learning Workloads: Analysis and Implications. Technical Report MSR-TR-2018-13, 2018. find
  27. Chris Jermaine, Subramanian Arumugam, Abhijit Pol, and Alin Dobra. Scalable Approximate Query Processing with the DBO Engine. ACM Trans. Database Syst., 33(4):23:1–23:54, December 2008. find
  28. Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. Caffe: Convolutional Architecture for Fast Feature Embedding. In MM, 2014.
  29. Soila Kavulya, Jiaqi Tan, Rajeev Gandhi, and Priya Narasimhan. An Analysis of Traces from a Production MapReduce Cluster. In CCGRID ’10, 2010. find
  30. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet Classification with Deep Convolutional Neural Networks. In NIPS, 2012.
  31. Mu Li, David G Andersen, Jun Woo Park, Alexander J Smola, Amr Ahmed, Vanja Josifovski, James Long, Eugene J Shekita, and Bor-Yiing Su. Scaling Distributed Machine Learning with the Parameter Server. In OSDI, 2014. find
  32. Matthew L Massie, Brent N Chun, and David E Culler. The Ganglia Distributed Monitoring System: Design, Implementation And Experience. Parallel Computing, 30(7):817–840, 2004. find
  33. Avner May, Alireza Bagheri Garakani, Zhiyun Lu, Dong Guo, Kuan Liu, Aurélien Bellet, Linxi Fan, Michael Collins, Daniel Hsu, Brian Kingsbury, et al. Kernel Approximation Methods for Speech Recognition. arXiv preprint arXiv:1701.03577, 2017. — [arXiv:1701.03577](https://arxiv.org/abs/1701.03577) find
  34. Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur. Recurrent Neural Network Based Language Model. In Eleventh Annual Conference of the International Speech Communication Association, 2010. find
  35. Open Platform for AI, 2018. https://github.com/microsoft/pai. ↗
  36. Kay Ousterhout, Patrick Wendell, Matei Zaharia, and Ion Stoica. Sparrow: Distributed, Low Latency Scheduling. In SOSP, 2013. find
  37. Yanghua Peng, Yixin Bao, Yangrui Chen, Chuan Wu, and Chuanxiong Guo. Optimus: An Efficient Dynamic Resource Scheduler for Deep Learning Clusters. In EuroSys, 2018. find
  38. DNN training workloads on Microsoft’s internal Philly clusters, 2019. https://github.com/msr-fiddle/philly-traces. ↗
  39. PyTorch, 2018. https://pytorch.org/. ↗
  40. Minsoo Rhu, Natalia Gimelshein, Jason Clemons, Arslan Zulfiqar, and Stephen W Keckler. vDNN: Virtualized Deep Neural Networks for Scalable, Memory-efficient Neural Network Design. In MICRO, 2016.
  41. Malte Schwarzkopf, Andy Konwinski, Michael Abd-El-Malek, and John Wilkes. Omega: Flexible, Scalable Schedulers for Large Compute Clusters. In EuroSys, 2013. find
  42. Frank Seide and Amit Agarwal. CNTK: Microsoft’s Open-Source Deep-Learning Toolkit. In KDD, 2016. find
  43. Konstantin Shvachko, Hairong Kuang, Sanjay Radia, and Robert Chansler. The Hadoop Distributed File System. In MSST, 2010. find
  44. Apache Hadoop 3.2.0 Submarine, 2019. https://hadoop.apache.org/docs/r3.2.0/hadoop-yarn/hadoop-yarn-applications/ hadoop-yarn-submarine/. ↗
  45. Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. LSTM Neural Networks for Language Modeling. In Thirteenth Annual Conference of the International Speech Communication Association, 2012. find
  46. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going Deeper With Convolutions. In CVPR, 2015. find
  47. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In NIPS, 2017.
  48. Vinod Kumar Vavilapalli, Arun C Murthy, Chris Douglas, Sharad Agarwal, Mahadev Konar, Robert Evans, Thomas Graves, Jason Lowe, Hitesh Shah, Siddharth Seth, et al. Apache Hadoop YARN: Yet Another Resource Negotiator. In SoCC, 2013. find
  49. Shivaram Venkataraman, Aurojit Panda, Ganesh Ananthanarayanan, Michael J. Franklin, and Ion Stoica. The Power of Choice in Data-aware Cluster Scheduling. In OSDI, 2014. find
  50. Abhishek Verma, Luis Pedrosa, Madhukar Korupolu, David Oppenheimer, Eric Tune, and John Wilkes. Large-scale Cluster Management at Google with Borg. In EuroSys, 2015. find
  51. Linnan Wang, Jinmian Ye, Yiyang Zhao, Wei Wu, Ang Li, Shuaiwen Leon Song, Zenglin Xu, and Tim Kraska. Superneurons: Dynamic GPU Memory Management for Training Deep Neural Networks. In PPoPP, 2018. find
  52. Wencong Xiao, Romil Bhardwaj, Ramachandran Ramjee, Muthian Sivathanu, Nipun Kwatra, Zhenhua Han, Pratyush Patel, Xuan Peng, Hanyu Zhao, Quanlu Zhang, Fan Yang, Lidong Zhou. Gandiva: Introspective Cluster Scheduling for Deep Learning. In OSDI, 2018. find
  53. Hangchen Yu and Christopher J. Rossbach. Full Virtualization for GPUs Reconsidered. In Workshop on Duplicating, Deconstructing, and Debunking (WDDD), 2017. find
  54. Matei Zaharia, Dhruba Borthakur, Joydeep Sen Sarma, Khaled Elmeleegy, Scott Shenker, and Ion Stoica. Delay Scheduling: A Simple Technique for Achieving Locality and Fairness in Cluster Scheduling. In EuroSys, 2010. find
  55. Kai Zeng, Sameer Agarwal, and Ion Stoica. iOLAP: Managing Uncertainty for Efficient Incremental OLAP. In SIGMOD, 2016. find
  56. Haoyu Zhang, Logan Stafman, Andrew Or, and Michael J. Freedman. SLAQ: Quality-driven Scheduling for Distributed Machine Learning. In SoCC, 2017. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jeon-2019-analysis-large-scale-multi-tenant/