Elamur Research⌕
☰
home / papers / 2018 / Horovod: fast and easy distributed deep learning in TensorFlow

Horovod: fast and easy distributed deep learning in TensorFlow

Paper · 2018 · Alexander Sergeev and Mike Del Balso

Summary

Moving TensorFlow training from one GPU to many is often blocked by two problems. The built-in distributed mode, which relies on parameter servers, forces developers to restructure their code and learn many unfamiliar concepts. Its communication costs also waste hardware: at Uber, about half the capacity of 128 GPUs was lost to overhead. The authors present Horovod, an open-source Python library (Apache 2.0) from Uber's Michelangelo platform. It performs synchronous data-parallel training, with gradients averaged by ring-allreduce instead of parameter servers. Starting from Baidu's draft implementation, the team made it a standalone package independent of the TensorFlow version, switched the communication backend to NVIDIA's NCCL, and added support for several GPUs per server. They also added a variable broadcast so every worker starts from the same state. MPI launches the processes, and users change only a few lines of code. On Inception V3 and ResNet-101 benchmarks, Horovod reached roughly 88% scaling efficiency, about twice the throughput of standard distributed TensorFlow. This makes multi-GPU training practical for ordinary model developers.

Key points

  • Standard distributed TensorFlow on 128 NVIDIA Pascal GPUs used only about half of the available resources for Inception V3 and ResNet-101. Horovod raised scaling efficiency to about 88% over 25GbE TCP.
  • The ring-allreduce approach follows Patarasuk and Yuan (2009) and Baidu's 2017 work. Each of N nodes exchanges chunks with two neighbours 2·(N−1) times, and the scheme is bandwidth-optimal when buffers are large.
  • Tensor Fusion packs small tensors of the same type into a shared buffer (64 MB by default) before reduction. It gave up to a 65% speedup for models with many layers on an unoptimized TCP network.
  • RDMA networking added only 3–4% for Inception V3 and ResNet-101, though it pushed efficiency past 90%. VGG-16, which has many parameters but few layers, gained about 30% from RDMA.
  • Packaging Horovod separately from TensorFlow cut installation from roughly an hour to a few minutes. A single-GPU script needs only four Horovod-specific additions: hvd.init(), GPU pinning by local rank, hvd.DistributedOptimizer, and BroadcastGlobalVariablesHook.

Interesting ideas

  • I1. Distribution through an optimizer wrapper. Horovod hides gradient averaging inside a wrapper around any existing optimizer, so the model code stays unchanged. Keeping a complex capability behind one familiar interface is a good way to get adoption for any infrastructure tool.
  • I2. Broadcasting initial state from rank 0. All workers copy their starting variables from a single process before training begins. This is a cheap and reliable way to keep replicas consistent in any parallel job where each copy needs identical starting state.
  • I3. Merging small messages before communication. Tensor Fusion recognises that the reduction algorithm only works well on large buffers, so it batches many small tensors into one. The same batching trick can help wherever per-message overhead dominates, such as network calls or database writes.
  • I4. Measuring against ideal linear scaling. The authors compare throughput with the single-GPU rate multiplied by the GPU count, which shows exactly how much hardware is being wasted. Choosing which model architecture to benchmark matters too: VGG-16's large parameter count exposed a communication bottleneck that the other models hid.
  • I5. Decoupling the tool from the host framework's release cycle. Shipping Horovod as a standalone package let teams on different TensorFlow versions use it without patching or rebuilding the framework. This is a useful pattern for any shared library in an organisation with mixed software versions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksInformation Systems · 1,730 worksHardware and Architecture · 167 works

Cite

Cite as Alexander Sergeev and Mike Del Balso. 2018. Horovod: fast and easy distributed deep learning in TensorFlow. arXiv:1802.05799.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1802.05799 · arXiv 1802.05799

@misc{sergeev2018horovod,
  title = {{Horovod: fast and easy distributed deep learning in TensorFlow}},
  author = {Alexander Sergeev and Mike {Del Balso}},
  year = {2018},
  doi = {10.48550/arxiv.1802.05799},
  eprint = {1802.05799},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1802.05799}
}

References

20 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenber… — TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
  2. François Chollet et al. Keras. https://github.com/fchollet/keras, 2015. [Online; accessed 6-December-2017]. ↗
  3. Jeremy Hermann and Mike Del Balso. Meet Michelangelo: Uber’s machine learning platform. https://eng.uber.com/michelangelo/, 2017. [Online; accessed 6-December-2017]. ↗
  4. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yua… — TensorFlow: A system for large-scale machine learning
  5. TensorFlow Authors. Tensorflow benchmarks. https://github.com/tensorflow/benchmarks, 2017. [Online; accessed 6-December-2017]. ↗
  6. Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training ImageNet in 1 hour, 2017, arXiv:1706.02677. find
  7. Wikipedia. Data parallelism — Wikipedia, the free encyclopedia. https://en.wikipedia.org/w/index.php?title=Data_parallelism&oldid=807618997, 2017. [Online; accessed 6-December-2017]. ↗
  8. Andrew Gibiansky. Bringing HPC techniques to deep learning. http://research.baidu.com/bringing-hpc-techniques-deep-learning, 2017. [Online; accessed 6-December-2017]. ↗
  9. Pitch Patarasuk and Xin Yuan. Bandwidth optimal all-reduce algorithms for clusters of workstations. J. Parallel Distrib. Comput., 69:117–124, 2009. find
  10. MPI Forum. Message Passing Interface (MPI) Forum Home Page. http://www.mpi-forum.org, 2017. [Online; accessed 6-December-2017]. ↗
  11. Edgar Gabriel, Graham E. Fagg, George Bosilca, Thara Angskun, Jack J. Dongarra, Jeffrey M. Squyres, Vishal Sahay, Prabhanjan Kambadur, Brian Barrett, Andrew Lumsdaine, Ralph H. Castain, David J. Daniel, Richard L. Graham, and Timothy S. Woodall. Open MPI: Goals, concept, and design of a next generat… find
  12. Andrew Gibiansky and Joel Hestness. baidu-research/tensorflow-allreduce. https://github.com/baidu-research/tensorflow-allreduce, 2017. [Online; accessed 6-December-2017]. ↗
  13. NVIDIA. NVIDIA collective communications library (NCCL). https://developer.nvidia.com/nccl, 2017. [Online; accessed 6-December-2017]. ↗
  14. TensorFlow Authors. Distributed TensorFlow. https://www.tensorflow.org/deploy/distributed\#putting_it_all_together_example_trainer_program, 2017. [Online; accessed 6-December-2017]. ↗
  15. Alexander Sergeev et al. uber/horovod: Distributed training framework for TensorFlow. https://github.com/uber/horovod, 2017. [Online; accessed 6-December-2017]. ↗
  16. Chromium Authors. The Trace Event Profiling Tool (about:tracing). https://www.chromium.org/developers/how-tos/trace-event-profiling-tool, 2017. [Online; accessed 6-December-2017]. ↗
  17. Alexander Sergeev. TensorFlow Benchmarks. https://github.com/alsrgv/benchmark/tree/horovod_v2, 2017. [Online; accessed 6-December-2017]. ↗
  18. Wikipedia. Remote direct memory access — Wikipedia, the free encyclopedia. https://en.wikipedia.org/w/index.php?title=Remote_direct_memory_access&oldid=812842097, 2017. [Online; accessed 6-December-2017]. ↗
  19. Wikipedia. InfiniBand — Wikipedia, the free encyclopedia. https://en.wikipedia.org/w/index.php?title=InfiniBand&oldid=810171440, 2017. [Online; accessed 6-December-2017]. RDMA over Converged Ethernet — Wikipedia, the free encyclo- ↗
  20. Wikipedia. pedia. https://en.wikipedia.org/w/index.php?title=RDMA_over_Converged_Ethernet&oldid=782744462, 2017. [Online; accessed 6-December-2017]. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sergeev-2018-horovod-fast-easy-distributed-deep/