Elamur Research⌕
☰
home / papers / 2018 / Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis

Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis

Paper · 2018 · Tal Ben-Nun and Torsten Hoefler

Summary

Training modern deep neural networks has become so computationally heavy that it effectively requires high-performance clusters. However, the many ways to parallelize this work are scattered across machine learning, systems and hardware research. Ben-Nun and Hoefler organize this landscape in a survey of 240 works, gathered by following citations backward from seminal papers (as far back as 1984) and adding results from keyword searches on Google Scholar and arXiv.

They first set up foundations: supervised learning, SGD, the Work-Depth model and MPI. They then discuss how concurrency trades off against generalization. From there they move up the stack:

  • concurrency inside single operators;
  • concurrency across whole networks during inference and training;
  • distributed training, covering asynchronous optimization, parameter-server versus decentralized designs, communication reduction and parallel neural architecture search.

Where feasible, each approach is characterized by its work, depth and average parallelism, which gives a common yardstick for comparison. The central conclusion is that almost every level of deep learning exposes parallelism. Loosening consistency requirements can unlock even more of it while keeping accuracy acceptable. This makes the survey a practical map for choosing a scaling strategy.

Key points

  • The survey covers 240 papers. Coverage of arXiv may be incomplete because deep learning output there grows roughly quadratically (cs.CV went from 577 to 5,693 yearly papers in Table 1).
  • Concurrency is analyzed with the Work-Depth (W-D) model across four levels: operators, networks, training, and distributed systems with agents and parameter servers.
  • Genetic-algorithm architecture search has run at very large scale: 18,000 nodes of the Titan supercomputer for 24 hours.
  • Regularized Evolution (AmoebaNets) reached 3.8% error on ImageNet and 2.13% on CIFAR-10. The best DenseNet instances reached 5.29% and 3.62%.
  • Whole-graph compilers (TensorFlow XLA, Tensor Comprehensions, Latte, TVM) report about 4× speedup over hand-tuned individual operators. The authors expect future work to target inter-layer and whole-DNN optimization.

Interesting ideas

  • I1. A shared cost model. Describing very different techniques with one abstract model (work, depth, average parallelism) makes them directly comparable. The same approach could rank data pipelines or NLP preprocessing steps by how well they parallelize.
  • I2. Relaxed consistency. Because nonlinear optimization tolerates noise, strict synchronization can often be loosened to gain concurrency at small accuracy cost. This is worth testing whenever a sequential bottleneck appears in iterative training.
  • I3. Tournament selection. Comparing population members in pairs replaces a global, synchronous "pick the fittest" step. The general trick is useful for any parallel search or hyperparameter sweep that would otherwise stall on a global reduction.
  • I4. Age-based removal. Regularized Evolution drops the oldest candidate each iteration instead of the worst. This cheap rule regularizes the search and could be applied to other evolutionary or population-based model selection.
  • I5. Literature-collection method. The survey combines recursive tracking of citations from seminal papers with keyword searches and publishes the categorized list online. This is a reusable, transparent recipe for building literature reviews in a shared research library.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓ETH ZurichZurich · Switzerland84 works · 9,181 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 worksHardware and Architecture · 167 works

Cite

Cite as Tal Ben-Nun and Torsten Hoefler. 2018. Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis. ACM Computing Surveys 2019.

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 1802.09941

@inproceedings{bennun2018demystifying,
  title = {{Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis}},
  author = {Tal Ben-Nun and Torsten Hoefler},
  booktitle = {ACM Computing Surveys 2019},
  year = {2018},
  eprint = {1802.09941},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1802.09941}
}

References

269 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. Abadi et al. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. (2015). http://www.tensorflow.org ↗
  2. A. Agarwal and J. C. Duchi. 2011. Distributed Delayed Stochastic Optimization. In Advances in Neural Information Processing Systems 24. 873–881. find
  3. A. F. Aji and K. Heafield. 2017. Sparse Communication for Distributed Gradient Descent. (2017). arXiv:1704.05021 find
  4. F. Akopyan et al. 2015. TrueNorth: Design and Tool Flow of a 65 mW 1 Million Neuron Programmable Neurosynaptic Chip. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 34, 10 (2015), 1537–1557. find
  5. D. Alistarh, D. Grubic, J. Li, R. Tomioka, and M. Vojnovic. 2017. QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding. In Advances in Neural Information Processing Systems 30. 1709–1720.
  6. D. Amodei et al. 2016. Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin. In Proc. 33rd International Conference on Machine Learning, Vol. 48. 173–182.
  7. J. Appleyard, T. Kociský, and P. Blunsom. 2016. Optimizing Performance of Recurrent Neural Networks on GPUs. (2016). arXiv:1604.01946 find
  8. N. S. Arora, R. D. Blumofe, and C. G. Plaxton. 1998. Thread Scheduling for Multiprogrammed Multiprocessors. In Proc. Tenth Annual ACM Symposium on Parallel Algorithms and Architectures (SPAA ’98). 119–129. find
  9. A. A. Awan, K. Hamidouche, J. M. Hashmi, and D. K. Panda. 2017. S-Caffe: Co-designing MPI Runtimes and Caffe for Scalable Deep Learning on Modern GPU Clusters. In Proc. 22nd ACM SIGPLAN Symp. on PPoPP. 193–205. find
  10. J. Ba and R. Caruana. 2014. Do Deep Nets Really Need to be Deep? In Advances in Neural Information Processing Systems 27. 2654–2662. find
  11. J. Ba, R. Grosse, and J. Martens. 2017. Distributed Second-Order Optimization using Kronecker-Factored Approximations. In International Conference on Learning Representations (ICLR). find
  12. B. Baker, O. Gupta, N. Naik, and R. Raskar. 2017. Designing Neural Network Architectures using Reinforcement Learning. In International Conference on Learning Representations (ICLR). find
  13. B. Baker, O. Gupta, R. Raskar, and N. Naik. 2017. Practical Neural Network Performance Prediction for Early Stopping. (2017). arXiv:1705.10823 — [arXiv:1705.10823](https://arxiv.org/abs/1705.10823) find
  14. J. Ballé, V. Laparra, and E. P. Simoncelli. 2017. End-to-end optimized image compression. In International Conference on Learning Representations (ICLR). find
  15. R. Belli and T. Hoefler. 2015. Notified Access: Extending Remote Memory Access Programming Models for Producer-Consumer Synchronization. In Proc. 29th IEEE Int’l Parallel & Distributed Processing Symposium (IPDPS’15). find
  16. T. Ben-Nun, E. Levy, A. Barak, and E. Rubin. 2015. Memory Access Patterns: The Missing Piece of the Multi-GPU Puzzle. In Proc. Int’l Conf. for High Performance Computing, Networking, Storage and Analysis (SC ’15). 19:1–19:12. find
  17. Y. Bengio. 2013. Deep Learning of Representations: Looking Forward. In Statistical Language and Speech Processing, SLSP. Proceedings. find
  18. Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle. 2007. Greedy Layer-Wise Training of Deep Networks. In Advances in Neural Information Processing Systems 19. 153–160. find
  19. Y. Bengio, P. Simard, and P. Frasconi. 1994. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks 5, 2 (1994), 157–166. find
  20. P. Blanchard, E. M. El Mhamdi, R. Guerraoui, and J. Stainer. 2017. Machine Learning with Adversaries: Byzantine Tolerant Gradient Descent. In Advances in Neural Information Processing Systems 30. 119–129. find
  21. R. D. Blumofe and C. E. Leiserson. 1999. Scheduling Multithreaded Computations by Work Stealing. J. ACM 46, 5 (1999), 720–748. find
  22. M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, X. Zhang, J. Zhao, and K. Zieba. 2016. End to End Learning for Self-Driving Cars. (2016). arXiv:1604.07316 — [arXiv:1604.07316](https://arxiv.org/abs/1604.07316) find
  23. L. Bottou, F. E. Curtis, and J. Nocedal. 2016. Optimization Methods for Large-Scale Machine Learning. (2016). arXiv:1606.04838 — [arXiv:1606.04838](https://arxiv.org/abs/1606.04838) find
  24. S. Boyd, A. Ghosh, B. Prabhakar, and D. Shah. 2005. Gossip algorithms: design, analysis and applications. In Proceedings IEEE 24th Annual Joint Conference of the IEEE Computer and Communications Societies., Vol. 3. 1653–1664. find
  25. S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein. 2011. Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers. Found. Trends Mach. Learn. 3, 1 (2011), 1–122. find
  26. R. P. Brent. 1974. The Parallel Evaluation of General Arithmetic Expressions. J. ACM 21, 2 (1974), 201–206. find
  27. A. Brock, T. Lim, J. M. Ritchie, and N. Weston. 2017. SMASH: One-Shot Model Architecture Search through HyperNetworks. (2017). arXiv:1708.05344 — [arXiv:1708.05344](https://arxiv.org/abs/1708.05344) find
  28. R. H. Byrd, S. L. Hansen, J. Nocedal, and Y. Singer. 2016. A Stochastic Quasi-Newton Method for Large-Scale Optimization. SIAM Journal on Optimization 26, 2 (2016), 1008–1031. find
  29. E. Chan, M. Heimlich, A. Purkayastha, and R. van de Geijn. 2007. Collective Communication: Theory, Practice, and Experience: Research Articles. Concurr. Comput. : Pract. Exper. 19, 13 (2007), 1749–1783. find
  30. W. Chan, N. Jaitly, Q. Le, and O. Vinyals. 2016. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In IEEE Int’l Conf. on Acoustics, Speech and Signal Processing (ICASSP). 4960–4964. find
  31. K. Chellapilla, S. Puri, and P. Simard. 2006. High Performance Convolutional Neural Networks for Document Processing. In Tenth International Workshop on Frontiers in Handwriting Recognition. find
  32. C.-Y. Chen, J. Choi, D. Brand, A. Agrawal, W. Zhang, and K. Gopalakrishnan. 2017. AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training. (2017). arXiv:1712.02679 — [arXiv:1712.02679](https://arxiv.org/abs/1712.02679) find
  33. K. Chen and Q. Huo. 2016. Scalable training of deep learning machines by incremental block training with intra-block parallel optimization and blockwise model-update filtering. In IEEE ICASSP. 5880–5884. find
  34. T. Chen et al. 2018. TVM: End-to-End Optimization Stack for Deep Learning. (2018). arXiv:1802.04799
  35. T. Chen, Z. Du, N. Sun, J. Wang, C. Wu, Y. Chen, and O. Temam. 2014. DianNao: A Small-footprint High-throughput Accelerator for Ubiquitous Machine-learning. In Proc. 19th Int’l Conf. on ASPLOS. 269–284. find
  36. T. Chen, B. Xu, C. Zhang, and C. Guestrin. 2016. Training Deep Nets with Sublinear Memory Cost. (2016). arXiv:1604.06174 find
  37. Y. Chen, J. Li, H. Xiao, X. Jin, S. Yan, and J. Feng. 2017. Dual Path Networks. In Advances in Neural Information Processing Systems 30. 4470–4478. find
  38. S. Chetlur et al. 2014. cuDNN: Efficient Primitives for Deep Learning. (2014). arXiv:1410.0759
  39. T. Chilimbi, Y. Suzue, J. Apacible, and K. Kalyanaraman. 2014. Project Adam: Building an Efficient and Scalable Deep Learning Training System. In 11th USENIX Symposium on Operating Systems Design and Implementation. 571–582. find
  40. K. Cho et al. 2014. Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. In Proc. Conference on Empirical Methods in Natural Language Processing (EMNLP). 1724–1734.
  41. F. Chollet. 2016. Xception: Deep Learning with Depthwise Separable Convolutions. (2016). arXiv:1610.02357 find
  42. C. Chu, S. K. Kim, Y. Lin, Y. Yu, G. Bradski, K. Olukotun, and A. Y. Ng. 2007. Map-Reduce for Machine Learning on Multicore. In Advances in Neural Information Processing Systems 19. 281–288. find
  43. I. H. Chung et al. 2017. Parallel Deep Neural Network Training for Big Data on Blue Gene/Q. IEEE Transactions on Parallel and Distributed Systems 28, 6 (2017), 1703–1714. find
  44. D. C. Cireşan, A. Giusti, L. M. Gambardella, and J. Schmidhuber. 2013. Mitosis Detection in Breast Cancer Histology Images with Deep Neural Networks. In Medical Image Computing and Computer-Assisted Intervention. 411–418. find
  45. A. Coates, B. Huval, T. Wang, D. J. Wu, A. Y. Ng, and B. Catanzaro. 2013. Deep Learning with COTS HPC Systems. In Proc. 30th International Conference on Machine Learning - Volume 28 (ICML’13). III–1337–III–1345. find
  46. N. Cohen, O. Sharir, and A. Shashua. 2016. Deep SimNets. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 4782–4791. find
  47. N. Cohen, O. Sharir, and A. Shashua. 2016. On the Expressive Power of Deep Learning: A Tensor Analysis. In 29th Annual Conference on Learning Theory, Vol. 49. 698–728. find
  48. R. Collobert, K. Kavukcuoglu, and C. Farabet. 2011. Torch7: A Matlab-like Environment for Machine Learning. In BigLearn, NIPS Workshop. find
  49. J. Cong and B. Xiao. 2014. Minimizing Computation in Convolutional Neural Networks. In Int’l Conf. on Artificial Neural Networks (ICANN). 281–290. find
  50. M. Courbariaux and Y. Bengio. 2016. BinaryNet: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1. (2016). arXiv:1602.02830
  51. M. Courbariaux, Y. Bengio, and J.-P. David. 2015. BinaryConnect: Training Deep Neural Networks with Binary Weights During Propagations. In Proc. 28th Int’l Conf. on NIPS - Volume 2. 3123–3131.
  52. H. Cui, H. Zhang, G. R. Ganger, P. B. Gibbons, and E. P. Xing. 2016. GeePS: Scalable Deep Learning on Distributed GPUs with a GPU-specialized Parameter Server. In Proc. EuroSys. 4:1–4:16. find
  53. D. Culler, R. Karp, D. Patterson, A. Sahay, K. E. Schauser, E. Santos, R. Subramonian, and T. von Eicken. 1993. LogP: Towards a Realistic Model of Parallel Computation. In Proc. Fourth ACM SIGPLAN Symp. on PPoPP. 1–12. find
  54. J. Daily et al. 2018. GossipGraD: Scalable Deep Learning using Gossip Communication based Asynchronous Gradient Descent. (2018). arXiv:1803.05880 find
  55. C. De Sa, C. Zhang, K. Olukotun, and C. Ré. 2015. Taming the Wild: A Unified Analysis of HOGWILD!-style Algorithms. In Proc. 28th Int’l Conf. on NIPS - Volume 2. 2674–2682. find
  56. J. Dean et al. 2012. Large Scale Distributed Deep Networks. In Proc. 25th International Conference on Neural Information Processing Systems - Volume 1 (NIPS’12). 1223–1231. find
  57. J. Dean and S. Ghemawat. 2008. MapReduce: Simplified Data Processing on Large Clusters. Commun. ACM 51, 1 (2008), 107–113. find
  58. O. Dekel, R. Gilad-Bachrach, O. Shamir, and L. Xiao. 2012. Optimal Distributed Online Prediction Using Mini-batches. JMLR 13, 1 (2012), 165–202. find
  59. O. Delalleau and Y. Bengio. 2011. Shallow vs. Deep Sum-Product Networks. In Advances in Neural Information Processing Systems 24. 666–674. find
  60. J. Demmel and G. Dinh. 2018. Communication-Optimal Convolutional Neural Nets. (2018). arXiv:1802.06905 — [arXiv:1802.06905](https://arxiv.org/abs/1802.06905) find
  61. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. 2009. ImageNet: A Large-Scale Hierarchical Image Database. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). find
  62. L. Deng, D. Yu, and J. Platt. 2012. Scalable stacking and learning for building deep architectures. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2133–2136. find
  63. T. Dettmers. 2015. 8-Bit Approximations for Parallelism in Deep Learning. (2015). arXiv:1511.04561 find
  64. G. Diamos et al. 2016. Persistent RNNs: Stashing Recurrent Weights On-Chip. In Proc. 33rd International Conference on Machine Learning, Vol. 48. 2024–2033. find
  65. T. G. Dietterich. 2000. Ensemble Methods in Machine Learning. In Proc. First International Workshop on Multiple Classifier Systems (MCS ’00). 1–15. find
  66. Z. Drezner and A. Barak. 1986. An asynchronous algorithm for scattering information between the active nodes of a multicomputer system. J. Parallel and Distrib. Comput. 3, 3 (1986), 344–351. find
  67. N. Dryden, T. Moon, S. A. Jacobs, and B. V. Essen. 2016. Communication Quantization for Data-Parallel Training of Deep Neural Networks. In 2nd Workshop on Machine Learning in HPC Environments (MLHPC). 1–8. find
  68. J. Duchi, E. Hazan, and Y. Singer. 2011. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. JMLR 12 (2011), 2121–2159. find
  69. V. Dumoulin and F. Visin. 2016. A guide to convolution arithmetic for deep learning. (2016). arXiv:1603.07285 — [arXiv:1603.07285](https://arxiv.org/abs/1603.07285) find
  70. J. L. Elman. 1990. Finding Structure in Time. Cognitive Science 14, 2 (1990), 179–211. find
  71. T. Elsken, J.-H. Metzen, and F. Hutter. 2017. Simple And Efficient Architecture Search for Convolutional Neural Networks. (2017). arXiv:1711.04528 find
  72. L. Ericson and R. Mbuvha. 2017. On the Performance of Network Parallel Training in Artificial Neural Networks. (2017). arXiv:1701.05130 — [arXiv:1701.05130](https://arxiv.org/abs/1701.05130) find
  73. P. Farber and K. Asanovic. 1997. Parallel neural network training on Multi-Spert. In Proceedings of 3rd International Conference on Algorithms and Architectures for Parallel Processing. 659–666. find
  74. K. Frans, J. Ho, X. Chen, P. Abbeel, and J. Schulman. 2017. Meta Learning Shared Hierarchies. (2017). arXiv:1710.09767 find
  75. M. P. Friedlander and M. W. Schmidt. 2011. Hybrid Deterministic-Stochastic Methods for Data Fitting. (2011). arXiv:1104.2373 — [arXiv:1104.2373](https://arxiv.org/abs/1104.2373) find
  76. A. Gaunt, M. Johnson, M. Riechert, D. Tarlow, R. Tomioka, D. Vytiniotis, and S. Webster. 2017. AMPNet: Asynchronous Model-Parallel Training for Dynamic Neural Networks. (2017). arXiv:1705.09786 find
  77. A. Gholami, A. Azad, P. H. Jin, K. Keutzer, and A. Buluç. 2018. Integrated Model, Batch, and Domain Parallelism in Training Neural Networks. In Proc. 30th on Symp. on Parallelism in Algorithms and Architectures, SPAA 2018. 77–86. find
  78. A. Gibiansky. 2017. Bringing HPC Techniques to Deep Learning. (2017). http://research.baidu.com/bringing-hpc-techniques-deep-learning/ ↗
  79. X. Glorot and Y. Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proc. Thirteenth International Conference on Artificial Intelligence and Statistics, Vol. 9. 249–256. find
  80. D. E. Goldberg and K. Deb. 1991. A Comparative Analysis of Selection Schemes Used in Genetic Algorithms. Foundations of Genetic Algorithms, Vol. 1. 69–93. find
  81. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. 2014. Generative Adversarial Nets. In Advances in Neural Information Processing Systems 27. 2672–2680.
  82. Google. 2017. TensorFlow XLA Overview. (2017). https://www.tensorflow.org/performance/xla ↗
  83. P. Goyal, P. Dollár, R. B. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. 2017. Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour. (2017). arXiv:1706.02677 find
  84. A. Graves et al. 2016. Hybrid computing using a neural network with dynamic external memory. Nature 538, 7626 (2016), 471–476. find
  85. W. Gropp, T. Hoefler, R. Thakur, and E. Lusk. 2014. Using Advanced MPI: Modern Features of the Message-Passing Interface. MIT Press. find
  86. A. Gruslys, R. Munos, I. Danihelka, M. Lanctot, and A. Graves. 2016. Memory-Efficient Backpropagation Through Time. In Advances in Neural Information Processing Systems 29. 4125–4133. find
  87. S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan. 2015. Deep Learning with Limited Numerical Precision. In Proc. 32nd International Conference on Machine Learning, Vol. 37. 1737–1746. find
  88. S. Gupta, W. Zhang, and F. Wang. 2016. Model Accuracy and Runtime Tradeoff in Distributed Deep Learning: A Systematic Study. In IEEE 16th International Conference on Data Mining (ICDM). 171–180. find
  89. S. Hadjis, C. Zhang, I. Mitliagkas, and C. Ré. 2016. Omnivore: An Optimizer for Multi-device Deep Learning on CPUs and GPUs. (2016). arXiv:1606.04487 find
  90. S. Han, H. Mao, and W. J. Dally. 2016. Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding. International Conference on Learning Representations (ICLR) (2016). find
  91. E. Hazan, A. Klivans, and Y. Yuan. 2018. Hyperparameter optimization: a spectral approach. In International Conference on Learning Representations (ICLR). find
  92. K. He, X. Zhang, S. Ren, and J. Sun. 2015. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. In Proc. IEEE International Conference on Computer Vision (ICCV ’15). 1026–1034. find
  93. K. He, X. Zhang, S. Ren, and J. Sun. 2016. Deep Residual Learning for Image Recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 770–778.
  94. X. He, D. Mudigere, M. Smelyanskiy, and M. Takac. 2017. Distributed Hessian-Free Optimization for Deep Neural Network. In AAAI Workshops. find
  95. G. Hinton. 2012. Neural Networks for Machine Learning, Lecture 6a: Overview of Mini-batch Gradient Descent. (2012). find
  96. G. Hinton, O. Vinyals, and J. Dean. 2015. Distilling the Knowledge in a Neural Network. In NIPS Deep Learning and Representation Learning Workshop.
  97. G. E. Hinton, S. Osindero, and Y. W. Teh. 2006. A Fast Learning Algorithm for Deep Belief Nets. Neural Computation 18, 7 (2006), 1527–1554. find
  98. Q. Ho et al. 2013. More Effective Distributed ML via a Stale Synchronous Parallel Parameter Server. In Proc. 26th International Conference on Neural Information Processing Systems - Volume 1 (NIPS’13). 1223–1231. find
  99. S. Hochreiter and J. Schmidhuber. 1997. Long Short-Term Memory. Neural Comput. 9, 8 (1997), 1735–1780. find
  100. T. Hoefler, A. Barak, A. Shiloh, and Z. Drezner. 2017. Corrected Gossip Algorithms for Fast Reliable Broadcast on Unreliable Systems. In Proc. 31st IEEE Int’l Parallel & Distributed Processing Symposium (IPDPS’17). find
  101. T. Hoefler and D. Moor. 2014. Energy, Memory, and Runtime Tradeoffs for Implementing Collective Communication Operations. Journal of Supercomputing Frontiers and Innovations 1, 2 (2014), 58–75. find
  102. T. Hoefler and T. Schneider. 2012. Optimization Principles for Collective Neighborhood Communications. In Proc. International Conference on High Performance Computing, Networking, Storage and Analysis. 98:1–98:10. find
  103. T. Hoefler and M. Snir. 2011. Generic Topology Mapping Strategies for Large-scale Parallel Architectures. In Proc. 2011 ACM International Conference on Supercomputing (ICS’11). 75–85. find
  104. T. Hoefler and J. L. Traeff. 2009. Sparse Collective Operations for MPI. In Proc. 23rd IEEE Int’l Parallel & Distributed Processing Symposium, HIPS’09 Workshop. find
  105. E. Hoffer, I. Hubara, and D. Soudry. 2017. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. In Advances in Neural Information Processing Systems 30. 1729–1739. find
  106. A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. 2017. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. (2017). arXiv:1704.04861
  107. K. Hsieh, A. Harlap, N. Vijaykumar, D. Konomis, G. R. Ganger, P. B. Gibbons, and O. Mutlu. 2017. Gaia: Geo-distributed Machine Learning Approaching LAN Speeds. In Proc. 14th USENIX Conf. on NSDI. 629–647. find
  108. G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger. 2017. Densely connected convolutional networks. In Proc. IEEE Conference on Computer Vision and Pattern Recognition. find
  109. I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio. 2016. Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations. (2016). arXiv:1609.07061
  110. D. A. Huffman. 1952. A Method for the Construction of Minimum-Redundancy Codes. Proc. IRE 40, 9 (1952), 1098–1101. find
  111. F. N. Iandola, M. W. Moskewicz, K. Ashraf, S. Han, W. J. Dally, and K. Keutzer. 2016. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and \<1MB model size. (2016). arXiv:1602.07360 find
  112. F. N. Iandola, M. W. Moskewicz, K. Ashraf, and K. Keutzer. 2016. FireCaffe: Near-Linear Acceleration of Deep Neural Network Training on Compute Clusters. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). find
  113. P. Ienne. 1993. Architectures for Neuro-Computers: Review and Performance Evaluation. Technical Report. EPFL, Lausanne, Switzerland. find
  114. D. J. Im, H. Ma, C. D. Kim, and G. W. Taylor. 2016. Generative Adversarial Parallelization. (2016). arXiv:1612.04021 — [arXiv:1612.04021](https://arxiv.org/abs/1612.04021) find
  115. Intel. 2009. Intel Math Kernel Library. Reference Manual. Intel Corporation. find
  116. Intel. 2017. MKL-DNN. (2017). https://01.org/mkl-dnn ↗
  117. S. Ioffe and C. Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proc. 32nd International Conference on Machine Learning (ICML’15). 448–456. find
  118. M. Jaderberg et al. 2017. Population Based Training of Neural Networks. (2017). arXiv:1711.09846 find
  119. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. 2014. Caffe: Convolutional architecture for fast feature embedding. In Proc. ACM Int’l Conf. on Multimedia. 675–678.
  120. J. Jiang, B. Cui, C. Zhang, and L. Yu. 2017. Heterogeneity-aware Distributed Parameter Servers. In Proc. 2017 ACM International Conference on Management of Data (SIGMOD ’17). 463–478. find
  121. P. H. Jin, Q. Yuan, F. N. Iandola, and K. Keutzer. 2016. How to scale distributed deep learning? ML Systems Workshop at NIPS (2016). find
  122. M. Johnson et al. 2016. Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation. (2016). arXiv:1611.04558
  123. R. Johnson and T. Zhang. 2013. Accelerating Stochastic Gradient Descent using Predictive Variance Reduction. In Advances in Neural Information Processing Systems 26. 315–323. find
  124. N. P. Jouppi et al. 2017. In-Datacenter Performance Analysis of a Tensor Processing Unit. In Proc. 44th Annual International Symposium on Computer Architecture (ISCA ’17). 1–12.
  125. L. Kaiser, A. N. Gomez, N. Shazeer, A. Vaswani, N. Parmar, L. Jones, and J. Uszkoreit. 2017. One Model To Learn Them All. (2017). arXiv:1706.05137 find
  126. T. Karras, T. Aila, S. Laine, and J. Lehtinen. 2017. Progressive Growing of GANs for Improved Quality, Stability, and Variation. (2017). arXiv:1710.10196 — [arXiv:1710.10196](https://arxiv.org/abs/1710.10196) find
  127. J. Keuper and F. Pfreundt. 2015. Asynchronous Parallel Stochastic Gradient Descent: A Numeric Core for Scalable Distributed Machine Learning Algorithms. In Proc. Workshop on MLHPC. 1:1–1:11. find
  128. H. Kim et al. 2016. DeepSpark: Spark-Based Deep Learning Supporting Asynchronous Updates and Caffe Compatibility. (2016). arXiv:1602.08191 find
  129. Y.-D. Kim, E. Park, S. Yoo, T. Choi, L. Yang, and D. Shin. 2016. Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications. In International Conference on Learning Representations (ICLR). find
  130. D. P. Kingma and J. Ba. 2015. Adam: A Method for Stochastic Optimization. In Proc. International Conference on Learning Representations (ICLR).
  131. A. Klein, S. Falkner, J. T. Springenberg, and F. Hutter. 2016. Learning curve prediction with Bayesian neural networks. In International Conference on Learning Representations (ICLR). find
  132. U. Köster et al. 2017. Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks. In Advances in Neural Information Processing Systems 30. 1740–1750. find
  133. S. Krishnan, Y. Xiao, and R. A. Saurous. 2018. Neumann Optimizer: A Practical Optimization Algorithm for Deep Neural Networks. In International Conference on Learning Representations (ICLR). find
  134. Alex Krizhevsky. 2009. Learning Multiple Layers of Features from Tiny Images. Master’s thesis. find
  135. A. Krizhevsky. 2014. One weird trick for parallelizing convolutional neural networks. (2014). arXiv:1404.5997
  136. A. Krizhevsky, I. Sutskever, and G. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems 25. 1097–1105.
  137. T. Kurth et al. 2017. Deep Learning at 15PF: Supervised and Semi-supervised Classification for Scientific Data. In Proc. Int’l Conf. for High Performance Computing, Networking, Storage and Analysis (SC ’17). 7:1–7:11. find
  138. G. Lacey, G. W. Taylor, and S. Areibi. 2016. Deep Learning on FPGAs: Past, Present, and Future. (2016). arXiv:1602.04283 find
  139. L. Lamport, R. Shostak, and M. Pease. 1982. The Byzantine Generals Problem. ACM Trans. Program. Lang. Syst. 4, 3 (1982), 382–401. find
  140. A. Lavin and S. Gray. 2016. Fast Algorithms for Convolutional Neural Networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
  141. Q. V. Le, J. Ngiam, A. Coates, A. Lahiri, B. Prochnow, and A. Y. Ng. 2011. On Optimization Methods for Deep Learning. In Proc. 28th International Conference on Machine Learning (ICML’11). 265–272. find
  142. Q. V. Le, M. Ranzato, R. Monga, M. Devin, K. Chen, G. S. Corrado, J. Dean, and A. Y. Ng. 2012. Building High-level Features Using Large Scale Unsupervised Learning. In Proc. 29th Int’l Conf. on Machine Learning (ICML’12). 507–514. find
  143. Y. LeCun, Y. Bengio, and G. Hinton. 2015. Deep learning. Nature 521, 7553 (2015), 436–444. find
  144. Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. 1989. Backpropagation Applied to Handwritten Zip Code Recognition. Neural Comput. 1, 4 (1989), 541–551. find
  145. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (1998), 2278–2324. find
  146. Y. LeCun and C. Cortes. 1998. The MNIST database of handwritten digits. (1998). http://yann.lecun.com/exdb/mnist ↗
  147. H. Lee, P. Pham, Y. Largman, and A. Y. Ng. 2009. Unsupervised feature learning for audio classification using convolutional deep belief networks. In Advances in Neural Information Processing Systems 22. 1096–1104. find
  148. S. Lee, S. Purushwalkam, M. Cogswell, D. J. Crandall, and D. Batra. 2015. Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks. (2015). arXiv:1511.06314 — [arXiv:1511.06314](https://arxiv.org/abs/1511.06314) find
  149. C. Li, Y. Yang, M. Feng, S. Chakradhar, and H. Zhou. 2016. Optimizing Memory Efficiency for Deep Convolutional Neural Networks on GPUs. In Proc. Int’l Conf. for Supercomputing (SC ’16). 54:1–54:12. find
  150. D. Li, X. Wang, and D. Kong. 2017. DeepRebirth: Accelerating Deep Neural Network Execution on Mobile Devices. (2017). arXiv:1708.04728 — [arXiv:1708.04728](https://arxiv.org/abs/1708.04728) find
  151. F. Li and B. Liu. 2016. Ternary Weight Networks. (2016). arXiv:1605.04711 — [arXiv:1605.04711](https://arxiv.org/abs/1605.04711) find
  152. M. Li et al. 2014. Scaling Distributed Machine Learning with the Parameter Server. In Proc. 11th USENIX Conference on Operating Systems Design and Implementation (OSDI’14). 583–598. find
  153. T. Li, J. Zhong, J. Liu, W. Wu, and C. Zhang. 2017. Ease.ml: Towards Multi-tenant Resource Sharing for Machine Learning Workloads. (2017). arXiv:1708.07308 — [arXiv:1708.07308](https://arxiv.org/abs/1708.07308) find
  154. Y. Li. 2017. Deep Reinforcement Learning: An Overview. (2017). arXiv:1701.07274 — [arXiv:1701.07274](https://arxiv.org/abs/1701.07274) find
  155. X. Lian et al. 2017. Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent. In Advances in Neural Information Processing Systems 30. 5336–5346. find
  156. X. Lian, Y. Huang, Y. Li, and J. Liu. 2015. Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization. In Proc. 28th Int’l Conf. on NIPS - Volume 2. 2737–2745. find
  157. M. Lin, Q. Chen, and S. Yan. 2014. Network In Network. In Proc. Int’l Conf. on Learning Representations (ICLR). find
  158. Y. Lin, S. Han, H. Mao, Y. Wang, and W. J. Dally. 2018. Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training. In Proc. International Conference on Learning Representations (ICLR). find
  159. C. Liu, B. Zoph, J. Shlens, W. Hua, L.-J. Li, L. Fei-Fei, A. Yuille, J. Huang, and K. Murphy. 2017. Progressive Neural Architecture Search. (2017). arXiv:1712.00559 find
  160. H. Liu, K. Simonyan, O. Vinyals, C. Fernando, and K. Kavukcuoglu. 2018. Hierarchical Representations for Efficient Architecture Search. In International Conference on Learning Representations (ICLR). find
  161. H. Liu, K. Simonyan, and Y. Yang. 2018. DARTS: Differentiable Architecture Search. (2018). arXiv:1806.09055 find
  162. X. Liu, J. Pool, S. Han, and W. J. Dally. 2018. Efficient Sparse-Winograd Convolutional Neural Networks. International Conference on Learning Representations (ICLR) (2018). find
  163. P. R. Lorenzo, J. Nalepa, L. S. Ramos, and J. R. Pastor. 2017. Hyper-parameter Selection in Deep Neural Networks Using Parallel Particle Swarm Optimization. In Proc. Genetic and Evolutionary Computation Conf. (GECCO ’17). 1864–1871. find
  164. I. Loshchilov and F. Hutter. 2017. SGDR: Stochastic Gradient Descent with Warm Restarts. In International Conference on Learning Representations (ICLR). find
  165. J. Martens. 2010. Deep Learning via Hessian-free Optimization. In Proc. 27th International Conference on Machine Learning (ICML’10). 735–742. find
  166. M. Mathieu, M. Henaff, and Y. LeCun. 2014. Fast Training of Convolutional Networks through FFTs. International Conference on Learning Representations (ICLR) (2014). find
  167. Message Passing Interface Forum. 2015. MPI: A Message-Passing Interface Standard Version 3.1. (2015). find
  168. Y. Miao, H. Zhang, and F. Metze. 2014. Distributed learning of multilingual DNN feature extractors using GPUs. In INTERSPEECH, 15th Annual Conference of the International Speech Communication Association. 830–834. find
  169. R. Miikkulainen et al. 2017. Evolving Deep Neural Networks. (2017). arXiv:1703.00548 — [arXiv:1703.00548](https://arxiv.org/abs/1703.00548) find
  170. V. Mnih et al. 2015. Human-level control through deep reinforcement learning. Nature 518, 7540 (2015), 529–533. find
  171. V. Mnih et al. 2016. Asynchronous Methods for Deep Reinforcement Learning. In Proc. 33rd International Conference on Machine Learning, Vol. 48. 1928–1937. find
  172. P. Moritz, R. Nishihara, and M. Jordan. 2016. A Linearly-Convergent Stochastic L-BFGS Algorithm. In Proc. 19th International Conference on Artificial Intelligence and Statistics, Vol. 51. 249–258. find
  173. P. Moritz, R. Nishihara, I. Stoica, and M. I. Jordan. 2016. SparkNet: Training Deep Networks in Spark. In Proc. International Conference on Learning Representations (ICLR). find
  174. U. A. Muller and A. Gunzinger. 1994. Neural net simulation on parallel computers. In Neural Networks, IEEE International Conference on, Vol. 6. 3961–3966. find
  175. M. M. Najafabadi, F. Villanustre, T. M. Khoshgoftaar, N. Seliya, R. Wald, and E. Muharemagic. 2015. Deep learning applications and challenges in big data analytics. Journal of Big Data 2, 1 (2015), 1. find
  176. R. Negrinho and G. Gordon. 2017. DeepArchitect: Automatically Designing and Training Deep Architectures. (2017). arXiv:1704.08792 — [arXiv:1704.08792](https://arxiv.org/abs/1704.08792) find
  177. A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro. 2009. Robust Stochastic Approximation Approach to Stochastic Programming. SIAM Journal on Optimization 19, 4 (2009), 1574–1609. find
  178. Y. Nesterov. 1983. A method of solving a convex programming problem with convergence rate O (1/k 2 ) . Soviet Mathematics Doklady 269 (1983), 543–547. find
  179. Netlib. 2017. Basic Linear Algebra Subprograms (BLAS). (2017). http://www.netlib.org/blas ↗
  180. J. Ngiam, Z. Chen, D. Chia, P. W. Koh, Q. V. Le, and A. Y. Ng. 2010. Tiled convolutional neural networks. In Advances in Neural Information Processing Systems 23. 1279–1287. find
  181. J. Nocedal and S. Wright. 2006. Numerical Optimization. find
  182. C. Noel and S. Osindero. 2014. Dogwild!-Distributed Hogwild for CPU & GPU. In NIPS Workshop on Distributed Machine Learning and Matrix Computations. find
  183. E. Nurvitadhi et al. 2017. Can FPGAs Beat GPUs in Accelerating Next-Generation Deep Neural Networks?. In Proc. ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (FPGA ’17). 5–14. find
  184. NVIDIA. 2017. CUBLAS Library Documentation. (2017). http://docs.nvidia.com/cuda/cublas ↗
  185. NVIDIA. 2017. Programming Tensor Cores in CUDA 9. (2017). https://devblogs.nvidia.com/programming-tensor-cores-cuda-9 ↗
  186. C. Olah. 2015. Understanding LSTM Networks. (2015). http://colah.github.io/posts/2015-08-Understanding-LSTMs ↗
  187. Y. Oyama et al. 2016. Predicting statistics of asynchronous SGD parameters for a large-scale distributed deep learning system on GPU supercomputers. In IEEE International Conference on Big Data (Big Data). 66–75. find
  188. Y. Oyama, T. Ben-Nun, T. Hoefler, and S. Matsuoka. 2018. Accelerating Deep Learning Frameworks with Micro-batches. In 2018 IEEE CLUSTER. find
  189. PaddlePaddle. 2017. Elastic Deep Learning. (2017). https://github.com/PaddlePaddle/cloud/tree/develop/doc/edl ↗
  190. T. Paine, H. Jin, J. Yang, Z. Lin, and T. S. Huang. 2013. GPU Asynchronous Stochastic Gradient Descent to Speed Up Neural Network Training. (2013). arXiv:1312.6186 — [arXiv:1312.6186](https://arxiv.org/abs/1312.6186) find
  191. S. J. Pan and Q. Yang. 2010. A Survey on Transfer Learning. IEEE Transactions on Knowledge and Data Engineering 22, 10 (2010), 1345–1359. find
  192. F. Petroski Such et al. 2017. Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning. (2017). arXiv:1712.06567 — [arXiv:1712.06567](https://arxiv.org/abs/1712.06567) find
  193. H. Pham, M. Y. Guan, B. Zoph, Q. V. Le, and J. Dean. 2018. Efficient Neural Architecture Search via Parameter Sharing. (2018). arXiv:1802.03268
  194. B. T. Polyak and A. B. Juditsky. 1992. Acceleration of Stochastic Approximation by Averaging. SIAM J. Control Optim. 30, 4 (1992), 838–855. find
  195. D. Povey, X. Zhang, and S. Khudanpur. 2014. Parallel training of Deep Neural Networks with Natural Gradient and Parameter Averaging. (2014). arXiv:1410.7455 — [arXiv:1410.7455](https://arxiv.org/abs/1410.7455) find
  196. H. Qi, E. R. Sparks, and A. Talwalkar. 2017. Paleo: A Performance Model for Deep Neural Networks. In Proc. International Conference on Learning Representations (ICLR). find
  197. N. Qian. 1999. On the momentum term in gradient descent learning algorithms. Neural Networks 12, 1 (1999). find
  198. R. Rabenseifner. 2004. Optimization of collective reduction operations. In Int’l Conf. on Computational Science. 1–9. find
  199. A. Rahimi and B. Recht. 2017. Reflections on Random Kitchen Sinks. http://www.argmin.net/2017/12/05/kitchen-sinks NIPS Test of Time Award Talk. ↗
  200. R. Raina, A. Madhavan, and A. Y. Ng. 2009. Large-scale Deep Unsupervised Learning Using Graphics Processors. In Proc. 26th Annual International Conference on Machine Learning (ICML ’09). 873–880. find
  201. S. Sundhar Ram, A. Nedic, and V. V. Veeravalli. 2009. Asynchronous gossip algorithms for stochastic optimization. In International Conference on Game Theory for Networks. 80–81. find
  202. M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi. 2016. XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks. (2016). arXiv:1603.05279
  203. E. Real, A. Aggarwal, Y. Huang, and Q. V Le. 2018. Regularized Evolution for Image Classifier Architecture Search. (2018). arXiv:1802.01548 find
  204. E. Real, S. Moore, A. Selle, S. Saxena, Y. L. Suematsu, J. Tan, Q. V. Le, and A. Kurakin. 2017. Large-Scale Evolution of Image Classifiers. In Proc. 34th International Conference on Machine Learning. 2902–2911. find
  205. B. Recht, C. Re, S. Wright, and F. Niu. 2011. Hogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent. In Advances in Neural Information Processing Systems 24. 693–701. find
  206. C. Renggli, D. Alistarh, and T. Hoefler. 2018. SparCML: High-Performance Sparse Communication for Machine Learning. (2018). arXiv:1802.08021 — [arXiv:1802.08021](https://arxiv.org/abs/1802.08021) find
  207. Herbert Robbins and Sutton Monro. 1951. A Stochastic Approximation Method. The Annals of Mathematical Statistics 22, 3 (1951), 400–407. find
  208. T. Salimans and D. P. Kingma. 2016. Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks. In Advances in Neural Information Processing Systems 29. 901–909. find
  209. J. Schmidhuber. 2015. Deep learning in neural networks: An overview. Neural Networks 61 (2015), 85–117. find
  210. F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu. 2014. 1-Bit Stochastic Gradient Descent and Application to Data-Parallel Distributed Training of Speech DNNs. In Interspeech. find
  211. F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu. 2014. On parallelizability of stochastic gradient descent for speech DNNs. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 235–239. find
  212. S. Shalev-Shwartz and S. Ben-David. 2014. Understanding machine learning: From theory to algorithms. Cambridge university press. find
  213. O. Shamir. 2016. Without-Replacement Sampling for Stochastic Gradient Methods. In Advances in Neural Information Processing Systems 29. 46–54. find
  214. R. Shokri and V. Shmatikov. 2015. Privacy-Preserving Deep Learning. In Proc. 22Nd ACM SIGSAC Conference on Computer and Communications Security (CCS ’15). 1310–1321. find
  215. D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al. 2017. Mastering the game of go without human knowledge. Nature 550, 7676 (2017), 354. find
  216. K. Simonyan and A. Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. Int’l Conf. Learning Representations (ICLR).
  217. A. J. R. Simpson. 2015. Instant Learning: Parallel Deep Neural Networks and Convolutional Bootstrapping. (2015). arXiv:1505.05972 — [arXiv:1505.05972](https://arxiv.org/abs/1505.05972) find
  218. S. L. Smith, P. Kindermans, and Q. V. Le. 2017. Don’t Decay the Learning Rate, Increase the Batch Size. (2017). arXiv:1711.00489 find
  219. J. Snoek, H. Larochelle, and R. P Adams. 2012. Practical Bayesian Optimization of Machine Learning Algorithms. In Advances in Neural Information Processing Systems 25. 2951–2959. find
  220. E. Solomonik and T. Hoefler. 2015. Sparse Tensor Algebra as a Parallel Programming Model. (2015). arXiv:1512.00066 — [arXiv:1512.00066](https://arxiv.org/abs/1512.00066) find
  221. M. Song, Y. Hu, H. Chen, and T. Li. 2017. Towards Pervasive and User Satisfactory CNN across GPU Microarchitectures. In IEEE International Symposium on High Performance Computer Architecture (HPCA). 1–12. find
  222. H. V. Sorensen and C. S. Burrus. 1993. Efficient computation of the DFT with only a subset of input or output points. IEEE Transactions on Signal Processing 41, 3 (1993), 1184–1200. find
  223. V. Strassen. 1969. Gaussian Elimination is Not Optimal. Numer. Math. 13, 4 (1969), 354–356. find
  224. N. Strom. 2015. Scalable distributed DNN training using commodity GPU cloud computing. In Sixteenth Annual Conference of the International Speech Communication Association. find
  225. V. Sze, Y. H. Chen, T. J. Yang, and J. S. Emer. 2017. Efficient Processing of Deep Neural Networks: A Tutorial and Survey. Proc. IEEE 105, 12 (2017), 2295–2329.
  226. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. 2015. Going Deeper with Convolutions. In Computer Vision and Pattern Recognition (CVPR). find
  227. G. Taylor, R. Burmeister, Z. Xu, B. Singh, A. Patel, and T. Goldstein. 2016. Training Neural Networks Without Gradients: A Scalable ADMM Approach. (2016). arXiv:1605.02026 find
  228. L. Truong et al. 2016. Latte: A Language, Compiler, and Runtime for Elegant and Efficient Deep Neural Networks. In Proc. 37th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI ’16). 209–223. find
  229. J. Tsitsiklis, D. Bertsekas, and M. Athans. 1986. Distributed asynchronous deterministic and stochastic gradient optimization algorithms. IEEE Trans. Automat. Control 31, 9 (1986), 803–812. find
  230. V. Vanhoucke, A. Senior, and M. Z. Mao. 2011. Improving the speed of neural networks on CPUs. In Deep Learning and Unsupervised Feature Learning Workshop, NIPS. find
  231. N. Vasilache et al. 2018. Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions. (2018). arXiv:1802.04730
  232. N. Vasilache, J. Johnson, M. Mathieu, S. Chintala, S. Piantino, and Y. LeCun. 2015. Fast Convolutional Nets With fbfft: A GPU Performance Evaluation. International Conference on Learning Representations (ICLR) (2015). find
  233. A. Vasudevan, A. Anderson, and D. Gregg. 2017. Parallel Multi Channel Convolution using General Matrix Multiplication. (2017). arXiv:1704.04428 — [arXiv:1704.04428](https://arxiv.org/abs/1704.04428) find
  234. P. Verbancsics and J. Harguess. 2015. Image Classification Using Generative Neuro Evolution for Deep Learning. In IEEE Winter Conference on Applications of Computer Vision. 488–493. find
  235. A. Viebke, S. Memeti, S. Pllana, and A. Abraham. 2017. CHAOS: a parallelization scheme for training convolutional neural networks on Intel Xeon Phi. The Journal of Supercomputing (2017). find
  236. H. Wang, B. Raj, and E. P. Xing. 2017. On the Origin of Deep Learning. (2017). arXiv:1702.07800 — [arXiv:1702.07800](https://arxiv.org/abs/1702.07800) find
  237. W. Wen, C. Xu, F. Yan, C. Wu, Y. Wang, Y. Chen, and H. Li. 2017. TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning. In Advances in Neural Information Processing Systems 30. 1509–1519. find
  238. P. J. Werbos. 1990. Backpropagation through time: what it does and how to do it. Proc. IEEE 78, 10 (1990), 1550–1560. find
  239. J. H. Wilkinson. 1994. Rounding Errors in Algebraic Processes. Dover Publications, Inc. find
  240. R. J. Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning 8, 3 (1992), 229–256. find
  241. S. Winograd. 1980. Arithmetic Complexity of Computations. Society for Industrial and Applied Mathematics. find
  242. L. Xie and A. Yuille. 2017. Genetic CNN. In IEEE International Conference on Computer Vision (ICCV). 1388–1397. find
  243. P. Xie, J. K. Kim, Y. Zhou, Q. Ho, A. Kumar, Y. Yu, and E. Xing. 2016. Lighter-communication Distributed Machine Learning via Sufficient Factor Broadcasting. In Proc. 32nd Conf. on Uncertainty in Artificial Intelligence (UAI’16). 795–804. find
  244. E. P. Xing, Q. Ho, W. Dai, J. K. Kim, J. Wei, S. Lee, X. Zheng, P. Xie, A. Kumar, and Y. Yu. 2015. Petuum: A New Platform for Distributed Machine Learning on Big Data. IEEE Transactions on Big Data 1, 2 (2015), 49–67. find
  245. K. Xu et al. 2015. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. (2015). arXiv:1502.03044
  246. O. Yadan, K. Adams, Y. Taigman, and M. Ranzato. 2013. Multi-GPU Training of ConvNets. (2013). arXiv:1312.5853 — [arXiv:1312.5853](https://arxiv.org/abs/1312.5853) find
  247. F. Yan, O. Ruwase, Y. He, and T. Chilimbi. 2015. Performance Modeling and Scalability Optimization of Distributed Deep Learning Systems. In Proc. 21st ACM Int’l Conf. on Knowledge Discovery and Data Mining (KDD ’15). 1355–1364. find
  248. Y. You, A. Buluç, and J. Demmel. 2017. Scaling Deep Learning on GPU and Knights Landing Clusters. In Proc. International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’17). 9:1–9:12. find
  249. Y. You, I. Gitman, and B. Ginsburg. 2017. Large Batch Training of Convolutional Networks. (2017). arXiv:1708.03888 — [arXiv:1708.03888](https://arxiv.org/abs/1708.03888) find
  250. Y. You, Z. Zhang, C. Hsieh, and J. Demmel. 2017. 100-epoch ImageNet Training with AlexNet in 24 Minutes. (2017). arXiv:1709.05011
  251. S. R. Young et al. 2017. Evolving Deep Networks Using HPC. In Proc. Workshop on Machine Learning in HPC Environments (MLHPC’17). 7:1–7:7. find
  252. F. Yu and V. Koltun. 2016. Multi-Scale Context Aggregation by Dilated Convolutions. In International Conference on Learning Representations (ICLR). find
  253. Y. Yu, J. Jiang, and X. Chi. 2016. Using Supercomputer to Speed up Neural Network Training. In IEEE 22nd International Conference on Parallel and Distributed Systems (ICPADS). 942–947. find
  254. H. Zhang, Z. Hu, J. Wei, P. Xie, G. Kim, Q. Ho, and E. P. Xing. 2015. Poseidon: A System Architecture for Efficient GPU-based Deep Learning on Multiple Machines. (2015). arXiv:1512.06216 find
  255. H. Zhang, Z. Zheng, S. Xu, W. Dai, Q. Ho, X. Liang, Z. Hu, J. Wei, P. Xie, and E. P. Xing. 2017. Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusters. In USENIX ATC. 181–193. find
  256. J. Zhang, I. Mitliagkas, and C. Ré. 2017. YellowFin and the Art of Momentum Tuning. (2017). arXiv:1706.03471 — [arXiv:1706.03471](https://arxiv.org/abs/1706.03471) find
  257. K. Zhang and X. W. Chen. 2014. Large-Scale Deep Belief Nets With MapReduce. IEEE Access 2 (2014), 395–403. find
  258. S. Zhang, A. Choromanska, and Y. LeCun. 2015. Deep Learning with Elastic Averaging SGD. In Proc. 28th International Conference on Neural Information Processing Systems - Volume 1 (NIPS’15). 685–693. find
  259. S. Zhang, Z. Du, L. Zhang, H. Lan, S. Liu, L. Li, Q. Guo, T. Chen, and Y. Chen. 2016. Cambricon-X: An accelerator for sparse neural networks. In 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). 1–12. find
  260. S. Zhang, C. Zhang, Z. You, R. Zheng, and B. Xu. 2013. Asynchronous stochastic gradient descent for DNN training. In IEEE International Conference on Acoustics, Speech and Signal Processing. 6660–6663. find
  261. W. Zhang, S. Gupta, X. Lian, and J. Liu. 2016. Staleness-aware async-SGD for Distributed Deep Learning. In Proc. Twenty-Fifth International Joint Conference on Artificial Intelligence (IJCAI’16). 2350–2356. find
  262. X. Zhang, M. McKenna, J. P. Mesirov, and D. L. Waltz. 1990. An Efficient Implementation of the Back-propagation Algorithm on the Connection Machine CM-2. In Advances in Neural Information Processing Systems 2. 801–809. find
  263. H. Zhao and J. Canny. 2014. Kylix: A Sparse Allreduce for Commodity Clusters. In 43rd International Conference on Parallel Processing. 273–282. find
  264. Z. Zhong, J. Yan, and C.-L. Liu. 2017. Practical Network Blocks Design with Q-Learning. (2017). arXiv:1708.05552 find
  265. S. Zhou, Z. Ni, X. Zhou, H. Wen, Y. Wu, and Y. Zou. 2016. DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients. (2016). arXiv:1606.06160
  266. M. A. Zinkevich, M. Weimer, A. Smola, and L. Li. 2010. Parallelized Stochastic Gradient Descent. In Proc. 23rd International Conference on Neural Information Processing Systems - Volume 2. 2595–2603. find
  267. A. Zlateski, K. Lee, and H. S. Seung. 2016. ZNNi: Maximizing the Inference Throughput of 3D Convolutional Networks on CPUs and GPUs. In Int’l Conf. for High Performance Computing, Networking, Storage and Analysis. 854–865. find
  268. B. Zoph and Q. V. Le. 2017. Neural Architecture Search with Reinforcement Learning. In International Conference on Learning Representations (ICLR).
  269. B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le. 2017. Learning Transferable Architectures for Scalable Image Recognition. (2017). arXiv:1707.07012

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nun-2018-demystifying-parallel-distributed-deep-learning/