Elamur Research⌕
☰
home / papers / 2017 / ImageNet Training in Minutes

ImageNet Training in Minutes

Paper · 2017 · Yang You, Zhao Zhang, Cho-Jui Hsieh et al.

Summary

Training a standard classifier on ImageNet-1k once required days or weeks: 90 epochs of ResNet-50 on one NVIDIA M40 took about 14 days. Yet the arithmetic involved could in theory run in seconds on the fastest supercomputer. The authors test whether LARS (Layer-wise Adaptive Rate Scaling), an optimizer that adjusts learning rates per layer and had earlier handled 32K batches on only 8 GPUs, can turn very large batches into real wall-clock savings on big clusters. They use synchronous data-parallel SGD on Intel Skylake CPUs and Intel Knights Landing (KNL) processors. On 1024 CPUs, AlexNet trains for 100 epochs in 11 minutes. On 2048 KNLs, ResNet-50 trains for 90 epochs in 20 minutes with accuracy matching the small-batch baseline. A shortened 64-epoch ResNet-50 run hits 74.9% top-1 in 14 minutes, edging out the prior record. The work shows that the optimization method, not only the hardware, limits how far synchronous training can scale. It also explains in simple terms why bigger batches reduce communication cost.

Key points

  • Starting point: 90-epoch ResNet-50 training needs roughly 10^18 single-precision operations. A machine delivering 2×10^17 operations per second could in principle finish it in about five seconds.
  • AlexNet: With batch size 32K, AlexNet reached 58.6% top-1 in 11 minutes on 1024 CPUs. At this batch size, local response normalization was replaced with batch norm. The same accuracy level took 6h 10m at batch 512 on a DGX-1.
  • ResNet-50 at 32K: With data augmentation, ResNet-50 reached 75.4% top-1 at batch 32K in 20 minutes on 2048 KNLs. The 64-epoch variant gave 74.9% in 14 minutes, compared with 15 minutes in Akiba et al.
  • Comparison with Facebook (Goyal et al. 2017): The authors' baseline is slightly weaker (75.3% vs 76.2%) because of lighter augmentation. At very large batches the ranking flips:
    • at 32K: 75.4% vs 72.4%;
    • at 64K: 73.2% vs 66.0%.
  • Cost argument: For a fixed number of epochs, a larger batch keeps total FLOPs unchanged but cuts the number of iterations. That means fewer messages and less transferred data, since communication volume scales as |W| × E × n / B. Supporting tables show that network latency and bandwidth costs far exceed per-flop time. They also show that a DRAM access (640 pJ) uses far more energy than a float multiply (3.7 pJ).

Interesting ideas

  • I1. Estimate the ideal runtime first. Divide the total operation count by peak hardware throughput before optimizing anything. This quickly shows how far current practice is from the physical limit, and the same check works for any large NLP or data-processing job.
  • I2. Measure parallel work per processor. The authors reason about how many examples each worker gets (for example, a 1K batch spread over 512 processors leaves just 2 per worker). This is a simple way to tell whether a cluster is limited by compute or by communication.
  • I3. Use a back-of-envelope communication model. Counting messages (iterations) and data moved (model size × number of updates) explains speedups without detailed profiling. It can guide choices of batch size or synchronization frequency in other distributed training setups.
  • I4. Evaluate scaling at a fixed epoch budget. Holding the number of passes over the data constant and plotting accuracy against batch size separates real optimizer robustness from simply training longer. This is a fair way to compare large-batch methods.
  • I5. Value synchronous SGD's reproducibility. Because synchronous SGD behaves like its sequential version, parallel runs can be debugged and checked against single-machine results. That reproducibility can justify choosing it over asynchronous schemes even when those seem faster.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California, BerkeleyBerkeley · United States209 works · 244,782 citations🎓University of California, DavisDavis · United States34 works · 42,940 citations

Also on the byline: UC Davis Health · Texas Advanced Computing Center.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 worksHardware and Architecture · 167 works

Cite

Cite as Yang You, Zhao Zhang, Cho-Jui Hsieh, James Demmel and Kurt Keutzer. 2017. ImageNet Training in Minutes. arXiv:1709.05011.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1709.05011 · arXiv 1709.05011

@misc{you2017imagenet,
  title = {{ImageNet Training in Minutes}},
  author = {Yang You, Zhao Zhang, Cho-Jui Hsieh, James Demmel and Kurt Keutzer},
  year = {2017},
  doi = {10.48550/arxiv.1709.05011},
  eprint = {1709.05011},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1709.05011}
}

References

29 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akiba, T.; Suzuki, S.; and Fukuda, K. 2017. Extremely large minibatch sgd: Training resnet-50 on imagenet in 15 minutes. arXiv preprint arXiv:1711.04325. find
  2. Amodei, D.; Anubhai, R.; Battenberg, E.; Case, C.; Casper, J.; Catanzaro, B.; Chen, J.; Chrzanowski, M.; Coates, A.; Diamos, G.; et al. 2015. Deep speech 2: End-to-end speech recognition in english and mandarin. arXiv preprint arXiv:1512.02595.
  3. Catanzaro, B. 2013. Deep learning with cots hpc systems. find
  4. Chen, J.; Monga, R.; Bengio, S.; and Jozefowicz, R. 2016. find
  5. Revisiting distributed synchronous sgd. arXiv preprint arXiv:1604.00981. Das, D.; Avancha, S.; Mudigere, D.; Vaidynathan, K.; Sridharan, S.; Kalamkar, D.; Kaul, B.; and Dubey, P. 2016. Distributed deep learning using synchronous stochastic gradient descent. arXiv preprint arXiv:1602.06709. — [arXiv:1604.00981](https://arxiv.org/abs/1604.00981) find
  6. Dean, J.; Corrado, G.; Monga, R.; Chen, K.; Devin, M.; Mao, M.; Senior, A.; Tucker, P.; Yang, K.; Le, Q. V.; et al. 2012. Large scale distributed deep networks. In Advances in neural information processing systems, 1223–1231. find
  7. Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. find
  8. CVPR 2009. IEEE Conference on, 248–255. IEEE. find
  9. Dongarra, J.; Meuer, M.; Simon, H.; and Strohmaier, E. 2017. Top500 supercomputer ranking. find
  10. Goyal, P.; Dollár, P.; Girshick, R.; Noordhuis, P.; Wesolowski, L.; Kyrola, A.; Tulloch, A.; Jia, Y.; and He, K. 2017. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677. find
  11. Gropp, W.; Lusk, E.; Doss, N.; and Skjellum, A. 1996. find
  12. A high-performance, portable implementation of the mpi message passing interface standard. Parallel computing 22(6):789–828. find
  13. He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770–778.
  14. Horowitz, M. Energy table for 45nm process. Iandola, F. N.; Ashraf, K.; Moskewicz, M. W.; and Keutzer, K. 2015. Firecaffe: near-linear acceleration of deep neural network training on compute clusters. CoRR abs/1511.00175. find
  15. Iandola, F. N.; Moskewicz, M. W.; Ashraf, K.; and Keutzer, K. 2016. Firecaffe: near-linear acceleration of deep neural network training on compute clusters. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2592–2600. find
  16. Jia, Y.; Shelhamer, E.; Donahue, J.; Karayev, S.; Long, J.; Girshick, R.; Guadarrama, S.; and Darrell, T. 2014. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM international conference on Multimedia, 675–678. ACM.
  17. Jin, P. H.; Yuan, Q.; Iandola, F.; and Keutzer, K. 2016. find
  18. How to scale distributed deep learning? arXiv preprint arXiv:1611.04581. Keskar, N. S.; Mudigere, D.; Nocedal, J.; Smelyanskiy, M.; and Tang, P. T. P. 2016. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836. find
  19. Krizhevsky, A. 2014. One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:1404.5997.
  20. Le, Q. V. 2013. Building high-level features using large scale unsupervised learning. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, 8595–8598. IEEE. find
  21. Li, M. 2017. Scaling Distributed Machine Learning with System and Algorithm Co-design. Ph.D. Dissertation, Intel. find
  22. Mitliagkas, I.; Zhang, C.; Hadjis, S.; and Ré, C. 2016. Asynchrony begets momentum, with an application to deep learning. In Communication, Control, and Computing (Allerton), 2016 54th Annual Allerton Conference on, 997–1004. IEEE. find
  23. Recht, B.; Re, C.; Wright, S.; and Niu, F. 2011. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Advances in Neural Information Processing Systems, 693–701. find
  24. Seide, F.; Fu, H.; Droppo, J.; Li, G.; and Yu, D. 2014a. 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech dnns. In Interspeech, 1058–1062. find
  25. Seide, F.; Fu, H.; Droppo, J.; Li, G.; and Yu, D. 2014b. find
  26. On parallelizability of stochastic gradient descent for speech dnns. In Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on, 235–239. IEEE. find
  27. You, Y.; Gitman, I.; and Ginsburg, B. 2017. Scaling sgd batch size to 32k for imagenet training. find
  28. Zhang, S.; Choromanska, A. E.; and LeCun, Y. 2015a. Deep learning with elastic averaging sgd. In Advances in Neural Information Processing Systems, 685–693. find
  29. Zhang, S.; Choromanska, A. E.; and LeCun, Y. 2015b. Deep learning with elastic averaging sgd. In Advances in Neural Information Processing Systems, 685–693. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/you-2017-imagenet-training-minutes/