Elamur Research⌕
☰
home / papers / 2019 / PyTorch: An Imperative Style, High-Performance Deep Learning Library

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Summary

Deep learning frameworks have usually made users choose. Graph-based tools such as TensorFlow, Caffe, CNTK and Theano gain speed by building a static computation graph before running it. Eager, define-by-run tools like Chainer are easier to write and debug but have been slower. This NeurIPS 2019 paper presents PyTorch, a Python library that runs tensor operations immediately, with automatic differentiation and GPU support. It also explains the design ideas and runtime engineering that let PyTorch match graph-based tools on speed. Models, optimizers and data loaders are all plain Python code, so standard debuggers, print statements and libraries such as NumPy work with them directly. On the speed side, the CPU queues GPU work asynchronously, a caching allocator reuses GPU memory, and heavy computation goes to cuDNN and cuBLAS. In single-machine benchmarks on six common models, PyTorch came within 17% of the fastest framework every time. Its rapid uptake by researchers suggests that ease of use and high performance can coexist.

Key points

  • Design principles: be Pythonic, put researchers first, give pragmatic performance (losing about 10% speed for a much simpler interface is acceptable, losing 100% is not) and "worse is better," meaning simple internals are preferred over complex, complete ones.
  • Interoperability: torch.from_numpy() / .numpy() and DLPack share memory with other libraries without copying data. Users can extend the library by subclassing torch.autograd.Function or torch.utils.data.Dataset.
  • Runtime behaviour: in a ResNet-50 trace, the CPU queues work faster than the GPU can run it, so the GPU stays almost fully busy (GPU execution took about 3× as long as CPU scheduling). After the first iteration, the caching allocator removes slow cudaMalloc/cudaFree calls.
  • Benchmarks (FP32 training throughput): comparison against Chainer, CNTK, MXNet, PaddlePaddle and TensorFlow on AlexNet, VGG-19, ResNet-50, MobileNet, GNMTv2 and NCF. PyTorch was fastest on VGG-19 (119 img/s), GNMTv2 (15,512 vs 9,631 tokens/s for TensorFlow) and NCF (5.4e6 vs 4.8e6 samples/s for TensorFlow).
  • Adoption: 296 ICLR 2019 submissions mentioned PyTorch. Its share of framework mentions on arXiv rose steadily after the January 2017 release. Planned work includes the PyTorch JIT and better distributed and model-parallel training.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

3466 papers of our library cite it (of the 140673 whose bibliographies we read); 113 of them have a page here, the first 100 below, 3353 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1912.01703 · arXiv 1912.01703

@misc{paszke2019pytorch,
  title = {{PyTorch: An Imperative Style, High-Performance Deep Learning Library}},
  author = {Paszke, Adam and Gross, Sam and Massa, Francisco and Lerer, Adam and Bradbury, James and Chanan, Gregory and Killeen, Trevor and Lin, Zeming and Gimelshein, Natalia and Antiga, Luca and Desmaison, Alban and Köpf, Andreas and Yang, Edward and DeVito, Zach and Raison, Martin and Tejani, Alykhan and Chilamkurthy, Sasank and Steiner, Benoit and Fang, Lu and Bai, Junjie and Chintala, Soumith},
  year = {2019},
  doi = {10.48550/arxiv.1912.01703},
  eprint = {1912.01703},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1912.01703}
}

References

44 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yangqing "Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor" Darrell. "caffe: Convolutional architecture for fast feature embedding". "arXiv preprint arXiv:1408.5093", "2014". find our card
  2. Frank Seide and Amit Agarwal. Cntk: Microsoft’s open-source deep-learning toolkit. In Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, pages 2135–2135, New York, NY, USA, 2016. ACM. find
  3. Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenber… find
  4. Theano Development Team. Theano: A Python framework for fast computation of mathematical expressions. arXiv e-prints, abs/1605.02688, May 2016. find our card
  5. Seiya Tokui, Kenta Oono, Shohei Hido, and Justin Clayton. Chainer: a next-generation open source framework for deep learning. In Proceedings of Workshop on Machine Learning Systems (LearningSys) in The Twenty-ninth Annual Conference on Neural Information Processing Systems (NIPS), 2015. find
  6. Ronan Collobert, Samy Bengio, and Johnny Mariéthoz. Torch: a modular machine learning software library. Technical report, Idiap, 2002. find
  7. G. Neubig, C. Dyer, Y. Goldberg, A. Matthews, W. Ammar, A. Anastasopoulos, M. Ballesteros, D. Chiang, D. Clothiaux, T. Cohn, K. Duh, M. Faruqui, C. Gan, D. Garrette, Y. Ji, L. Kong, A. Kuncoro, G. Kumar, C. Malaviya, P. Michel, Y. Oda, M. Richardson, N. Saphra, S. Swayamdipta, and P. Yin. DyNet: The… find our card
  8. Philip S. Abrams. An APL Machine. PhD thesis, Stanford University, 1970. find
  9. The MathWorks, Inc., Natick, Massachusetts, United States. MATLAB and Statistics Toolbox. find
  10. R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. find
  11. Jeff Bezanson, Alan Edelman, Stefan Karpinski, and Viral B Shah. Julia: A fresh approach to numerical computing. SIAM review, 59(1):65–98, 2017. find
  12. Travis Oliphant. NumPy: http://www.numpy.org/. A guide to NumPy. ↗
  13. Gaël Guennebaud, Benoît Jacob, et al. Eigen v3. http://eigen.tuxfamily.org, 2010. ↗
  14. Y LeCun and L Bottou. Lush reference manual. http://lush.sourceforge.net, 2002. ↗
  15. Atilim Gunes Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: A survey. J. Mach. Learn. Res., 18(1):5595–5637, January 2017. find
  16. Dougal Maclaurin. Modeling, Inference and Optimization with Composable Differentiable Procedures. PhD thesis, Harvard University, April 2016. find
  17. Matthew Johnson et. al. Jax. https://github.com/google/jax, 2018. ↗
  18. Mike Innes et. al. Flux.jl. https://github.com/FluxML/Flux.jl, 2018. ↗
  19. Eric Jones, Travis Oliphant, Pearu Peterson, et al. SciPy: Open source scientific tools for Python, 2001–. http://www.scipy.org/. ↗
  20. Wes McKinney. Data structures for statistical computing in python. In Proceedings of the 9th Python in Science Conference, 51-56, 2010. find
  21. Pierre Sermanet, Koray Kavukcuoglu, and Yann LeCun. Eblearn: Open-source energy-based learning in c++. In 2009 21st IEEE International Conference on Tools with Artificial Intelligence, pages 693–697. IEEE, 2009. USA: Trelgol Publishing, 2006. Technical report, code available at find
  22. Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan D. Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. cudnn: Efficient primitives for deep learning. CoRR, abs/1410.0759, 2014. find
  23. Andrew Lavin. maxdnn: An efficient convolution kernel for deep learning with maxwell gpus, January 2015. find
  24. Andrew Lavin and Scott Gray. Fast algorithms for convolutional neural networks. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4013–4021, 2016. find
  25. Ronan Collobert, Koray Kavukcuoglu, and Clément Farabet. Torch7: A matlab-like environment for machine learning. In NIPS 2011, 2011. find
  26. Richard Gabriel. The rise of worse is better. http://dreamsongs.com/RiseOfWorseIsBetter.html. ↗
  27. Yann LeCun and Corinna Cortes. http://yann.lecun.com/exdb/mnist/. ↗
  28. Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, John Quan, Stephen Gaffney, Stig Petersen, Karen Simonyan, Tom Schaul, Hado van Hasselt, David Silver, Timothy P. Lillicrap,… — [arXiv:1708.04782](https://arxiv.org/abs/1708.04782) find
  29. DMLC. Dlpack: Open in memory tensor structure. https://github.com/dmlc/dlpack. ↗
  30. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS Workshop, 2017. find
  31. Dan Piponi. Automatic differentiation, C++ templates, and photogrammetry. J. Graphics, GPU, & Game Tools, 9(4):41–55, 2004. find
  32. Holger Leuck and Hans-Hellmut Nagel. Automatic differentiation facilitates of-integration into steering-angle-based road vehicle tracking. In 1999 Conference on Computer Vision and Pattern Recognition (CVPR ’99), 23-25 June 1999, Ft. Collins, CO, USA, pages 2360–2365, 1999. find
  33. The Python team. The cpython https://wiki.python.org/moin/GlobalInterpreterLock. ↗
  34. Giovanni Petrantoni and Jörg Wollenschläger. xyz/nimtorch. find
  35. Austin Huang, Junji Hashimoto, https://github.com/hasktorch/hasktorch. ↗
  36. G. Synnaeve, Z. Lin, J. Gehring, D. Gant, V. Mella, V. Khalidov, N. Carion, and N. Usunier. Forward modeling for partial observation strategy games - a starcraft defogger. In Advances in Neural Information Processing Systems, pages 10761–10771, 2018. find
  37. The PyTorch team. Torch Script. https://pytorch.org/docs/stable/jit.html. ↗
  38. Justin Luitjens. Cuda streams. GPU technology conference, 2014. find
  39. Emery D. Berger, Kathryn S. McKinley, Robert D. Blumofe, and Paul R. Wilson. Hoard: A scalable memory allocator for multithreaded applications. In Proceedings of the Ninth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS IX, pages 117–128, New… find
  40. J. Evans. A scalable concurrent malloc(3) implementation for freebsd. In In BSDCan — The Technical BSD Conference, May 2006. find
  41. S. Ghemawat and P. Menage. Tcmalloc: Thread-caching malloc. MNIST handwritten digit database. global interpreter lock. Nimtorch. https://github.com/fragcolor-and Sam Stites. Hasktorch. ↗
  42. Benjamin Recht, Christopher Ré, Stephen J. Wright, and Feng Niu. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011. Proceedings of a meeting held 12… find
  43. Matthew Hertz and Emery D. Berger. Quantifying the performance of garbage collection vs. explicit memory management. In Proceedings of the 20th Annual ACM SIGPLAN Conference on Object-oriented Programming, Systems, Languages, and Applications, OOPSLA ’05, pages 313–326, New York, NY, USA, 2005. ACM. find
  44. The PyTorch team. Pytorch Autograd Profiler. https://pytorch.org/docs/1.0.1/autograd.html\#profiler. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/