Elamur Research
☰
home / papers / 2016 / TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1603.04467

Summary

Google's first-generation distributed training system, DistBelief, was used by more than 50 teams. It proved too rigid for fast research and hard to bring to small devices. This white paper presents TensorFlow, its successor. TensorFlow is both an interface for defining machine learning computations and a system for running them. A program is written as a stateful dataflow graph. Nodes are operations, which have device-specific kernels for CPU or GPU. Edges carry typed multi-dimensional tensors or pure control dependencies. Variables hold model parameters that persist between runs. Clients use a Session to build the graph and repeatedly execute chosen subgraphs. The same graph can run with little modification on a phone or on a cluster with thousands of GPUs. A master process distributes the work across worker processes, and replication supports various forms of parallelism. The authors report that TensorFlow is faster and more flexible than DistBelief and that many internal teams have migrated to it. Its main significance is that one system covers both research experiments and production deployment, which removes the need to maintain separate stacks. It is released as open source.

Key points

  • Open-sourced in November 2015 under the Apache 2.0 license (www.tensorflow.org), with C++ and Python front ends.
  • It targets everything from Android and iOS inference to training on hundreds of machines. Internal models reach hundreds of billions of parameters.
  • Core abstractions are operations, kernels, tensors, control dependencies, Variables (with Assign and AssignAdd) and Sessions (Extend and Run). Tensor element types include 8–64-bit integers, float, double, complex and string.
  • It grew out of experience with more than one hundred ML projects at Google, covering areas such as speech, vision, translation, NLP and geographic information extraction.
  • Its design mixes ideas from Dryad, Flume, CIEL, Spark and Naiad. The authors list Halide-style cross-operation compilation as future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

304 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 292 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1603.04467 · arXiv 1603.04467

@misc{abadi2016tensorflow,
  title = {{TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems}},
  author = {Abadi, Martín and Agarwal, Ashish and Barham, Paul and Brevdo, Eugene and Chen, Zhifeng and Citro, Craig and Corrado, Greg S. and Davis, Andy and Dean, Jeffrey and Devin, Matthieu and Ghemawat, Sanjay and Goodfellow, Ian and Harp, Andrew and Irving, Geoffrey and Isard, Michael and Jia, Yangqing and Jozefowicz, Rafal and Kaiser, Lukasz and Kudlur, Manjunath and Levenberg, Josh and Mane, Dan and Monga, Rajat and Moore, Sherry and Murray, Derek and Olah, Chris and Schuster, Mike and Shlens, Jonathon and Steiner, Benoit and Sutskever, Ilya and Talwar, Kunal and Tucker, Paul and Vanhoucke, Vincent and Vasudevan, Vijay and Viegas, Fernanda and Vinyals, Oriol and Warden, Pete and Wattenberg, Martin and Wicke, Martin and Yu, Yuan and Zheng, Xiaoqiang},
  year = {2016},
  doi = {10.48550/arxiv.1603.04467},
  eprint = {1603.04467},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1603.04467}
}

References

56 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martı́n Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenbe… find
  2. Anelia Angelova, Alex Krizhevsky, and Vincent Vanhoucke. Pedestrian detection with a large-field-of-view deep network. In Robotics and Automation (ICRA), 2015 IEEE International Conference on, pages 704–711. IEEE, 2015. CalTech PDF. find
  3. Arvind and David E. Culler. of computer science vol. 1, Annual review 1986. chapter Dataflow Architectures, pages 225–253. 1986. www.dtic.mil/cgi-bin/GetTRDoc?Location=U2&doc=GetTRDoc.pdf&AD=ADA166235. find
  4. Arvind and Rishiyur S. Nikhil. Executing a program on the MIT tagged-token dataflow architecture. IEEE Trans. Comput., 39(3):300–318, 1990. dl.acm.org/citation.cfm?id=78583. find
  5. Jimmy Ba, Volodymyr Mnih, and Koray Kavukcuoglu. Multiple object recognition with visual attention. arXiv preprint arXiv:1412.7755, 2014. arxiv.org/abs/1412.7755. — [arXiv:1412.7755](https://arxiv.org/abs/1412.7755) find
  6. Françoise Beaufays. The neural networks behind Google Voice transcription, 2015. googleresearch.blogspot.com/2015/08/the-neural-networks-behind-google-voice.html. find
  7. James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. Theano: A CPU and GPU math expression compiler. In Proceedings of the Python for scientific computing conference (SciPy), volume 4, page 3… find
  8. Craig Chambers, Ashish Raniwala, Frances Perry, Stephen Adams, Robert R Henry, Robert Bradshaw, and Nathan Weizenbaum. FlumeJava: easy, efficient data-parallel pipelines. In ACM Sigplan Notices, volume 45, pages 363–375. ACM, 2010. research.google.com/pubs/archive/35650.pdf. find
  9. Sharan Chetlur, Cliff Woolley, Philippe Vandermer-sch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. cuDNN: Efficient primitives for deep learning. arXiv preprint arXiv:1410.0759, 2014. arxiv.org/abs/1410.0759. find
  10. Trishul Chilimbi, Yutaka Suzue, Johnson Apacible, and Karthik Kalyanaraman. Project Adam: Building an efficient and scalable deep learning training system. In 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14), pages 571–582, 2014. www.usenix.org/system/files/conference/o… find
  11. Jack Clark. Google turning its lucrative web search over to AI machines, 2015. www.bloomberg.com/news/articles/2015-10-26/google-turning-its-lucrative-web-search-over-to-ai-machines. find
  12. Cliff Click. Global code motion/global value numbering. In ACM SIGPLAN Notices, volume 30, pages 246– 257. ACM, 1995. courses.cs.washington.edu/courses/ cse501/06wi/reading/click-pldi95.pdf. find
  13. Ronan Collobert, Samy Bengio, and Johnny Mariéthoz. Torch: A modular machine learning software library. Technical report, IDIAP, 2002. infoscience.epfl.ch/record/82802/files/rr02-46.pdf. find
  14. Jeffrey Dean, Gregory S. Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Quoc V. Le, Mark Z. Mao, Marc’Aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, and Andrew Y. Ng. Large scale distributed deep networks. In NIPS, 2012. Google Research PDF. find
  15. Jack J Dongarra, Jeremy Du Croz, Sven Hammar-ling, and Iain S Duff. A set of level 3 basic linear algebra subprograms. ACM Transactions on Mathematical Software (TOMS), 16(1):1–17, 1990. www.maths.manchester.ac.uk/˜sven/pubs/Level3BLAS-1-TOMS16-90.pdf. find
  16. Andrea Frome, Greg S Corrado, Jonathon Shlens, Samy Bengio, Jeff Dean, Tomas Mikolov, et al. DeVISE: A deep visual-semantic embedding model. In Advances in Neural Information Processing Systems, pages 2121–2129, 2013. research.google.com/pubs/archive/41473.pdf. find
  17. Javier Gonzalez-Dominguez, Ignacio Lopez-Moreno, Pedro J Moreno, and Joaquin Gonzalez-Rodriguez. Frame-by-frame language identification in short utterances using deep neural networks. Neural Networks, 64:49–58, 2015. find
  18. Otavio Good. How Google Translate squeezes deep learning onto a phone, 2015. googleresearch.blogspot.com/2015/07/how-google-translate-squeezes-deep.html. find
  19. Ian J. Goodfellow, Yaroslav Bulatov, Julian Ibarz, Sacha Arnoud, and Vinay Shet. Multi-digit number recognition from Street View imagery using deep convolutional neural networks. In International Conference on Learning Representations, 2014. arxiv.org/pdf/1312.6082. — [arXiv:1312.6082](https://arxiv.org/abs/1312.6082) find
  20. Georg Heigold, Vincent Vanhoucke, Alan Senior, Patrick Nguyen, Marc’Aurelio Ranzato, Matthieu Devin, and Jeffrey Dean. Multilingual acoustic models using distributed deep neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pages 8619–8623. IEE… find
  21. Geoffrey E. Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, and Brian Kingsbury. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal… find
  22. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. ftp.idsia.ch/pub/juergen/lstm.pdf. find
  23. Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. CoRR, abs/1502.03167, 2015. arxiv.org/abs/1502.03167. find
  24. Michael Isard, Mihai Budiu, Yuan Yu, Andrew Birrell, and Dennis Fetterly. Dryad: distributed data-parallel programs from sequential building blocks. In ACM SIGOPS Operating Systems Review, volume 41, pages 59–72. ACM, 2007. www.michaelisard.com/pubs/eurosys07.pdf. find
  25. Benoı̂t Jacob, Gaël Guennebaud, et al. Eigen library for linear algebra. eigen.tuxfamily.org. find
  26. Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the ACM International Conference on Multimedia, pages 675–678. ACM, 2014. arxiv.org/pdf/140… find our card
  27. Andrej Karpathy, George Toderici, Sachin Shetty, Tommy Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convolutional neural networks. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, pages 1725–1732. IEEE, 2014. research.google.com/pubs/archi… find
  28. A Krizhevsky. Cuda-convnet, code.google.com/p/cuda-convnet/. 2014. find
  29. Alex Krizhevsky. One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:1404.5997, 2014. arxiv.org/abs/1404.5997. find
  30. Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The CIFAR-10 dataset. www.cs.toronto.edu/˜kriz/cifar.html. find
  31. Quoc Le, Marc’Aurelio Ranzato, Rajat Monga, Matthieu Devin, Greg Corrado, Kai Chen, Jeff Dean, and Andrew Ng. Building high-level features using large scale unsupervised learning. In ICML’2012, 2012. Google Research PDF. find
  32. Yann LeCun, Corinna Cortes, and Christopher JC Burges. The MNIST database of handwritten digits, 1998. yann.lecun.com/exdb/mnist/. find
  33. Mu Li, Dave Andersen, and Alex Smola. server. parameterserver.org. Parameter find
  34. Chris J Maddison, Aja Huang, Ilya Sutskever, and David Silver. Move evaluation in Go using deep convolutional neural networks. arXiv preprint arXiv:1412.6564, 2014. arxiv.org/abs/1412.6564. — [arXiv:1412.6564](https://arxiv.org/abs/1412.6564) find
  35. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In International Conference on Learning Representations: Workshops Track, 2013. arxiv.org/abs/1301.3781. find
  36. Derek G Murray, Frank McSherry, Rebecca Isaacs, Michael Isard, Paul Barham, and Martı́n Abadi. Naiad: a timely dataflow system. In Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles, pages 439–455. ACM, 2013. Microsoft Research PDF. find
  37. Derek G. Murray, Malte Schwarzkopf, Christopher Smowton, Steven Smit, Anil Madhavapeddy, and Steven Hand. Ciel: a universal execution engine for distributed data-flow computing. In Proceedings of the Ninth USENIX Symposium on Networked Systems Design and Implementation, 2011. Usenix PDF. find
  38. Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, Alessandro De Maria, Vedavyas Panneershelvam, Mustafa Suleyman, Charles Beattie, Stig Petersen, et al. Massively parallel methods for deep reinforcement learning. arXiv preprint arXiv:1507.04296, 2015. arxiv.org/abs/1507.0429… find
  39. CUDA Nvidia. Cublas library. NVIDIA Corporation, Santa Clara, California, 15, 2008. developer.nvidia.com/cublas. find
  40. Jonathan Ragan-Kelley, Connelly Barnes, Andrew Adams, Sylvain Paris, Frédo Durand, and Saman Amarasinghe. Halide: A language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines. ACM SIGPLAN Notices, 48(6):519– 530, 2013. people.csail.mit.edu/fredo/tmp/… find
  41. Bharath Ramsundar, Steven Kearnes, Patrick Riley, Dale Webster, David Konerding, and Vijay Pande. Massively multitask networks for drug discovery. arXiv preprint arXiv:1502.02072, 2015. arxiv.org/abs/1502.02072. — [arXiv:1502.02072](https://arxiv.org/abs/1502.02072) find
  42. Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Advances in Neural Information Processing Systems, pages 693–701, 2011. papers.nips.cc/paper/4390-hogwild-a-lock-free-approach-to-parallelizing-stochastic-grad… find
  43. Chuck Rosenberg. Improving Photo Search: A step across the semantic gap, googleresearch.blogspot.com/2013/06/improving-photo-search-step-across.html. find
  44. Christopher J Rossbach, Yuan Yu, Jon Currey, Jean-Philippe Martin, and Dennis Fetterly. Dandelion: a compiler and runtime for heterogeneous systems. In Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles, pages 49–68. ACM, 2013. research-srv.microsoft.com/pubs/201110/sosp1… find
  45. David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by backpropagating errors. Cognitive modeling, 5:3, 1988. www.cs.toronto.edu/ hinton/absps/naturebp.pdf. find
  46. Haşim Sak, Andrew Senior, Kanishka Rao, Françoise Beaufays, and Johan Schalkwyk. Google Voice Search: faster and more accurate, 2015. googleresearch.blogspot.com/2015/09/google-voice-search-faster-and-more.html. find
  47. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. In NIPS, 2014. papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural. find
  48. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR’2015, 2015. arxiv.org/abs/1409.4842. — [arXiv:1409.4842](https://arxiv.org/abs/1409.4842) find
  49. Seiya Tokui. Chainer: A powerful, flexible and intuitive framework of neural networks. chainer.org. find
  50. Vincent Vanhoucke. Speech recognition and deep learning, 2015. googleresearch.blogspot.com/2012/08/speechrecognition-and-deep-learning.html. find
  51. Abhishek Verma, Luis Pedrosa, Madhukar Korupolu, David Oppenheimer, Eric Tune, and John Wilkes. Large-scale cluster management at Google with Borg. In Proceedings of the Tenth European Conference on Computer Systems, page 18. ACM, 2015. research.google.com/pubs/archive/43438.pdf. find
  52. O. Vinyals, L. Kaiser, T. Koo, S. Petrov, I. Sutskever, and G. Hinton. Grammar as a foreign language. Technical report, arXiv:1412.7449, 2014. arxiv.org/abs/1412.7449. find
  53. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In NIPS, 2015. arxiv.org/abs/1506.03134. find
  54. Dong Yu, Adam Eversole, Mike Seltzer, Kaisheng Yao, Zhiheng Huang, Brian Guenter, Oleksii Kuchaiev, Yu Zhang, Frank Seide, Huaming Wang, et al. An introduction to computational networks and the computational network toolkit. Technical report, Tech. Rep. MSR, Microsoft Research, 2014, 2014. research.… find
  55. Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael J Franklin, Scott Shenker, and Ion Stoica. Resilient 2013. distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In Proceedings of the 9th USENIX conference on Networked Sy… find
  56. Matthew D. Zeiler, Marc’Aurelio Ranzato, Rajat Monga, Mark Mao, Ke Yang, Quoc Le, Patrick Nguyen, Andrew Senior, Vincent Vanhoucke, Jeff Dean, and Geoffrey E. Hinton. On rectified linear units for speech processing. In ICASSP, 2013. research.google.com/pubs/archive/40811.pdf. find