Elamur Research⌕
☰
home / papers / 2016 / TensorFlow: A system for large-scale machine learning

TensorFlow: A system for large-scale machine learning

Paper · 2016 · Martin Abadi, Paul Barham, Jianmin Chen et al.

Summary

The paper presents TensorFlow, Google Brain's successor to its earlier DistBelief system. It addresses a gap among existing tools. Single-machine frameworks such as Caffe, Theano and Torch are flexible but do not scale across clusters. Parameter-server systems scale, but they hard-code how shared model state is managed.

TensorFlow represents both the computation and the mutable state it works on as one dataflow graph. Its vertices may own and update variables, and its edges carry tensors. The runtime places graph nodes across machines and devices (multicore CPUs, GPUs, TPUs) and inserts the needed communication automatically. As a result, parameter-server behaviour becomes ordinary user-level code that can be changed and experimented with.

The authors evaluate the system on image classification and language modeling. The language-modeling experiment trains an LSTM on the One Billion Word Benchmark. The results show good scaling, and synchronous replication turns out to be workable despite the common preference for asynchronous updates.

The work matters because one programming model covers research, large-scale training and production inference, from datacenters down to mobile devices.

Key points

  • Before the paper, more than 60 Google teams had used TensorFlow. After the open-source release, the repository passed 8,000 forks and the binaries were downloaded about 500,000 times.
  • The language-modeling experiment uses LSTM-512-512 on the One Billion Word Benchmark. The vocabulary is cut from 800,000 to the 40,000 most frequent words.
  • In the full softmax, a 512 × 40,000 weight matrix is sharded across PS tasks.
    • Adding a second PS task raised throughput more than growing the worker count from 4 to 32 or from 32 to 256.
  • Sampled softmax, which draws 512 classes per batch, makes the softmax layer's communication and computation 78 times smaller.
  • TensorFlow reaches training steps of about 2 seconds on large models and clusters. For comparison, SparkNet needs around 20 seconds just to broadcast weights to five workers and collect their updates.

Interesting ideas

  • I1. State as graph nodes. Variables and their update operations are treated as ordinary graph vertices instead of a fixed system service. This lets users invent new optimizers or consistency schemes without touching the core. The same principle suits any pipeline system where "infrastructure" logic should stay customizable.
  • I2. Moving computation to where the data lives. Running the softmax multiplication on the shards that hold the weights reduces network traffic. This fits any setup with a huge lookup table or embedding matrix, such as large-vocabulary NLP models.
  • I3. Sampling to shrink the output layer. Scoring only the true class plus a random set of negatives replaces a dense vocabulary-sized product. It gives large savings for tasks with many classes, such as language modeling for morphologically rich languages with big vocabularies.
  • I4. Portable device abstraction. Operations target a generic device interface and can be specialized per hardware type. This hedges against uncertainty about which accelerators will dominate, a useful design choice for any long-lived compute library.
  • I5. Scaling experiments that vary one resource at a time. Sweeping PS tasks and workers separately shows which resource is the real bottleneck. This is a cheap evaluation habit worth copying when tuning distributed jobs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States430 works · 424,363 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksInformation Systems · 1,437 worksHardware and Architecture · 81 worksComputer Networks and Communications · 64 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Machine learning · 51 works

Cite

Cite as Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu and Xiaoqiang Zheng. 2016. TensorFlow: A system for large-scale machine learning. arXiv:1605.08695.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1605.08695 · arXiv 1605.08695

@misc{abadi2016tensorflow,
  title = {{TensorFlow: A system for large-scale machine learning}},
  author = {Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu and Xiaoqiang Zheng},
  year = {2016},
  doi = {10.48550/arxiv.1605.08695},
  eprint = {1605.08695},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1605.08695}
}

References

32 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. [1] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. J. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Józefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mane, R. Monga, S. Moore, D. G. Murray, C. Olah, M. Schuster, J.… — TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
  2. [2] R. Al-Rfou, G. Alain, A. Almahairi, C. Anger-Journal of Machine Learning Research, 3:1137– mueller, D. Bahdanau, N. Ballas, F. Bastien, 1155, 2003. www.iro.umontreal.ca/˜lisa/pointeurs/J. Bayer, A. Belikov, A. Belopolsky, Y. Bengio, BengioDucharmeVincentJauvin jmlr.pdf. A. Bergeron, J. Bergstra,…
  3. IEEE, 2015. CalTech PDF. [13] S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, [4] Arvind and D. E. Culler. Annual review J. Tran, B. Catanzaro, and E. Shelhamer. cuDNN: of computer science vol. 1, 1986. chapter Efficient primitives for deep learning. arXiv preprint Dataflow Architectures, pages 2… find
  4. [15] S. Chintala. convnet-benchmarks, github.com/soumith/convnet-benchmarks. 2016. [24] J. Gonzalez-Dominguez, I. Lopez-Moreno, P. J. Moreno, and J. Gonzalez-Rodriguez. Frame-by-frame language identification in [16] E. S. Chung, J. D. Davis, and J. Lee. LIN-short utterances using deep neural network… — Deep Residual Learning for Image Recognition
  5. IEEE, 2013. re-Eleventh European Conference on Computer Sys-search.google.com/pubs/archive/40807.pdf. tems, EuroSys ’16, 2016. www.pdl.cmu.edu/PDL-[28] B. Hindman, A. Konwinski, M. Zaharia, A. Gh-FTP/CloudComputing/GeePS-cui-eurosys16.pdf. odsi, A. D. Joseph, R. Katz, S. Shenker, and I. Sto-[20] A.… find
  6. [22] J. Dean and S. Ghemawat. Mapreduce: Simplified data processing on large clusters. In Proceedings of the 6th Conference on Symposium on Opeart-ing Systems Design & Implementation - Volume 6, OSDI’04, Berkeley, CA, USA, 2004. USENIX Association. research.google.com/archive/mapreduce- [30] G. E. H… find
  7. [29] G. E. Hinton. Learning distributed representations of concepts. In Proceedings of the Eighth Annual Conference of the Cognitive Science Society, pages 1–12. Hillsdale, NJ: Erlbaum, 1986. www.cogsci.ucsd.edu/˜ajyu/Teaching/Cogs202 - sp13/Readings/hinton86.pdf. find
  8. [31] P. Hunt, M. Konar, F. P. Junqueira, and B. Reed. [40] A. Karpathy, G. Toderici, S. Shetty, T. Leung, ZooKeeper: Wait-free coordination for internet-R. Sukthankar, and L. Fei-Fei. Large-scale video scale systems. In Proceedings of the 2010 classification with convolutional neural networks. In US… find
  9. IEEE, Berkeley, CA, USA, 2010. USENIX Associa-2014. research.google.com/pubs/archive/42455.pdf. tion. www.usenix.org/legacy/event/atc10/tech/full - [41] A. Krizhevsky. One weird trick for paralleliz-papers/Hunt.pdf. ing convolutional neural networks. arXiv preprint [32] S. Ioffe and C. Szegedy. Batc… find
  10. [34] B. Jacob, G. Guennebaud, et al. Eigen library for [43] H. Larochelle, Y. Bengio, J. Louradour, and linear algebra. eigen.tuxfamily.org. P. Lamblin. Exploring strategies for training deep neural networks. Journal of Machine Learn-[35] S. Jean, K. Cho, R. Memisevic, and Y. Bengio. ing Research, 1… find
  11. [36] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast fea- [46] M. Li, D. G. Andersen, J. Park, A. J. Smola, A. Ahmed, V. Josifovski, J. Long, E. J. Shekita, ture embedding. In Proceedings of the ACM Inter-a… — Caffe: Convolutional Architecture for Fast Feature Embedding
  12. [37] M. I. Jordan. Serial order: A parallel dis-www.usenix.org/system/files/conference/osdi14/osdi14-tributed processing approach. ICS report paper-chilimbi.pdf. 8608, Institute for Cognitive Science, UCSD, La Jolla, 1986. cseweb.ucsd.edu/˜gary/PAPER-[47] M. Li, T. Zhang, Y. Chen, and A. J. Smola. S… find
  13. [39] R. Józefowicz, O. Vinyals, M. Schuster, N. Shazeer, [48] C. J. Maddison, A. Huang, I. Sutskever, and D. Silver. Move evaluation in Go using deep convolutional and Y. Wu. Exploring the limits of lan-neural networks. arXiv preprint arXiv:1412.6564, guage modeling. CoRR, abs/1602.02410, 2016. 201… — [arXiv:1412.6564](https://arxiv.org/abs/1412.6564) find
  14. [49] F. McSherry, M. Isard, and D. G. Murray. Scal- [58] K. Ovtcharov, O. Ruwase, J.-Y. Kim, J. Fowers, ability! But at what COST? In Proceedings K. Strauss, and E. Chung. Toward acceler-of the 15th USENIX Conference on Hot Top-ating deep learning at scale using specialized ics in Operating Systems,… find
  15. USENIX Association. Performance Chips. HOTCHIPS, August 2015. re-www.usenix.org/system/files/conference/hotos15/search.microsoft.com/apps/pubs/default.aspx?id=246506. hotos15-paper-mcsherry.pdf. [59] R. Pascanu, T. Mikolov, and Y. Bengio. On the difficulty of training recurrent neural net-[50] T. Mi… find
  16. JMLR.org, 2013. vector space. In International Conference on www.jmlr.org/proceedings/papers/v28/pascanu13.pdf. find
  17. Learning Representations: Workshops Track, 2013. arxiv.org/abs/1301.3781.
  18. [51] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Ried- [61] J. Ragan-Kelley, C. Barnes, A. Adams, S. Paris, miller, A. K. Fidjeland, G. Ostrovski, S. Petersen, F. Durand, and S. Amarasinghe. Halide: A C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Ku-l… — [doi:10.1038/nature14236.5min.pdf](https://doi.org/10.1038/nature14236.5min.pdf) find
  19. [52] P. Moritz, R. Nishihara, I. Stoica, and M. I. Jor- [62] B. Recht, C. Re, S. Wright, and F. Niu. Hogwild: A lock-free approach to parallelizing stochastic dan. SparkNet: Training deep networks in Spark. gradient descent. In Advances in Neural In-In International Conference on Learning Represen-f… find
  20. ACM, 2013. P. Barham, and M. Abadi. Naiad: a timely dataflow research-srv.microsoft.com/pubs/201110/sosp13-system. In Proceedings of the Twenty-Fourth ACM dandelion-final.pdf. Symposium on Operating Systems Principles, pages 439–455. find
  21. ACM, 2013. Microsoft Research PDF. find
  22. [64] D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning representations by back-[55] A. Nair, P. Srinivasan, S. Blackwell, C. Alci-propagating errors. Cognitive modeling, 5:3, 1988. cek, R. Fearon, A. De Maria, V. Panneershel-www.cs.toronto.edu/ hinton/absps/naturebp.pdf. vam, M. Suleyman,… find
  23. [57] NVIDIA Corporation. NCCL: Optimized primitives for collective multi-gpu communication, 2016. [66] A. Smola and S. Narayanamurthy. An ar-github.com/NVIDIA/nccl. chitecture for parallel topic models. Proc. find
  24. [60] K. Powell. Nvidia devtech blog post. blogs.nvidia.com/blog/2015/03/17/digits-devbox/. find
  25. VLDB Endow., 3(1-2):703–710, Sept. 2010. vldb.org/pvldb/vldb2010/papers/R63.pdf. find
  26. [67] I. Sutskever, J. Martens, G. E. Dahl, and G. E. [75] M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. J. Franklin, S. Shenker, Hinton. On the importance of initialization and and I. Stoica. Resilient distributed datasets: A momentum in deep learning. In Proceedings fault-tolera… find
  27. Implementation. USENIX Association, 2012. jmlr.org/proceedings/papers/v28/sutskever13.pdf. www.usenix.org/system/files/conference/nsdi12/nsdi12-[68] I. Sutskever, O. Vinyals, and Q. V. Le. Sequence final138.pdf. to sequence learning with neural networks. In NIPS, 2014. papers.nips.cc/paper/5346-sequ… — [arXiv:1409.4842](https://arxiv.org/abs/1409.4842) find
  28. [70] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. CoRR, abs/1512.00567, 2015. arxiv.org/abs/1512.00567. find
  29. [71] C. tao Chu, S. K. Kim, Y. an Lin, Y. Yu, G. Bradski, K. Olukotun, and A. Y. Ng. Map-reduce for machine learning on multicore. In B. Schölkopf, J. C. Platt, and T. Hoffman, editors, Advances in Neural Information Processing Systems 19, pages 281–288. MIT Press, 2007. papers.nips.cc/paper/3150-m… find
  30. [72] A. Verma, L. Pedrosa, M. Korupolu, D. Oppenheimer, E. Tune, and J. Wilkes. Large-scale cluster management at Google with Borg. In Proceedings of the Tenth European Conference on Computer Systems, page 18. ACM, 2015. research.google.com/pubs/archive/43438.pdf. find
  31. [73] O. Vinyals, L. Kaiser, T. Koo, S. Petrov, I. Sutskever, and G. Hinton. Grammar as a foreign language. Technical report, arXiv:1412.7449, 2014. arxiv.org/abs/1412.7449. find
  32. [74] Y. Yu, M. Isard, D. Fetterly, M. Budiu, U. Erlingsson, P. K. Gunda, and J. Currey. DryadLINQ: A system for general-purpose distributed data-parallel computing using a high-level language. In Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation, OSDI’08, pages… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/abadi-2016-tensorflow-system-large-scale-machine/