Elamur Research
☰
home / papers / 2015 / MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems

MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/1512.01274

Summary

Deep learning models keep growing in size and cost, and most frameworks commit to one programming style. Imperative code (numpy, Torch7) is flexible. Declarative graph definitions (Caffe, Theano, TensorFlow) allow global optimization. MXNet, built by researchers from several universities and companies, offers both styles in one library.

Users describe network structure with declarative Symbol graphs, which support automatic differentiation. Imperative NDArray operations handle data and parameter updates. KVStore, a two-level parameter server, keeps GPUs and machines in sync. All three run lazily through a single shared dependency engine that tracks both reads and writes of resources, so mixed code runs as fast as a purely symbolic program. Before execution, the graph is pruned, operators are fused, and memory is assigned by two linear-time reuse heuristics.

On standard convnet benchmarks, MXNet matches Torch7 and Caffe in speed. It also needs much less memory and scales super-linearly on a 10-machine GPU cluster. The work matters as a lightweight, portable toolkit with bindings for many languages, usable on hardware from phones up to distributed clusters.

Key points

  • Three core abstractions:

    • Symbol: declarative graphs with forward and backward passes.
    • NDArray: imperative tensors.
    • KVStore: push/pull synchronization with sequential or eventual consistency.

    Bindings exist for C++, Python, R, Go and Julia. The prediction code fits in one dependency-free C++ file of about 50K lines.

  • Memory planning uses the "inplace" (reference counting) and "co-share" (sharing between nodes that cannot run in parallel) heuristics. Together they cut internal memory about 2x for training and 4x for prediction. VGG training needs under 16MB of extra memory.

  • On convnet-benchmarks (GTX 980, batch size 32), MXNet's speed is comparable to Torch7 and Caffe. TensorFlow was about 2x slower, likely because it used an older CUDNN.

  • GoogLeNet with batch normalization was trained on ILSVRC12 (1.3M images, 1,000 classes) using Amazon EC2 g2.8x instances with 4 GK104 GPUs each.

    • One data pass took about 14K s on 1 machine versus 1.4K s on 10 machines.
    • The distributed run overtook single-machine accuracy after about 10 passes, which the authors describe as super-linear speedup.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

194 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 188 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1512.01274 · arXiv 1512.01274

@misc{chen2015mxnet,
  title = {{MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems}},
  author = {Chen, Tianqi and Li, Mu and Li, Yutian and Lin, Min and Wang, Naiyan and Wang, Minjie and Xiao, Tianjun and Xu, Bing and Zhang, Chiyuan and Zhang, Zheng},
  year = {2015},
  doi = {10.48550/arxiv.1512.01274},
  eprint = {1512.01274},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1512.01274}
}

References

14 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian Goodfellow, Arnaud Bergeron, Nicolas Bouchard, David Warde-Farley, and Yoshua Bengio. Theano: new features and speed improvements. arXiv preprint arXiv:1211.5590, 2012. find
  2. Soumith Chintala. Easy benchmarking of all public open-source implementations of convnets, 2015. https://github.com/soumith/convnet-benchmarks. ↗
  3. Ronan Collobert, Koray Kavukcuoglu, and Clément Farabet. Torch7: A matlab-like environment for machine learning. In BigLearn, NIPS Workshop, number EPFL-CONF-192376, 2011. find
  4. J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, Q. Le, M. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, and A. Ng. Large scale distributed deep networks. In Neural Information Processing Systems, 2012. find
  5. Chainer Developers. Chainer: A powerful, flexible, and intuitive framework of neural networks, 2015. http://chainer.org/. ↗
  6. Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015. find
  7. Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the ACM International Conference on Multimedia, pages 675– 678. ACM, 2014. find our card
  8. M. Li, D. G. Andersen, J. Park, A. J. Smola, A. Amhed, V. Josifovski, J. Long, E. Shekita, and B. Y. Su. Scaling distributed machine learning with the parameter server. In OSDI, 2014. find
  9. M. Li, D. G. Andersen, A. J. Smola, and K. Yu. Communication efficient distributed machine learning with the parameter server. In Neural Information Processing Systems, 2014. find
  10. Min Lin, Shuo Li, Xuan Luo, and Shuicheng Yan. Purine: A bi-graph based deep learning framework. arXiv preprint arXiv:1412.6249, 2014. — [arXiv:1412.6249](https://arxiv.org/abs/1412.6249) find
  11. Abadi Martn, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, D… find
  12. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211–252, 201… find our card
  13. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, pages 1–42, 2014. find our card
  14. Minjie Wang, Tianjun Xiao, Jianpeng Li, Jiaxing Zhang, Chuntao Hong, and Zheng Zhang. Minerva: A scalable and highly efficient training platform for deep learning, 2014. find