Elamur Research⌕
☰
home / papers / 2016 / vDNN: Virtualized Deep Neural Networks for Scalable, Memory-Efficient Neural Network Design

vDNN: Virtualized Deep Neural Networks for Scalable, Memory-Efficient Neural Network Design

Paper · 2016 · Minsoo Rhu, Natalia Gimelshein, Jason Clemons et al.

Summary

Training a deep network on a GPU normally requires the whole network's memory footprint to fit in GPU DRAM. This forces researchers to shrink models, cut batch sizes, pick slower but leaner convolution algorithms, or split work across several GPUs. Rhu et al. (NVIDIA) argue that this limit is largely artificial. Frameworks reserve memory for every layer at once, but SGD training runs one layer at a time in a fixed order that repeats every iteration.

They build vDNN, a runtime memory manager that treats GPU and CPU memory as one virtual pool. Intermediate feature maps take most of the memory. vDNN frees them when no later reuse exists. Otherwise it moves them to host memory and fetches them back before backward propagation needs them, overlapping these copies with computation.

On a 12 GB Titan X, vDNN can train networks needing up to 67 GB. This includes VGG-16 at batch 256, which needs 28 GB. The slowdown is 1–18% compared with an ideal GPU that has unlimited memory. The approach is transparent to the programmer.

Key points

  • Under network-wide allocation, 53–79% of allocated GPU memory sits unused at any moment, and the waste grows with network depth.
  • Baseline frameworks could not train 6 of the 10 studied DNNs, which required 14–67 GB against the Titan X's 12 GB. vDNN cut the average GPU memory use of these six by 73–98%.
  • Average GPU memory reductions were 89% for AlexNet, 91% for OverFeat and 95% for GoogLeNet.
  • VGG-16 with batch 256 (28 GB; a single GPU otherwise fits only batch 64) trains on one Titan X with an 18% slowdown relative to the oracular GPU.
  • For non-linear graphs such as GoogLeNet, where outputs fork and join, a per-layer reference count of consumer layers decides when a feature map can be released.
  • Memory is accounted for by type: weights, feature maps, gradient buffers (dX/dY, reused across layers) and the cuDNN workspace.

Interesting ideas

  • I1. Use the static execution order as a schedule. Training repeats the same layer sequence millions of times, so data movement can be planned ahead rather than handled by reactive paging. Any workload with a fixed, repeating pipeline could prefetch or evict data this way.
  • I2. Sort data by reuse distance. Data with no future use is freed immediately. Data needed much later is moved to slower, larger memory and brought back just in time. This simple three-way policy (keep, offload, release) carries over to other memory hierarchies, such as GPU→CPU or RAM→disk.
  • I3. Reference counting on the layer graph. Counting how many consumers still need a tensor gives a cheap, safe release rule for DAG-shaped models, not only linear chains.
  • I4. Hide transfers behind computation. Running copies concurrently with layer computation is what keeps the slowdown small. This is worth copying whenever data must move between memory tiers.
  • I5. Compare peak per-layer usage with total allocation. Reporting what fraction of reserved memory is actually live is a quick diagnostic for over-provisioning. Benchmarking against an oracle with unlimited memory gives an honest upper bound on overhead.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢NvidiaSanta Clara · United States37 works · 12,656 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 worksInformation Systems · 1,730 worksHardware and Architecture · 167 works

🧭 Topics 1Research subjects and languages the work is about.Efficient models & compression · 297 works

Cite

Cite as Minsoo Rhu, Natalia Gimelshein, Jason Clemons, Arslan Zulfiqar and Stephen W. Keckler. 2016. vDNN: Virtualized Deep Neural Networks for Scalable, Memory-Efficient Neural Network Design. MICRO 2016.

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 1602.08124

@inproceedings{rhu2016vdnn,
  title = {{vDNN: Virtualized Deep Neural Networks for Scalable, Memory-Efficient Neural Network Design}},
  author = {Minsoo Rhu, Natalia Gimelshein, Jason Clemons, Arslan Zulfiqar and Stephen W. Keckler},
  booktitle = {MICRO 2016},
  year = {2016},
  eprint = {1602.08124},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1602.08124}
}

References

53 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Krizhevsky, I. Sutskever, and G. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” in Proceedings of the Advances in Neural Information Processing Systems, 2012.
  2. A. Graves and J. Schmidhuber, “Framewise Phoneme Classification With Bidirectional LSTM and Other Neural Network Architectures,” in Neural Networks, 2005. find
  3. R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa, “Natural Language Processing (Almost) From Scratch,” in arxiv.org, 2011.
  4. Tensorflow, “https://www.tensorflow.org,” 2016. find
  5. Torch, “http://torch.ch,” 2016. find
  6. Theano, “http://deeplearning.net/tutorial,” 2016. find
  7. Caffe, “http://caffe.berkeleyvision.org,” 2016. find
  8. NVIDIA, “cuDNN: GPU Accelerated Deep Learning,” 2016. find
  9. S. Bahrampour, N. Ramakrishnan, L. Schott, and M. Shah, “Comparative Study of Caffe, Neon, Theano, and Torch for Deep Learning,” in arxiv.org, 2016. find
  10. The Next Platform (www.nextplatform.com), “Baidu Eyes Deep Learning Strategy in Wake of New GPU Options,” 2016. find
  11. G. Diamos, S. Sengupta, B. Catanzaro, M. Chrzanowski, A. Coates, E. Elsen, J. Engel, A. Hannun, and S. Satheesh, “Persistent RNNs: Stashing Recurrent Weights On-Chip,” in Proceedings of the International Conference on Machine Learning, 2016. find
  12. A. Krizhevsky, “One Weird Trick For Parallelizing Convolutional Neural Networks,” in arxiv.org, 2014.
  13. ImageNet, “http://image-net.org,” 2016. find
  14. K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” in Proceedings of the International Conference on Learning Representations, 2015.
  15. K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in arxiv.org, 2015.
  16. Wired (www.wired.com), “Microsoft Neural Net Shows Deep Learning Can Get Way Deeper,” 2016. find
  17. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going Deeper with Convolutions,” in arxiv.org, 2014. find
  18. G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Weinberger, “Deep Networks with Stochastic Depth,” in arxiv.org, 2016.
  19. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-Based Learning Applied to Document Recognition,” in Proceedings of the IEEE, 1998. find
  20. T. Chen, Z. Du, N. Sun, J. Wang, C. Wu, Y. Chen, and O. Temam, “DianNao: a small-footprint high-throughput accelerator for ubiquitous machine-learning,” in Proceedings of International Conference on Architectural Support for Programming Languages and Operating Systems, 2014. find
  21. Y. Chen, T. Luo, S. Liu, S. Zhang, L. He, J. Wang, L. Li, T. Chen, Z. Xu, N. Sun, and O. Temam, “DaDianNao: A Machine-Learning Supercomputer,” in Proceedings of ACM/IEEE International Symposium on Microarchitecture, 2014. find
  22. Y. Chen, T. Krishna, J. Emer, and V. Sze, “Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks,” in IEEE International Conference on Solid-State Circuits, 2016. find
  23. S. Han, X. Liu, H. Mao, J. Pu, A. Pedram, M. Horowitz, and W. Dally, “EIE: Efficient Inference Engine on Compressed Deep Neural Network,” in Proceedings of ACM/IEEE International Symposium on Computer Architecture, 2016.
  24. Y. Chen, J. Emer, and V. Sze, “Eyeriss: A Spatial Architecture for Energy-Efficient Dataflow for Convolutional Neural Networks,” in Proceedings of ACM/IEEE International Symposium on Computer Architecture, 2016. find
  25. R. LiKamWa, Y. Hou, M. Polansky, Y. Gao, and L. Zhong, “RedEye: Analog ConvNet Image Sensor Architecture for Continuous Mobile Vision,” in Proceedings of ACM/IEEE International Symposium on Computer Architecture, 2016. find
  26. B. Reagen, P. Whatmough, R. Adolf, S. Rama, H. Lee, S. Lee, J. Miguel, H. Lobato, G. Wei, and D. Brooks, “Minerva: Enabling Low-Power, High-Accuracy Deep Neural Network Accelerators,” in Proceedings of ACM/IEEE International Symposium on Computer Architecture, 2016. find
  27. P. Chi, S. Li, C. Xu, T. Zhang, J. Zhao, Y. Liu, Y. Wang, and Y. Xie, “A Novel Processing-in-memory Architecture for Neural Network Computation in ReRAM-based Main Memory,” in Proceedings of ACM/IEEE International Symposium on Computer Architecture, 2016. find
  28. A. Shafiee, A. Nag, N. Muralimanohar, R. Balasubra-monian, J. P. Strachan, M. Hu, R. S. Williams, and V. Srikumar, “ISAAC: A Convolutional Neural Network Accelerator with In-Situ Analog Arithmetic in Crossbars,” in Proceedings of ACM/IEEE International Symposium on Computer Architecture, 2016. find
  29. J. Albericio, P. Judd, T. Hetherington, T. Aamodt, N. E. Jerger, and A. Moshovos, “Cnvlutin: Ineffectual-Neuron-Free Deep Convolutional Neural Network Computing,” in Proceedings of ACM/IEEE International Symposium on Computer Architecture, 2016. find
  30. P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun, “OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks,” in arxiv.org, 2013.
  31. S. Chintala, “https://github.com/torch/nn/pull/235,” 2015. find
  32. S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, and E. Shelhamer, “cuDNN: Efficient Primitives for Deep Learning,” in Proceedings of the Advances in Neural Information Processing Systems, 2014.
  33. OpenMP Architecture Review Board, “OpenMP Application Program Interface (version 4.0),” 2013. find
  34. T. Zheng, D. Nellans, A. Zulfiqar, M. Stephenson, and S. W. Keckler, “Toward High-Performance Paged-Memory for GPUs,” in Proceedings of IEEE International Symposium on High-Performance Computer Architecture, 2016. find
  35. NVIDIA, “NVIDIA NVLINK High-Speed Interconnect,” 2016. find
  36. NVIDIA, “NVIDIA CUDA Programming Guide,” 2016. find
  37. NVIDIA, “https://github.com/NVIDIA/cnmem,” 2016. find
  38. S. Gross and M. Wilber, “Training and Investigating Residual Nets,” 2016. find
  39. T. Chen, M. Li, Y. Li, M. Lin, N. Wang, M. Wang, T. Xiao, B. Xu, C. Zhang, and Z. Zhang, “MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems,” in Proceedings of the 2015 Workshop on Machine Learning Systems, 2015.
  40. NVIDIA, “GeForce GTX Titan X (Maxwell),” 2015. find
  41. S. Chintala, “https://github.com/soumith/convnet-benchmarks,” 2016. find
  42. NVIDIA, “CUDA Toolkit 7.5 Documentation: Profiler,” 2016. find
  43. S. Hanson and L. Pratt, “Comparing Biases for Minimal Network Construction with Back-propagation,” in Proceedings of the Advances in Neural Information Processing Systems, 1989. find
  44. Y. LeCun, S. Denker, and S. Solla, “Optimal Brain Damage,” in Proceedings of the Advances in Neural Information Processing Systems, 1990. find
  45. B. Hassibi and D. Stork, “Second Order Derivatives for Network Pruning: Optimal Brain Surgeon,” in Proceedings of the Advances in Neural Information Processing Systems, 1993. find
  46. S. Han, J. Pool, J. Tran, and W. Dally, “Learning Both Weights and Connections for Efficient Neural Networks,” in Proceedings of the Advances in Neural Information Processing Systems, 2015. find
  47. S. Han, H. Mao, and W. Dally, “Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding,” in Proceedings of the International Conference on Learning Representations, 2016. find
  48. V. Vanhoucke, A. Senior, and M. Mao, “Improving the Speed of Neural Networks on CPUs,” in Proceedings of Deep Learning and Unsupervised Feature Learning, 2011. find
  49. P. Judd, J. Albericio, T. Hetherington, T. Aamodt, N. E. Jerger, R. Urtasun, and A. Moshovos, “Reduced-Precision Strategies for Bounded Memory in Deep Neural Nets,” in arxiv.org, 2016. find
  50. Y. Gong, L. Liu, M. Yang, and L. Bourdev, “Compressing Deep Convolutional Networks Using Vector Quantization,” in arxiv.org, 2014. find
  51. B. Pichai, L. Hsu, and A. Bhattacharjee, “Architectural Support for Address Translation on GPUs: Designing Memory Management Units for CPU/GPUs with Unified Address Spaces,” in Proceedings of ACM International Conference on Architectural Support for Programming Languages and Operating Systems, 2014. find
  52. J. Power, M. Hill, and D. Wood, “Supporting x86-64 Address Translation for 100s of GPU Lanes,” in Proceedings of IEEE International Symposium on High-Performance Computer Architecture, 2014. find
  53. Z. Du, R. Fasthuber, T. Chen, P. Ienne, L. Li, T. Luo, X. Feng, Y. Chen, and O. Temam, “ShiDianNao: Shifting Vision Processing Closer to the Sensor,” in Proceedings of ACM/IEEE International Symposium on Computer Architecture, 2015. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rhu-2016-vdnn-virtualized-deep-neural-networks/