Elamur Research⌕
☰
home / papers / 2019 / Once-for-All: Train One Network and Specialize it for Efficient Deployment

Once-for-All: Train One Network and Specialize it for Efficient Deployment

Paper · 2019 · Han Cai, Chuang Gan, Tianzhe Wang et al.

Summary

Deploying neural networks across many devices (phones, CPUs, GPUs, FPGAs, microcontrollers) and latency budgets usually means designing or searching a new architecture and training it from scratch for each target, so the cost grows with every new scenario. The authors separate training from architecture search. They train one large "once-for-all" (OFA) network whose weights are shared by a huge family of sub-networks that differ in depth, width, kernel size and input resolution. For a new device, a sub-network is picked using cheap accuracy and latency predictors, with no retraining. To keep so many weight-sharing sub-networks accurate, they propose progressive shrinking: train the biggest model first, then fine-tune step by step so that ever smaller variants are supported, distilling from the large model. On ImageNet, the chosen sub-networks match or beat individually trained, NAS-designed models such as MobileNetV3 and EfficientNet. Design cost per scenario becomes roughly constant, which saves GPU hours and CO2.

Key points

  • The OFA space contains more than 10^19 sub-networks. Slimmable Nets, by comparison, support only 4 width settings.
  • Results on ImageNet:
    • Up to 4.0% higher top-1 accuracy than MobileNetV3.
    • At equal accuracy, 1.5× lower measured latency than MobileNetV3 and 2.6× lower than EfficientNet.
    • 80.0% top-1 with 595M MACs, the first result at 80% in the mobile (<600M MACs) setting.
  • The progressive shrinking schedule:
    • Kernel size {7, 5, 3}, sampling one sub-network per step.
    • Depth {4, 3, 2} per stage, sampling two sub-networks per step.
    • Width expand ratio {6, 4, 3}, sampling four sub-networks per step.
  • The accuracy predictor is a 3-layer MLP with 400 hidden units. Its input is one-hot encodings of each layer's kernel size and expand ratio (zeros for skipped layers) plus the resolution. It reaches 0.21% RMSE.
  • OFA took first place in the DSP classification track of the 3rd LPCVC, and in both the classification and detection tracks of the 4th edition of that competition. Code and 50 pretrained models are available at mit-han-lab/once-for-all.

Interesting ideas

  • I1. Train once, specialize many times. Separating expensive shared training from cheap per-target selection turns a per-scenario cost into a one-time cost. This fits any setting with many deployment targets, for example NLP models for servers and mobile apps.
  • I2. Shrinking as generalized pruning. Each new stage starts from the larger network and keeps the most important channels, ranked by weight importance, as the shared core. This gives small variants a good starting point instead of letting them compete from random weights.
  • I3. Increase sub-network sampling as flexibility grows. The number of sub-networks whose gradients are aggregated per step rises (1, then 2, then 4) as more elastic dimensions are added. This is a simple way to control interference among weight-sharing models.
  • I4. Use a cheap surrogate for search. A small MLP over a one-hot architecture encoding replaces expensive evaluation of candidates. The authors also show that lower predictor error tends to give better final picks, so predictor quality is worth checking explicitly.
  • I5. Report measured latency and carbon cost. Comparing methods by latency measured on real devices, and accounting for GPU hours and CO2, gives a more honest picture of efficiency than MAC counts alone.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Massachusetts Institute of TechnologyCambridge · United States243 works · 76,506 citations🏢IBMArmonk · United States154 works · 72,606 citations🎓Shanghai Jiao Tong UniversityShanghai · China212 works · 19,544 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang and Song Han. 2019. Once-for-All: Train One Network and Specialize it for Efficient Deployment. arXiv:1908.09791.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1908.09791 · arXiv 1908.09791

@misc{cai2019once,
  title = {{Once-for-All: Train One Network and Specialize it for Efficient Deployment}},
  author = {Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang and Song Han},
  year = {2019},
  doi = {10.48550/arxiv.1908.09791},
  eprint = {1908.09791},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1908.09791}
}

References

44 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anubhav Ashok, Nicholas Rhinehart, Fares Beainy, and Kris M Kitani. N2n learning: Network to network compression via policy gradient reinforcement learning. In ICLR, 2018. find
  2. Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient architecture search by network transformation. In AAAI, 2018a.
  3. Han Cai, Jiacheng Yang, Weinan Zhang, Song Han, and Yong Yu. Path-level network transformation for efficient architecture search. In ICML, 2018b. find
  4. Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct neural architecture search on target task and hardware. In ICLR, 2019. URL https://arxiv.org/pdf/1812.00332.pdf. ↗
  5. Brian Cheung, Alex Terekhov, Yubei Chen, Pulkit Agrawal, and Bruno Olshausen. Superposition of many models into one. In NeurIPS, 2019. find
  6. Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In NeurIPS, 2015.
  7. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. find
  8. Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. arXiv preprint arXiv:1904.00420, 2019. — [arXiv:1904.00420](https://arxiv.org/abs/1904.00420) find
  9. Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In NeurIPS, 2015. find
  10. Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In ICLR, 2016. find
  11. Cong Hao, Xiaofan Zhang, Yuhong Li, Sitao Huang, Jinjun Xiong, Kyle Rupnow, Wen-mei Hwu, and Deming Chen. Fpga/dnn co-design: An efficient design methodology for 1ot intelligence on the edge. In 2019 56th ACM/IEEE Design Automation Conference (DAC), pp. 1–6. find
  12. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
  13. Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Amc: Automl for model compression and acceleration on mobile devices. In ECCV, 2018.
  14. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
  15. Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In ICCV 2019, 2019. find
  16. Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.
  17. Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017. find
  18. Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Q Weinberger. Multi-scale dense networks for resource efficient image classification. In ICLR, 2018. find
  19. Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 0.5 mb model size. arXiv preprint arXiv:1602.07360, 2016. find
  20. Weiwen Jiang, Lei Yang, Edwin Sha, Qingfeng Zhuge, Shouzhen Gu, Yiyu Shi, and Jingtong Hu. Hardware/software co-exploration of neural architectures. arXiv preprint arXiv:1907.04650, 2019a. — [arXiv:1907.04650](https://arxiv.org/abs/1907.04650) find
  21. Weiwen Jiang, Xinyi Zhang, Edwin H-M Sha, Lei Yang, Qingfeng Zhuge, Yiyu Shi, and Jingtong Hu. Accuracy vs. efficiency: Achieving both through fpga-implementation aware neural architecture search. In Proceedings of the 56th Annual Design Automation Conference 2019, pp. 1–6, 2019b. find
  22. Jason Kuen, Xiangfei Kong, Zhe Lin, Gang Wang, Jianxiong Yin, Simon See, and Yap-Peng Tan. Stochastic downsampling for cost-adjustable inference and improved regularization in convolutional networks. In CVPR, 2018. find
  23. Ji Lin, Yongming Rao, Jiwen Lu, and Jie Zhou. Runtime neural pruning. In NeurIPS, 2017. find
  24. Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In ECCV, 2018. find
  25. Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In ICLR, 2019. find
  26. Lanlan Liu and Jia Deng. Dynamic deep neural networks: Optimizing accuracy-efficiency trade-offs by selective execution. In AAAI, 2018. find
  27. Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In ICCV, 2017. find
  28. Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016. find
  29. Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In ECCV, 2018. find
  30. Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In AAAI, 2019. find
  31. Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018. find
  32. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. In ACL, 2019.
  33. Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2820–2828, 2019. find
  34. Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E Gonzalez. Skipnet: Learning dynamic routing in convolutional networks. In ECCV, 2018. find
  35. Samuel Williams, Andrew Waterman, and David Patterson. Roofline: An insightful visual performance model for floating-point programs and multicore architectures. Technical report, Lawrence Berkeley National Lab.(LBNL), Berkeley, CA (United States), 2009. find
  36. Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In CVPR, 2019.
  37. Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris. Blockdrop: Dynamic inference paths in residual networks. In CVPR, 2018. find
  38. Jiahui Yu and Thomas Huang. Autoslim: Towards one-shot architecture search for channel numbers. arXiv preprint arXiv:1903.11728, 2019a. — [arXiv:1903.11728](https://arxiv.org/abs/1903.11728) find
  39. Jiahui Yu and Thomas Huang. Universally slimmable networks and improved training techniques. In ICCV, 2019b. find
  40. Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. In ICLR, 2019. find
  41. Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In CVPR, 2018. find
  42. Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization. In ICLR, 2017. find
  43. Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. In ICLR, 2017.
  44. Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In CVPR, 2018.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cai-2019-once-all-train-one-network/