Elamur Research⌕
☰
home / papers / 2018 / ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware

ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware

Paper · 2018 · Han Cai, Ligeng Zhu and Song Han

Summary

Neural architecture search typically costs thousands of GPU hours. Differentiable variants such as DARTS and One-Shot are cheaper, but they keep the activations of every candidate operation in memory, so memory grows with the number of choices. As a result, earlier methods search on cheaper stand-ins, such as smaller datasets, shallow networks or short training, and then transfer one repeated cell. That cell may not suit the real task or device. ProxylessNAS treats search as pruning paths inside an over-parameterized network. Each edge holds all candidate operations, but the architecture parameters are binarized so that only one path is active at a time. These gates are trained with a BinaryConnect-style gradient. Hardware cost is made learnable by expressing latency as a differentiable regularization term, with a REINFORCE variant as an alternative. Search runs directly on ImageNet and CIFAR-10 at roughly the cost of normal training, and every block is learned individually. The resulting models outperform strong manual and searched baselines in both accuracy and latency. They also show that different hardware favors differently shaped networks.

Key points

  • On CIFAR-10, the found model reaches 2.08% test error with 5.7M parameters. This beats AmoebaNet-B while using about 6× fewer parameters.
  • On ImageNet, the model hits 75.1% top-1 accuracy. That is 3.1% above MobileNetV2, and the model runs 1.2× faster in measured GPU latency.
  • The mobile ImageNet search took about 200 GPU hours, which the authors estimate is 200× less than MnasNet (around 40,000 V100 GPU hours). The resulting model matches MobileNetV2 1.4 accuracy while running 1.8× faster.
  • Path binarization cuts memory use by about an order of magnitude. The CIFAR-10 search space has 7 candidate operations: several depthwise-separable convolutions, a dilated depthwise-separable convolution, identity, and average and max pooling.
  • The mobile latency predictor was built from 5k sampled architectures (4k for fitting, the rest for testing) measured on a Google Pixel 1 with TensorFlow-Lite. It has an RMSE of 0.75 ms.
  • Pretrained models and evaluation code are available at github.com/MIT-HAN-LAB/ProxylessNAS.

Interesting ideas

  • I1. Search as pruning. Instead of training a separate controller, the method trains one oversized network and lets learned gates decide which branches survive. This links architecture choice to model compression and could be reused wherever we want to choose among module variants, such as layers in a text model.
  • I2. Sample one path per step. Activating a single sampled branch, rather than mixing all of them, keeps memory near that of a compact model. This makes searching large option sets feasible on ordinary GPUs and full-size data, without falling back to toy proxies.
  • I3. Recompute inactive branches in backward. Outputs of unused paths are needed only to compute gate gradients. They can be computed during the backward pass and freed immediately, which saves memory without approximation. The same trick applies to any gated mixture where some branches are idle in the forward pass.
  • I4. Cheap latency model as a differentiable penalty. A small regressor fitted on measurements from one device can replace a farm of phones. Folding its prediction into the loss lets deployment cost be optimized jointly with accuracy, which is relevant for shipping NLP models to CPUs or mobile devices.
  • I5. Search with estimates, report measurements. The predicted latency guides the search, but final numbers come from real on-device runs. This is a good evaluation habit whenever a surrogate metric drives optimization.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Massachusetts Institute of TechnologyCambridge · United States243 works · 76,506 citations🎓Shanghai Jiao Tong UniversityShanghai · China212 works · 19,544 citations🎓Simon Fraser UniversityBurnaby · Canada19 works · 6,205 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.GPU & parallel computing · 41 works

Cite

Cite as Han Cai, Ligeng Zhu and Song Han. 2018. ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware. arXiv:1812.00332.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1812.00332 · arXiv 1812.00332

@misc{cai2018proxylessnas,
  title = {{ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware}},
  author = {Han Cai, Ligeng Zhu and Song Han},
  year = {2018},
  doi = {10.48550/arxiv.1812.00332},
  eprint = {1812.00332},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1812.00332}
}

References

38 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. April Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc Le. Understanding and simplifying one-shot architecture search. In ICML, 2018. find
  2. Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Smash: one-shot model architecture search through hypernetworks. In ICLR, 2018. find
  3. Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient architecture search by network transformation. In AAAI, 2018a.
  4. May Han Cai, Jiacheng Yang, Weinan Zhang, Song Han, and Yong Yu. Path-level network transformation for efficient architecture search. In ICML, 2018b. find
  5. Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In NIPS, 2015.
  6. Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552, 2017. — [arXiv:1708.04552](https://arxiv.org/abs/1708.04552) find
  7. Jin-Dong Dong, An-Chieh Cheng, Da-Cheng Juan, Wei Wei, and Min Sun. Dpp-net: Device-aware progressive search for pareto-optimal neural architectures. In ECCV, 2018. find
  8. Thomas Elsken, Jan-Hendrik Metzen, and Frank Hutter. Simple and efficient architecture search for convolutional neural networks. arXiv preprint arXiv:1711.04528, 2017. find
  9. Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Multi-objective architecture search for cnns. arXiv preprint arXiv:1804.09081, 2018a. — [arXiv:1804.09081](https://arxiv.org/abs/1804.09081) find
  10. Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. arXiv preprint arXiv:1808.05377, 2018b. — [arXiv:1808.05377](https://arxiv.org/abs/1808.05377) find
  11. Dongyoon Han, Jiwhan Kim, and Junmo Kim. Deep pyramidal residual networks. In CVPR, 2017. find
  12. Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In NIPS, 2015. find
  13. Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In ICLR, 2016. find
  14. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
  15. Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han. Amc: Automl for model compression and acceleration on mobile devices. In ECCV, 2018.
  16. Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.
  17. Chi-Hung Hsu, Shu-Huan Chang, Da-Cheng Juan, Jia-Yu Pan, Yu-Ting Chen, Wei Wei, and Shih-Chieh Chang. Monas: Multi-objective neural architecture search using reinforcement learning. arXiv preprint arXiv:1806.10332, 2018. — [arXiv:1806.10332](https://arxiv.org/abs/1806.10332) find
  18. Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In ECCV, 2016.
  19. Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017. find
  20. Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and¡ 0.5 mb model size. arXiv preprint arXiv:1602.07360, 2016. find
  21. Purushotham Kamath, Abhishek Singh, and Debo Dutta. Neural architecture construction using envelopenets. arXiv preprint arXiv:1803.06744, 2018. find
  22. Chenxi Liu, Barret Zoph, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In ECCV, 2018a. find
  23. Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu. Hierarchical representations for efficient architecture search. In ICLR, 2018b. find
  24. Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055, 2018c. find
  25. Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In ICCV, 2017. find
  26. Renqian Luo, Fei Tian, Tao Qin, and Tie-Yan Liu. Neural architecture optimization. arXiv preprint arXiv:1808.07233, 2018. find
  27. Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In ECCV, 2018. find
  28. Hieu Pham, Melody Y Guan, Barret Zoph, Quoc V Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In ICML, 2018.
  29. Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. arXiv preprint arXiv:1802.01548, 2018. find
  30. Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018. find
  31. Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, and Quoc V Le. Mnasnet: Platform-aware neural architecture search for mobile. arXiv preprint arXiv:1807.11626, 2018. find
  32. Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quantization. arXiv, 2018. find
  33. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. In Reinforcement Learning. 1992. find
  34. Yoshihiro Yamada, Masakazu Iwamura, and Koichi Kise. Shakedrop regularization. arXiv preprint arXiv:1802.02375, 2018. — [arXiv:1802.02375](https://arxiv.org/abs/1802.02375) find
  35. Zhao Zhong, Junjie Yan, Wei Wu, Jing Shao, and Cheng-Lin Liu. Practical block-wise neural network architecture generation. In CVPR, 2018. find
  36. Ligeng Zhu, Ruizhi Deng, Michael Maire, Zhiwei Deng, Greg Mori, and Ping Tan. Sparsely aggregated convolutional networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 186–201, 2018. find
  37. Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. In ICLR, 2017.
  38. Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In CVPR, 2018.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cai-2018-proxylessnas-direct-neural-architecture-search/