Elamur Research⌕
☰
home / papers / 2019 / Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation

Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation

Paper · 2019 · Xin Chen, Lingxi Xie, Jun Wu et al.

Summary

Differentiable NAS methods such as DARTS cut search costs a lot, but the cells they find often lose accuracy when used in the final model or moved to another dataset. The authors attribute this to a "depth gap": DARTS searches inside a shallow 8-cell network, while the chosen cell is evaluated in a 20-cell one, so it favours operations that suit shallow nets. P-DARTS splits the search into stages and makes the search network deeper at each stage. Two techniques make this workable:

  • Search space approximation removes the weakest candidate operations between stages, which keeps memory and time in check.
  • Search space regularization stops the search from collapsing onto skip-connect, using a decaying operation-level Dropout and a cap on skip-connects.

The search takes about 7 hours on a single GPU. The resulting cell reaches 2.50% test error on CIFAR10 and transfers well to ImageNet under the mobile setting. This shows that depth consistency between search and evaluation matters, and that it can be bought cheaply by giving up operation breadth.

Key points

  • Staged depth: the search network grows from 5 to 11 to 17 cells across stages. After each stage, the lowest-scoring candidate operations on every edge are removed.
  • Cost: search takes about 0.3 GPU-days on one Tesla P100 (16GB). Peak memory is about 9.8 GB in stage 1 and about 14.0–14.5 GB in later stages. The authors call this the fastest NAS result reaching 3% CIFAR10 error, ahead of ENAS.
  • Results:
    • CIFAR10: 2.50% test error with 3.4M parameters.
    • CIFAR100: 15.92% test error with 3.6M parameters.
    • ImageNet transfer: 24.4% / 7.4% top-1/top-5 error.
  • Ablation on approximation: running only the final stage on a randomly sampled operation set gives at best 3.43% error, versus 2.58% with the progressively pruned set.
  • Observed cell structure: deeper search networks produce cells with deeper internal connections, while shallow ones keep shallow links.

Interesting ideas

  • I1. Match proxy conditions to deployment conditions. If a design choice is tuned in a cheaper setting, move that setting gradually toward the real one during tuning. This can catch preferences that only hold in the cheap setting, and it applies to any proxy-based model selection.
  • I2. Trade breadth for depth during search. Use early, cheap stages to prune the candidate set, then spend the saved budget on a more realistic, costlier configuration. This general coarse-to-fine pruning pattern fits any expensive search under a fixed memory limit.
  • I3. Temporarily handicap "shortcut" options. Options that lower the loss quickly but learn little can dominate early optimization. Dropout on those paths, decayed over time, lets slower but stronger alternatives get trained before the competition is judged fairly.
  • I4. Add hard rules on top of learned scores. Capping how many skip-connects the final cell may contain is a simple guard against a known failure mode of differentiable selection. It could carry over to other soft-selection or mixture-weight schemes.
  • I5. Compare against a random subset of equal size. Testing the pruned search space against randomly sampled ones of the same size, with several seeds and the best result kept, cleanly isolates whether the pruning itself adds value.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Huawei TechnologiesShenzhen · China96 works · 15,432 citations🎓Tongji UniversityShanghai · China33 works · 2,993 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Xin Chen, Lingxi Xie, Jun Wu and Qi Tian. 2019. Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation. arXiv:1904.12760.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1904.12760 · arXiv 1904.12760

@misc{chen2019progressive,
  title = {{Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation}},
  author = {Xin Chen, Lingxi Xie, Jun Wu and Qi Tian},
  year = {2019},
  doi = {10.48550/arxiv.1904.12760},
  eprint = {1904.12760},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1904.12760}
}

References

37 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. B. Baker, O. Gupta, N. Naik, and R. Raskar. Designing neural network architectures using reinforcement learning. In ICLR, 2017. find
  2. H. Cai, L. Zhu, and S. Han. ProxylessNAS: Direct neural architecture search on target task and hardware. arXiv preprint arXiv:1812.00332, 2018.
  3. X. Chu, B. Zhang, H. Ma, R. Xu, J. Li, and Q. Li. Fast, accurate and lightweight super-resolution with neural architecture search. arXiv preprint arXiv:1901.07261, 2019. — [arXiv:1901.07261](https://arxiv.org/abs/1901.07261) find
  4. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009. find
  5. T. DeVries and G. W. Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552, 2017. — [arXiv:1708.04552](https://arxiv.org/abs/1708.04552) find
  6. T. Elsken, J. H. Metzen, and F. Hutter. Neural architecture search: A survey. arXiv preprint arXiv:1808.05377, 2018. — [arXiv:1808.05377](https://arxiv.org/abs/1808.05377) find
  7. P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. Accurate, large minibatch SGD: Training ImageNet in 1 hour. arXiv preprint arXiv:1706.02677, 2017. find
  8. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
  9. A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.
  10. G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In CVPR, 2017. find
  11. S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015. find
  12. A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009. find
  13. A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet classification with deep convolutional neural networks. In NIPS, 2012.
  14. G. Larsson, M. Maire, and G. Shakhnarovich. FractalNet: Ultra-deep neural networks without residuals. In ICLR, 2017. find
  15. C. Liu, L.-C. Chen, F. Schroff, H. Adam, W. Hua, A. Yuille, and L. Fei-Fei. Auto-DeepLab: Hierarchical neural architecture search for semantic image segmentation. arXiv preprint arXiv:1901.02985, 2019. find
  16. C. Liu, B. Zoph, M. Neumann, J. Shlens, W. Hua, L.-J. Li, L. Fei-Fei, A. Yuille, J. Huang, and K. Murphy. Progressive neural architecture search. In ECCV, 2018. find
  17. H. Liu, K. Simonyan, O. Vinyals, C. Fernando, and K. Kavukcuoglu. Hierarchical representations for efficient architecture search. In ICLR, 2018. find
  18. H. Liu, K. Simonyan, and Y. Yang. DARTS: Differentiable architecture search. arXiv preprint arXiv:1806.09055, 2018. find
  19. N. Ma, X. Zhang, H.-T. Zheng, and J. Sun. ShuffleNet V2: Practical guidelines for efficient cnn architecture design. In ECCV, 2018. find
  20. A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. De-Vito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in Pytorch. In NIPS-W, 2017. find
  21. H. Pham, M. Y. Guan, B. Zoph, Q. V. Le, and J. Dean. Efficient neural architecture search via parameter sharing. In ICML, 2018.
  22. E. Real, A. Aggarwal, Y. Huang, and Q. V. Le. Regularized evolution for image classifier architecture search. arXiv preprint arXiv:1802.01548, 2018. find
  23. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. ImageNet large scale visual recognition challenge. IJCV, 115(3):211–252, 2015.
  24. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
  25. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. JMLR, 15(1):1929–1958, 2014.
  26. R. K. Srivastava, K. Greff, and J. Schmidhuber. Training very deep networks. In NIPS, 2015. find
  27. K. O. Stanley and R. Miikkulainen. Evolving neural networks through augmenting topologies. Evolutionary Computation, 10(2):99–127, 2002. find
  28. M. Suganuma, S. Shirakawa, and T. Nagao. A genetic programming approach to designing convolutional neural network architectures. In GECCO, 2017. find
  29. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In CVPR, 2015. find
  30. C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016. find
  31. M. Tan, B. Chen, R. Pang, V. Vasudevan, and Q. V. Le. MnasNet: Platform-aware neural architecture search for mobile. arXiv preprint arXiv:1807.11626, 2018. find
  32. L. Xie and A. Yuille. Genetic CNN. In ICCV, 2017. find
  33. S. Xie, H. Zheng, C. Liu, and L. Lin. SNAS: Stochastic neural architecture search. arXiv preprint arXiv:1812.09926, 2018. — [arXiv:1812.09926](https://arxiv.org/abs/1812.09926) find
  34. X. Zhang, X. Zhou, M. Lin, and J. Sun. ShuffleNet: An extremely efficient convolutional neural network for mobile devices. In CVPR, 2018. find
  35. Z. Zhong, J. Yan, W. Wu, J. Shao, and C.-L. Liu. Practical block-wise neural network architecture generation. In CVPR, 2018. find
  36. B. Zoph and Q. V. Le. Neural architecture search with reinforcement learning. In ICLR, 2017.
  37. B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le. Learning transferable architectures for scalable image recognition. In CVPR, 2018.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2019-progressive-differentiable-architecture-search-bridging/