Elamur Research⌕
☰
home / papers / 2020 / Designing Network Design Spaces

Designing Network Design Spaces

Paper · 2020 · Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick et al.

Summary

Neural architecture search usually returns one network tuned to one setting, such as a specific hardware target. That output says little about why the network works or how to adapt it elsewhere. The authors instead design whole design spaces, meaning parametrized families of architectures. They judge each space by sampling and training many models and examining the resulting error distribution, building on the methodology of Radosavovic et al. [21].

They start from a loosely constrained space called AnyNet, where stage widths and depths vary freely. A sequence of human-guided refinement steps then produces simpler spaces that keep or improve population quality. The result is RegNet, a low-dimensional space where stage widths and depths follow a quantized linear rule. The space is designed cheaply on ImageNet (400MF models, 10 epochs) and then shown to hold for larger compute, longer schedules and other block types.

RegNet models clearly beat ResNe(X)t and do well in the mobile regime. They also outperform EfficientNet at comparable flops and training setups while running up to 5× faster on GPUs. The work matters because it yields transferable design rules rather than one isolated model.

Key points

  • The design process goes from AnyNet to RegNet. Space quality is measured with the error empirical distribution function (EDF) over sampled models, for example n = 500 models in the 400MF, 10-epoch regime.
  • RegNet's main finding is that good networks have stage widths and depths explained by a quantized linear function of the stage/block index.
  • Some findings go against common practice. The best models have roughly 20 blocks regardless of compute regime, and they use neither a bottleneck nor an inverted bottleneck.
  • Training settings were checked on populations of models. lr = 0.1 and wd = 5·10⁻⁵ proved stable across flop regimes and schedule lengths. Among the block types tested (X, R, V, VR), the X block performed best, and residual connections did not help the V block.
  • Training tricks strongly affect EfficientNet-B0 results: without them, error is about 2% worse. Generalization was also tested on ImageNetV2, and code and models are released in pycls.

Interesting ideas

  • I1. Evaluate distributions, not single winners. Sample and train many cheap models from a configuration space and compare error distributions, which removes much of the luck of a single run. The same idea could be used to compare tokenizers, data mixes or model variants in NLP.
  • I2. Refine the space step by step. Each step adds a constraint, such as shared parameters or monotone widths, and keeps it only if the population does not get worse. The result is an interpretable rule rather than a black-box search outcome.
  • I3. Explore in a low-cost regime, then confirm at scale. Run the exploration at small compute and few epochs, then check that the conclusions hold for bigger models and longer training. This makes broad sweeps affordable on limited budgets.
  • I4. Bootstrap confidence ranges for good settings. An empirical bootstrap over sampled models gives a likely range for the best value of each parameter, such as depth, width, learning rate or weight decay. This approach can be applied to hyperparameter studies in any domain.
  • I5. Control the training setup in comparisons. Retrain baselines under the same recipe before comparing. The EfficientNet-B0 gap shows that training enhancements alone can account for differences that might otherwise be credited to the architecture.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MetaMenlo Park · United States214 works · 241,492 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He and Piotr Dollar. 2020. Designing Network Design Spaces. arXiv:2003.13678.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2003.13678 · arXiv 2003.13678

@misc{radosavovic2020designing,
  title = {{Designing Network Design Spaces}},
  author = {Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He and Piotr Dollar},
  year = {2020},
  doi = {10.48550/arxiv.2003.13678},
  eprint = {2003.13678},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2003.13678}
}

References

35 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. F. Chollet. Xception: Deep learning with depthwise separable convolutions. In CVPR, 2017. 7 find
  2. E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le. AutoAugment: Learning augmentation policies from data. arXiv:1805.09501, 2018. 8, 11 find
  3. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 2, 3, 8, 10 find
  4. T. DeVries and G. W. Taylor. Improved regularization of convolutional neural networks with cutout. arXiv:1708.04552, 2017. 8 — [arXiv:1708.04552](https://arxiv.org/abs/1708.04552) find
  5. B. Efron and R. J. Tibshirani. An introduction to the bootstrap. CRC press, 1994. 3 find
  6. P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv:1706.02677, 2017. 11 find
  7. K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, 2015. 2 find
  8. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016. 1, 2, 3, 9
  9. A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861, 2017. 2, 8
  10. J. Hu, L. Shen, and G. Sun. Squeeze-and-excitation networks. In CVPR, 2018. 7 find
  11. G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In CVPR, 2017. 2 find
  12. S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015. 4 find
  13. A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012. 1, 2
  14. G. Larsson, M. Maire, and G. Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. In ICLR, 2017. 8, 11 find
  15. Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1989. 1 find
  16. C.-Y. Lee, S. Xie, P. Gallagher, Z. Zhang, and Z. Tu. Deeply-supervised nets. In AISTATS, 2015. 8 find
  17. C. Liu, B. Zoph, M. Neumann, J. Shlens, W. Hua, L.-J. Li, L. Fei-Fei, A. Yuille, J. Huang, and K. Murphy. Progressive neural architecture search. In ECCV, 2018. 2, 8 find
  18. H. Liu, K. Simonyan, and Y. Yang. Darts: Differentiable architecture search. In ICLR, 2019. 1, 2, 8 find
  19. N. Ma, X. Zhang, H.-T. Zheng, and J. Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In ECCV, 2018. 8 find
  20. H. Pham, M. Y. Guan, B. Zoph, Q. V. Le, and J. Dean. Efficient neural architecture search via parameter sharing. In ICML, 2018. 2
  21. I. Radosavovic, J. Johnson, S. Xie, W.-Y. Lo, and P. Dollár. On network design spaces for visual recognition. In ICCV, 2019. 1, 2, 3, 4, 11 find
  22. P. Ramachandran, B. Zoph, and Q. V. Le. Searching for activation functions. arXiv:1710.05941, 2017. 10 find
  23. E. Real, A. Aggarwal, Y. Huang, and Q. V. Le. Regularized evolution for image classifier architecture search. In AAAI, 2019. 2, 8 find
  24. B. Recht, R. Roelofs, L. Schmidt, and V. Shankar. Do imagenet classifiers generalize to imagenet? arXiv:1902.10811, 2019. 2, 10 find
  25. M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018. 2, 7, 8 find
  26. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015. 1, 2
  27. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In CVPR, 2015. 2 find
  28. C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016. 2 find
  29. M. Tan and Q. V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. ICML, 2019. 1, 2, 7, 9, 10, 11
  30. T. Tieleman and G. Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. Coursera: Neural networks for machine learning, 2012. 11 find
  31. S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He. Aggregated residual transformations for deep neural networks. In CVPR, 2017. 2, 4, 9 find
  32. S. Zagoruyko and N. Komodakis. Wide residual networks. In BMVC, 2016. 2 find
  33. X. Zhang, X. Zhou, M. Lin, and J. Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In CVPR, 2018. 8 find
  34. B. Zoph and Q. V. Le. Neural architecture search with reinforcement learning. In ICLR, 2017. 1
  35. B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le. Learning transferable architectures for scalable image recognition. In CVPR, 2018. 2, 8

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/radosavovic-2020-designing-network-design-spaces/