Elamur Research
☰
home / papers / 2017 / Learning Transferable Architectures for Scalable Image Recognition

Learning Transferable Architectures for Scalable Image Recognition

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1707.07012

Summary

Designing convolutional networks for image classification by hand demands a lot of expert effort, and running Neural Architecture Search (NAS) directly on a large dataset like ImageNet is prohibitively costly. The authors solve this by searching on a small proxy dataset, CIFAR-10, and then transferring the result to ImageNet. Their main contribution is the "NASNet search space". Instead of designing a whole network, an RNN controller trained with reinforcement learning (policy gradient) designs only two reusable building blocks. The Normal Cell preserves the feature-map size, and the Reduction Cell halves its height and width. The full network is a manually fixed stack of these cells, so depth and filter count can be adjusted without repeating the search. The authors also introduce ScheduledDropPath, a regularizer that improves generalization. The resulting NASNet models reach state-of-the-art accuracy on CIFAR-10 and ImageNet with less computation than human-designed rivals. They also scale down well to mobile budgets and supply features that improve COCO object detection. This shows that automatically discovered architectures can outperform expert designs at scale.

Key points

  • ImageNet: the best NASNet reaches 82.7% top-1 and 96.2% top-5 accuracy. That is 1.2% top-1 above the strongest human-designed model while using 9 billion fewer FLOPS, about 28% less compute.
  • CIFAR-10: NASNet achieves a 2.4% error rate. Searching for cells instead of full networks made the search roughly 7× faster.
  • Mobile-sized NASNet: it gets 74.0% top-1 on ImageNet, 3.1% better than comparably sized architectures built for mobile and embedded use.
  • COCO detection with Faster-RCNN: NASNet features reach 43.1% mAP, the best single-model result and over 4% above the previous best. The mobile variant (NASNet-A, 4 @ 1056) gets 29.6% mAP, more than 5% above earlier mobile Faster-RCNN systems.
  • Search setup: each cell has B = 5 blocks, built from 13 candidate operations (convolutions, separable convolutions, pooling, identity). Outputs are merged by addition or concatenation. With equal compute, RL found a best model over 1% more accurate on CIFAR-10 than the best one from random search.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

269 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 268 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1707.07012 · arXiv 1707.07012

@misc{zoph2017learning,
  title = {{Learning Transferable Architectures for Scalable Image Recognition}},
  author = {Zoph, Barret and Vasudevan, Vijay and Shlens, Jonathon and Le, Quoc V.},
  year = {2017},
  doi = {10.48550/arxiv.1707.07012},
  eprint = {1707.07012},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1707.07012}
}

References

71 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. Andrychowicz, M. Denil, S. Gomez, M. W. Hoffman, D. Pfau, T. Schaul, and N. de Freitas. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems, pages 3981–3989, 2016. find
  2. J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. find our card
  3. B. Baker, O. Gupta, N. Naik, and R. Raskar. Designing neural network architectures using reinforcement learning. In International Conference on Learning Representations, 2016. find
  4. J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl. Algorithms for hyper-parameter optimization. In Neural Information Processing Systems, 2011. find
  5. J. Bergstra and Y. Bengio. Random search for hyperparameter optimization. Journal of Machine Learning Research, 2012. find
  6. J. Bergstra, D. Yamins, and D. D. Cox. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. International Conference on Machine Learning, 2013. find
  7. J. Chen, R. Monga, S. Bengio, and R. Jozefowicz. Revisiting distributed synchronous sgd. In International Conference on Learning Representations Workshop Track, 2016. find
  8. Y. Chen, J. Li, H. Xiao, X. Jin, S. Yan, and J. Feng. Dual path networks. arXiv preprint arXiv:1707.01083, 2017. — [arXiv:1707.01083](https://arxiv.org/abs/1707.01083) find
  9. F. Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017. find
  10. D.-A. Clevert, T. Unterthiner, and S. Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). In International Conference on Learning Representations, 2016. find our card
  11. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2009. find
  12. T. DeVries and G. W. Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552, 2017. — [arXiv:1708.04552](https://arxiv.org/abs/1708.04552) find
  13. J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In International Conference on Machine Learning, volume 32, pages 647–655, 2014. find
  14. Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. Abbeel. RL 2 : Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779, 2016. find
  15. C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, 2017. find
  16. D. Floreano, P. Dürr, and C. Mattiussi. Neuroevolution: from architectures to learning. Evolutionary Intelligence, 2008. find
  17. K. Fukushima. A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, page 93202, 1980. find
  18. X. Gastaldi. Shake-shake regularization of 3-branch residual networks. In International Conference on Learning Representations Workshop Track, 2017. find
  19. D. Ha, A. Dai, and Q. V. Le. Hypernetworks. In International Conference on Learning Representations, 2017. find
  20. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2016. find
  21. K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision, 2016. find
  22. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 1997. find
  23. S. Hochreiter, A. Younger, and P. Conwell. Learning to learn using gradient descent. Artificial Neural Networks, pages 87–94, 2001. find
  24. A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017. find our card
  25. J. Hu, L. Shen, and G. Sun. Squeeze-and-excitation networks. arXiv preprint arXiv:1709.01507, 2017. — [arXiv:1709.01507](https://arxiv.org/abs/1709.01507) find
  26. G. Huang, Z. Liu, and K. Q. Weinberger. Densely connected convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition, 2017. find
  27. G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Weinberger. Deep networks with stochastic depth. In European Conference on Computer Vision, 2016. find
  28. J. Huang, V. Rathod, C. Sun, M. Zhu, A. Korattikara, A. Fathi, I. Fischer, Z. Wojna, Y. Song, S. Guadarrama, et al. Speed/accuracy trade-offs for modern convolutional object detectors. In IEEE Conference on Computer Vision and Pattern Recognition, 2017. find
  29. S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Learning Representations, 2015. find
  30. R. Jozefowicz, W. Zaremba, and I. Sutskever. An empirical exploration of recurrent network architectures. In International Conference on Learning Representations, 2015. find
  31. A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009. find
  32. A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing System, 2012. find
  33. G. Larsson, M. Maire, and G. Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. arXiv preprint arXiv:1605.07648, 2016. — [arXiv:1605.07648](https://arxiv.org/abs/1605.07648) find
  34. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998. find
  35. K. Li and J. Malik. Learning to optimize neural nets. arXiv preprint arXiv:1703.00441, 2017. — [arXiv:1703.00441](https://arxiv.org/abs/1703.00441) find
  36. T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017. find
  37. T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. arXiv preprint arXiv:1708.02002, 2017. find our card
  38. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pages 740–755. Springer, 2014. find our card
  39. I. Loshchilov and F. Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017. find
  40. H. Mendoza, A. Klein, M. Feurer, J. T. Springenberg, and F. Hutter. Towards automatically-tuned neural networks. In Proceedings of the 2016 Workshop on Automatic Machine Learning, pages 58–65, 2016. find
  41. T. Miconi. Neural networks with differentiable structure. arXiv preprint arXiv:1606.06216, 2016. — [arXiv:1606.06216](https://arxiv.org/abs/1606.06216) find
  42. R. Miikkulainen, J. Liang, E. Meyerson, A. Rawal, D. Fink, O. Francon, B. Raju, A. Navruzyan, N. Duffy, and B. Hodjat. Evolving deep neural networks. arXiv preprint arXiv:1703.00548, 2017. — [arXiv:1703.00548](https://arxiv.org/abs/1703.00548) find
  43. R. Negrinho and G. Gordon. DeepArchitect: Automatically designing and training deep architectures. arXiv preprint arXiv:1704.08792, 2017. — [arXiv:1704.08792](https://arxiv.org/abs/1704.08792) find
  44. N. Pinto, D. Doukhan, J. J. DiCarlo, and D. D. Cox. A high-throughput screening approach to discovering good forms of biologically inspired visual representation. PLoS Computational Biology, 5(11):e1000579, 2009. find
  45. S. Ravi and H. Larochelle. Optimization as a model for few-shot learning. In International Conference on Learning Representations, 2017. find
  46. E. Real, S. Moore, A. Selle, S. Saxena, Y. L. Suematsu, Q. Le, and A. Kurakin. Large-scale evolution of image classifiers. In International Conference on Machine Learning, 2017. find
  47. S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems, pages 91–99, 2015. find our card
  48. S. Saxena and J. Verbeek. Convolutional neural fabrics. In Advances in Neural Information Processing Systems, 2016. find
  49. T. Schaul and J. Schmidhuber. Metalearning. Scholarpedia, 2010. find
  50. F. Schroff, D. Kalenichenko, and J. Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 815–823, 2015. find our card
  51. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. find
  52. A. Shrivastava, R. Sukthankar, J. Malik, and A. Gupta. Beyond skip connections: Top-down modulation for object detection. arXiv preprint arXiv:1612.06851, 2016. — [arXiv:1612.06851](https://arxiv.org/abs/1612.06851) find
  53. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015. find our card
  54. J. Snoek, H. Larochelle, and R. P. Adams. Practical Bayesian optimization of machine learning algorithms. In Neural Information Processing Systems, 2012. find
  55. J. Snoek, O. Rippel, K. Swersky, R. Kiros, N. Satish, N. Sundaram, M. Patwary, M. Ali, R. P. Adams, et al. Scalable Bayesian optimization using deep neural networks. In International Conference on Machine Learning, 2015. find
  56. N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014. find
  57. K. O. Stanley, D. B. D’Ambrosio, and J. Gauci. A hypercube-based encoding for evolving large-scale neural networks. Artificial Life, 2009. find
  58. C. Szegedy, S. Ioffe, V. Vanhoucke, and A. Alemi. Inceptionv4, Inception-Resnet and the impact of residual connections on learning. In International Conference on Learning Representations Workshop Track, 2016. find
  59. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In IEEE Conference on Computer Vision and Pattern Recognition, 2015. find
  60. C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the Inception architecture for computer vision. In IEEE Conference on Computer Vision and Pattern Recognition, 2016. find
  61. D. Ulyanov, A. Vedaldi, and V. Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022, 2016. — [arXiv:1607.08022](https://arxiv.org/abs/1607.08022) find
  62. J. X. Wang, Z. Kurth-Nelson, D. Tirumala, H. Soyer, J. Z. Leibo, R. Munos, C. Blundell, D. Kumaran, and M. Botvinick. Learning to reinforcement learn. arXiv preprint arXiv:1611.05763, 2016. — [arXiv:1611.05763](https://arxiv.org/abs/1611.05763) find
  63. T. Weyand, I. Kostrikov, and J. Philbin. Planet-photo geolocation with convolutional neural networks. In European Conference on Computer Vision, 2016. find
  64. O. Wichrowska, N. Maheswaranathan, M. W. Hoffman, S. G. Colmenarejo, M. Denil, N. de Freitas, and J. Sohl-Dickstein. Learned optimizers that scale and generalize. arXiv preprint arXiv:1703.04813, 2017. — [arXiv:1703.04813](https://arxiv.org/abs/1703.04813) find
  65. D. Wierstra, F. J. Gomez, and J. Schmidhuber. Modeling systems with internal state using evolino. In The Genetic and Evolutionary Computation Conference, 2005. find
  66. R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. In Machine Learning, 1992. find
  67. L. Xie and A. Yuille. Genetic CNN. arXiv preprint arXiv:1703.01513, 2017. — [arXiv:1703.01513](https://arxiv.org/abs/1703.01513) find
  68. S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017. find
  69. X. Zhang, Z. Li, C. C. Loy, and D. Lin. Polynet: A pursuit of structural diversity in very deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017. find
  70. X. Zhang, X. Zhou, L. Mengxiao, and J. Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. arXiv preprint arXiv:1707.01083, 2017. — [arXiv:1707.01083](https://arxiv.org/abs/1707.01083) find
  71. B. Zoph and Q. V. Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations, 2017. find our card