Elamur Research
☰
home / papers / 2019 / EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1905.11946

Summary

Convolutional networks are usually enlarged along a single axis, such as more layers, more channels or bigger input images. Doing this by hand tends to waste compute and leave accuracy on the table. Tan and Le study how depth, width and resolution interact. They conclude that the three should grow together at fixed relative rates.

Their compound scaling rule ties all three to one coefficient. Depth, width and image size are multiplied by constants α, β and γ raised to the same power. These constants come from a small grid search on the base model. The rule also improves scaled MobileNets and ResNet.

For a stronger starting point, the authors used neural architecture search to build a compact baseline, EfficientNet-B0. They then scaled it into a family running from B1 to B7. On ImageNet, these models match or beat much larger ConvNets with far fewer parameters and FLOPS. They also transfer well to other image classification datasets. The main contribution is a simple, principled recipe for enlarging models under a compute budget.

Key points

  • EfficientNet-B7 reaches 84.3% ImageNet top-1, equal to GPipe. It uses about 66M parameters instead of 556M, making it 8.4x smaller and 6.1x faster at inference.
  • EfficientNet-B4 lifts top-1 accuracy from 76.3% (ResNet-50) to 83.0% at a similar FLOPS cost. EfficientNet-B1 is 7.6x smaller and 5.7x faster than ResNet-152.
  • Ablation on EfficientNet-B0: the authors scaled B0 to roughly 1.8B FLOPS in different ways. Compound scaling (d=1.4, w=1.2, r=1.3) gave 81.1%, while depth-only, width-only or resolution-only scaling gave about 79%. That is a gain of up to 2.5%.
  • Transfer learning: the models set state-of-the-art results on 5 of 8 datasets, including CIFAR-100 (91.7%) and Flowers (98.8%). They need up to 21x fewer parameters than earlier ConvNets.
  • Code: source is released in the tensorflow/tpu GitHub repository under models/official/efficientnet.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Cited by

437 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 428 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1905.11946 · arXiv 1905.11946

@misc{tan2019efficientnet,
  title = {{EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks}},
  author = {Tan, Mingxing and Le, Quoc V.},
  year = {2019},
  doi = {10.48550/arxiv.1905.11946},
  eprint = {1905.11946},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1905.11946}
}

References

52 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Berg, T., Liu, J., Woo Lee, S., Alexander, M. L., Jacobs, D. W., and Belhumeur, P. N. Birdsnap: Large-scale fine-grained visual categorization of birds. CVPR, pp. 2011–2018, 2014. find
  2. Bossard, L., Guillaumin, M., and Van Gool, L. Food-101– mining discriminative components with random forests. ECCV, pp. 446–461, 2014. find
  3. Cai, H., Zhu, L., and Han, S. Proxylessnas: Direct neural architecture search on target task and hardware. ICLR, 2019. find
  4. Chollet, F. Xception: Deep learning with depthwise separable convolutions. CVPR, pp. 1610–02357, 2017. find
  5. Cubuk, E. D., Zoph, B., Mane, D., Vasudevan, V., and Le, Q. V. Autoaugment: Learning augmentation policies from data. CVPR, 2019. find
  6. Elfwing, S., Uchibe, E., and Doya, K. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Networks, 107:3–11, 2018. find
  7. Gholami, A., Kwon, K., Wu, B., Tai, Z., Yue, X., Jin, P., Zhao, S., and Keutzer, K. Squeezenext: Hardware-aware neural network design. ECV Workshop at CVPR’18, 2018. find
  8. Han, S., Mao, H., and Dally, W. J. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. ICLR, 2016. find
  9. He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. CVPR, pp. 770–778, 2016. find
  10. He, K., Gkioxari, G., Dollár, P., and Girshick, R. Mask r-cnn. ICCV, pp. 2980–2988, 2017. find
  11. He, Y., Lin, J., Liu, Z., Wang, H., Li, L.-J., and Han, S. Amc: Automl for model compression and acceleration on mobile devices. ECCV, 2018. find
  12. Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016. find
  13. Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017. find our card
  14. Hu, J., Shen, L., and Sun, G. Squeeze-and-excitation networks. CVPR, 2018. find
  15. Huang, G., Sun, Y., Liu, Z., Sedra, D., and Weinberger, K. Q. Deep networks with stochastic depth. ECCV, pp. 646–661, 2016. find
  16. Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. Densely connected convolutional networks. CVPR, 2017. find
  17. Huang, Y., Cheng, Y., Chen, D., Lee, H., Ngiam, J., Le, Q. V., and Chen, Z. Gpipe: Efficient training of giant neural networks using pipeline parallelism. arXiv preprint arXiv:1808.07233, 2018. find
  18. Iandola, F. N., Han, S., Moskewicz, M. W., Ashraf, K., Dally, W. J., and Keutzer, K. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and \<0.5 mb model size. arXiv preprint arXiv:1602.07360, 2016. — [arXiv:1602.07360](https://arxiv.org/abs/1602.07360) find
  19. Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. ICML, pp. 448–456, 2015. find
  20. Kornblith, S., Shlens, J., and Le, Q. V. Do better imagenet models transfer better? CVPR, 2019. find
  21. Krause, J., Deng, J., Stark, M., and Fei-Fei, L. Collecting a large-scale dataset of fine-grained cars. Second Workshop on Fine-Grained Visual Categorizatio, 2013. find
  22. Krizhevsky, A. and Hinton, G. Learning multiple layers of features from tiny images. Technical Report, 2009. find
  23. Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. In NIPS, pp. 1097–1105, 2012. find
  24. Lin, H. and Jegelka, S. Resnet with one-neuron hidden layers is a universal approximator. NeurIPS, pp. 6172– 6181, 2018. find
  25. Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B., and Belongie, S. Feature pyramid networks for object detection. CVPR, 2017. find
  26. Liu, C., Zoph, B., Shlens, J., Hua, W., Li, L.-J., Fei-Fei, L., Yuille, A., Huang, J., and Murphy, K. Progressive neural architecture search. ECCV, 2018. find
  27. Lu, Z., Pu, H., Wang, F., Hu, Z., and Wang, L. The expressive power of neural networks: A view from the width. NeurIPS, 2018. find
  28. Ma, N., Zhang, X., Zheng, H.-T., and Sun, J. Shufflenet v2: Practical guidelines for efficient cnn architecture design. ECCV, 2018. find
  29. Mahajan, D., Girshick, R., Ramanathan, V., He, K., Paluri, M., Li, Y., Bharambe, A., and van der Maaten, L. Exploring the limits of weakly supervised pretraining. arXiv preprint arXiv:1805.00932, 2018. — [arXiv:1805.00932](https://arxiv.org/abs/1805.00932) find
  30. Maji, S., Rahtu, E., Kannala, J., Blaschko, M., and Vedaldi, A. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013. — [arXiv:1306.5151](https://arxiv.org/abs/1306.5151) find
  31. Ngiam, J., Peng, D., Vasudevan, V., Kornblith, S., Le, Q. V., and Pang, R. Domain adaptive transfer learning with specialist models. arXiv preprint arXiv:1811.07056, 2018. — [arXiv:1811.07056](https://arxiv.org/abs/1811.07056) find
  32. Nilsback, M.-E. and Zisserman, A. Automated flower classification over a large number of classes. ICVGIP, pp. 722–729, 2008. find
  33. Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. Cats and dogs. CVPR, pp. 3498–3505, 2012. find
  34. Raghu, M., Poole, B., Kleinberg, J., Ganguli, S., and Sohl-Dickstein, J. On the expressive power of deep neural networks. ICML, 2017. find
  35. Ramachandran, P., Zoph, B., and Le, Q. V. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2018. find
  36. Real, E., Aggarwal, A., Huang, Y., and Le, Q. V. Regularized evolution for image classifier architecture search. AAAI, 2019. find
  37. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3): 211–252, 2015. find our card
  38. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. Mobilenetv2: Inverted residuals and linear bottlenecks. CVPR, 2018. find
  39. Sharir, O. and Shashua, A. On the expressive power of overlapping architectures of deep learning. ICLR, 2018. find
  40. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958, 2014. find
  41. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. Going deeper with convolutions. CVPR, pp. 1–9, 2015. find
  42. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vision. CVPR, pp. 2818–2826, 2016. find
  43. Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A. A. Inception-v4, inception-resnet and the impact of residual connections on learning. AAAI, 4:12, 2017. find
  44. Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M., Howard, A., and Le, Q. V. MnasNet: Platform-aware neural architecture search for mobile. CVPR, 2019. find
  45. Xie, S., Girshick, R., Dollár, P., Tu, Z., and He, K. Aggregated residual transformations for deep neural networks. CVPR, pp. 5987–5995, 2017. find
  46. Yang, T.-J., Howard, A., Chen, B., Zhang, X., Go, A., Sze, V., and Adam, H. Netadapt: Platform-aware neural network adaptation for mobile applications. ECCV, 2018. find
  47. Zagoruyko, S. and Komodakis, N. Wide residual networks. BMVC, 2016. find
  48. Zhang, X., Li, Z., Loy, C. C., and Lin, D. Polynet: A pursuit of structural diversity in very deep networks. CVPR, pp. 3900–3908, 2017. find
  49. Zhang, X., Zhou, X., Lin, M., and Sun, J. Shufflenet: An extremely efficient convolutional neural network for mobile devices. CVPR, 2018. find
  50. Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., and Torralba, A. Learning deep features for discriminative localization. CVPR, pp. 2921–2929, 2016. find
  51. Zoph, B. and Le, Q. V. Neural architecture search with reinforcement learning. ICLR, 2017. find our card
  52. Zoph, B., Vasudevan, V., Shlens, J., and Le, Q. V. Learning transferable architectures for scalable image recognition. CVPR, 2018. find our card