Elamur Research⌕
☰
home / papers / 2018 / Efficient Neural Architecture Search via Parameter Sharing

Efficient Neural Architecture Search via Parameter Sharing

Paper · 2018 · Hieu Pham, Melody Y. Guan, Barret Zoph et al.

Summary

Neural Architecture Search (NAS) can find strong models, but it is very costly: each candidate network is trained to convergence, scored, and then its weights are thrown away. Earlier work used hundreds of GPUs for several days. ENAS cuts this cost by treating every candidate architecture as a subgraph of one large directed acyclic graph whose weights all candidates share. An LSTM controller picks subgraphs, meaning which connections to use and which operation or activation each node applies. Training alternates between two steps. First, the shared weights are updated with cross-entropy on training data. Second, the controller is updated with REINFORCE, using a reward measured on validation data. The authors test ENAS on language modelling (Penn Treebank) and image classification (CIFAR-10). On Penn Treebank the discovered recurrent cell reaches a test perplexity of 55.8, the best result among methods that skip post-training processing. On CIFAR-10 the error is 2.89%, close to NASNet's 2.65%. Each search ran on one GPU in under 16 hours, more than 1000x fewer GPU-hours than standard NAS. This makes automated model design practical on modest hardware.

Key points

  • Cost: a full search ran on a single Nvidia GTX 1080Ti in less than 16 hours. The NAS baseline it is compared against used 450 GPUs for 3–4 days, or about 32,400–43,200 GPU-hours.
  • Results:
    • Penn Treebank: test perplexity 55.8, versus 62.4 for the original NAS.
    • CIFAR-10: 2.89% test error, versus 2.65% for NASNet.
  • Recurrent search space: each of the N nodes chooses an earlier node as its input and one of four activations (tanh, ReLU, identity, sigmoid). That gives 4^N × N! cells, roughly 10^15 when N = 12. Nodes that feed no other node are averaged to form the cell output.
  • Controller and training details:
    • The controller is an LSTM with 100 hidden units that makes its choices one after another.
    • It is trained with Adam and REINFORCE, with a moving-average baseline.
    • The reward is validation accuracy for images and c / validation perplexity for text.
    • The shared weights get a gradient from a single sampled child model per step (M = 1).
  • Ablations:
    • Random architectures from the same spaces do much worse: perplexity 81.2 on Penn Treebank, and 5.86% (whole network) or 6.77% (cells) error on CIFAR-10. ENAS reaches 4.23% and 3.54% in the same comparison.
    • Training the shared weights without updating the controller gives 8.92% error. An ensemble of 250 such sampled configurations still only reaches 5.49%.

Interesting ideas

  • I1. One supergraph for all candidates. If every option you want to compare can be written as a path through a single shared graph, you train shared weights once instead of training every option from scratch. This could cheaply compare encoder variants or feature pipelines for NLP tasks with limited compute.
  • I2. Reward on held-out data, not training loss. The controller is pushed toward models that generalize, not ones that fit training data well. Any automated tuning loop, such as hyperparameters or data-mixing choices, can use the same separation.
  • I3. Cheap final selection. Instead of fully retraining many sampled candidates, ENAS scores them on one validation minibatch with the shared weights and retrains only the best one. This is a useful budget trick whenever full evaluation is the expensive step.
  • I4. Ablations against the search space itself. The authors separate the value of the search method from the quality of the search space. They compare against random sampling from the same space and against an untrained controller. This is a good habit for any paper claiming a clever search or selection procedure.
  • I5. Watch for hidden constraints in weight-generation shortcuts. The authors argue that SMASH's hypernetwork limits child weights to a low-rank space, so it may favour architectures that only look good under that limit. Keeping shared weights unconstrained avoids this, which is worth checking in any proxy-evaluation setup.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States506 works · 330,053 citations🎓Carnegie Mellon UniversityPittsburgh · United States598 works · 191,226 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le and Jeff Dean. 2018. Efficient Neural Architecture Search via Parameter Sharing. arXiv:1802.03268.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1802.03268 · arXiv 1802.03268

@misc{pham2018efficient,
  title = {{Efficient Neural Architecture Search via Parameter Sharing}},
  author = {Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le and Jeff Dean},
  year = {2018},
  doi = {10.48550/arxiv.1802.03268},
  eprint = {1802.03268},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1802.03268}
}

References

49 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Baker, Bowen, Gupta, Otkrist, Naik, Nikhil, and Raskar, Ramesh. Designing neural network architectures using reinforcement learning. In ICLR, 2017a. find
  2. Baker, Bowen, Otkrist, Gupta, Raskar, Ramesh, and Naik, Nikhil. Accelerating neural architecture search using performance prediction. Arxiv, 1705.10823, 2017b. find
  3. Bello, Irwan, Pham, Hieu, Le, Quoc V., Norouzi, Mohammad, and Bengio, Samy. Neural combinatorial optimization with reinforcement learning. In ICLR Workshop, 2017a. find
  4. Bello, Irwan, Zoph, Barret, Vasudevan, Vijay, and Le, Quoc V. Neural optimizer search with reinforcement learning. In ICML, 2017b. find
  5. Brock, Andrew, Lim, Theodore, Ritchie, James M., and Weston, Nick. SMASH: one-shot model architecture search through hypernetworks. ICLR, 2018. find
  6. Cai, Han, Chen, Tianyao, Zhang, Weinan, Yu, Yong., and Wang, Jun. Efficient architecture search by network transformation. In AAAI, 2018.
  7. Chollet, Francois. Xception: Deep learning with depthwise separable convolutions. In CVPR, 2017. find
  8. Collins, Jasmine, Sohl-Dickstein, Jascha, and Sussillo, David. Capacity and trainability in recurrent neural networks. In ICLR, 2017. find
  9. Deng, Boyang, Yan, Junjie, and Lin, Dahua. Peephole: Predicting network performance before training. Arxiv, 1705.10823, 2017. find
  10. DeVries, Terrance and Taylor, Graham W. Improved regularization of convolutional neural networks with cutout. Arxiv, 1708.04552, 2017. find
  11. Gal, Yarin and Ghahramani, Zoubin. A theoretically grounded application of dropout in recurrent neural networks. In NIPS, 2016.
  12. Gastaldi, Xavier. Shake-shake regularization of 3-branch residual networks. In ICLR Workshop Track, 2016. find
  13. Grave, Edouard, Joulin, Armand, and Usunier, Nicolas. Improving neural language models with a continuous cache. In ICLR, 2017.
  14. Ha, David, Dai, Andrew, and Le, Quoc V. Hypernetworks. In ICLR, 2017. find
  15. He, Kaiming, Zhang, Xiangyu, Rein, Shaoqing, and Sun, Jian. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In CVPR, 2015. find
  16. He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian. Deep residual learning for image recognition. In CPVR, 2016a.
  17. He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian. Identity mappings in deep residual networks. In CPVR, 2016b. find
  18. Hochreiter, Sepp and Schmidhuber, Jürgen. Long short-term memory. In Neural Computations, 1997. find
  19. Huang, Gao, Liu, Zhuang, van der Maaten, Laurens, and Weinberger, Kilian Q. Densely connected convolutional networks. In CVPR, 2016. find
  20. Inan, Hakan, Khosravi, Khashayar, and Socher, Richard. Tying word vectors and word classifiers: a loss framework for language modeling. In ICLR, 2017.
  21. Ioffe, Sergey and Szegedy, Christian. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015. find
  22. Kingma, Diederik P. and Ba, Jimmy Lei. Adam: A method for stochastic optimization. In ICLR, 2015.
  23. Krause, Ben, Kahembwe, Emmanuel, Murray, Iain, and Renals, Steve. Dynamic evaluation of neural sequence models. Arxiv, 1709.07432, 2017. find
  24. Krizhevsky, Alex. Learning multiple layers of features from tiny images. Technical report, 2009. find
  25. Larsson, Gustav, Maire, Michael, and Shakhnarovich, Gregory. Fractalnet: Ultra-deep neural networks without residuals. In ICLR, 2017. find
  26. Lin, Min, Chen, Qiang, and Yan, Shuicheng. Network in network. Arxiv, 1312.4400, 2013. find
  27. Liu, Chenxi, Zoph, Barret, Shlens, Jonathon, Hua, Wei, Li, Li-Jia, Fei-Fei, Li, Yuille, Alan, Huang, Jonathan, and Murphy, Kevin. Progressive neural architecture search. Arxiv, 1712.00559, 2017. find
  28. Liu, Hanxiao, Simonyan, Karen, Vinyals, Oriol, Fernando, Chrisantha, and Kavukcuoglu, Koray. Hierarchical representations for efficient architecture search. In ICLR, 2018. find
  29. Loshchilov, Ilya and Hutter, Frank. Sgdr: Stochastic gradient descent with warm restarts. In ICLR, 2017. find
  30. Luong, Minh-Thang, Le, Quoc V., Sutskever, Ilya, Vinyals, Oriol, and Kaiser, Lukasz. Multi-task sequence to sequence learning. In ICLR, 2016.
  31. Marcus, Mitchell, Kim, Grace, Marcinkiewicz, Mary Ann, MacIntyre, Robert, Bies, Ann, Ferguson, Mark, Katz, Karen, and Schasberger, Britta. The penn treebank: Annotating predicate argument structure. In Proceedings of the Workshop on Human Language Technology, 1994.
  32. Melis, Gábor, Dyer, Chris, and Blunsom, Phil. On the state of the art of evaluation in neural language models. Arxiv, 1707.05589, 2017. find
  33. Merity, Stephen, Keskar, Nitish Shirish, and Socher, Richard. Regularizing and optimizing LSTM language models. Arxiv, 1708.02182, 2017.
  34. Negrinho, Renato and Gordon, Geoff. Deeparchitect: Automatically designing and training deep architectures. In CPVR, 2017. find
  35. Nesterov, Yurii E. A method for solving the convex programming problem with convergence rate o(1/k 2 ). Soviet Mathematics Doklady, 1983. find
  36. Razavian, Ali Sharif, Azizpour, Hossein, Josephine, Sullivan, and Carlsson, Stefan. Cnn features off-the-shelf: an astounding baseline for recognition. In CVPR, 2014. find
  37. Real, Esteban, Moore, Sherry, Selle, Andrew, Saxena, Saurabh, Leon, Yutaka Suematsu, Tan, Jie, Le, Quoc, and Kurakin, Alex. Large-scale evolution of image classifiers. In ICML, 2017. find
  38. Real, Esteban, Aggarwal, Alok, Huang, Yanping, and Le, Quoc V. Peephole: Predicting network performance before training. Arxiv, 1802.01548, 2018. find
  39. Saxena, Shreyas and Verbeek, Jakob. Convolutional neural fabrics. In NIPS, 2016. find
  40. Schulman, John, Heess, Nicolas, Weber, Theophane, and Abbeel, Pieter. Gradient estimation using stochastic computation graphs. In NIPS, 2015. find
  41. Veniat, Tom and Denoyer, Ludovic. Learning time-efficient deep architectures with budgeted super networks. Arxiv, 1706.00046, 2017. find
  42. Williams, Ronald J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 1992. find
  43. Yang, Zhilin, Dai, Zihang, Salakhutdinov, Ruslan, and Cohen, William. Breaking the softmax bottleneck: A high-rank rnn language model. In ICLR, 2018. find
  44. Zaremba, Wojciech, Sutskever, Ilya, and Vinyals, Oriol. Recurrent neural network regularization. Arxiv, 1409.2329, 2014.
  45. Zhong, Zhao, Yan, Junjie, and Liu, Cheng-Lin. Practical network blocks design with q-learning. AAAI, 2018. find
  46. Zilly, Julian Georg, Srivastava, Rupesh Kumar, Koutnı́k, Jan, and Schmidhuber, Jürgen. Recurrent highway networks. In ICML, 2017.
  47. Zoph, Barret and Le, Quoc V. Neural architecture search with reinforcement learning. In ICLR, 2017.
  48. Zoph, Barret, Yuret, Deniz, May, Jonathan, and Knight, Kevin. Transfer learning for low-resource neural machine translation. In EMNLP, 2016. find
  49. Zoph, Barret, Vasudevan, Vijay, Shlens, Jonathon, and Le, Quoc V. Learning transferable architectures for scalable image recognition. In CVPR, 2018.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pham-2018-efficient-neural-architecture-search-parameter/