Elamur Research
☰
home / papers / 2016 / Neural Architecture Search with Reinforcement Learning

Neural Architecture Search with Reinforcement Learning

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1611.01578

Summary

Designing neural network architectures still depends heavily on expert intuition and lengthy trial and error. Zoph and Le treat this design process as a learning problem. An RNN "controller" outputs a network specification token by token, for example filter sizes, strides and filter counts for each convolutional layer. The resulting "child network" is then trained, and its validation accuracy serves as a reward. Because that reward cannot be differentiated, the controller is updated with the REINFORCE policy-gradient rule, so over time it favors architectures that score well. The search space is extended with skip connections and branching, and a second variant composes recurrent cells. Training is spread across many machines with asynchronous updates. Starting from scratch, the method found a CIFAR-10 ConvNet competitive with the best human-designed models and a Penn Treebank recurrent cell that beat LSTM and prior state-of-the-art baselines. The work showed that automated, learned search can match or exceed manual architecture engineering, and it helped launch the field of neural architecture search.

Key points

  • CIFAR-10: the discovered ConvNet reaches 3.65% test error. This is 0.09 points better than the previous best model with a comparable design scheme, and 1.05x faster.
  • Penn Treebank (word level): the generated recurrent cell achieves perplexity 62.4, an improvement of 3.6 over the earlier state of the art. It also outperforms the standard LSTM cell.
  • Transfer: the same cell, reused for character-level language modeling on PTB, sets a new best result of 1.214.
  • Training method: the controller is trained with REINFORCE (Williams, 1992). Variance is reduced with a baseline equal to an exponential moving average of past accuracies. A parameter-server setup lets K controller replicas each train m child models in parallel.
  • Skip connections: these are predicted through set-selection attention over "anchor points." Each layer uses sigmoid scores to decide which earlier layers feed into it, and the inputs it selects are concatenated along the depth dimension.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

357 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 350 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1611.01578 · arXiv 1611.01578

@misc{zoph2016neural,
  title = {{Neural Architecture Search with Reinforcement Learning}},
  author = {Zoph, Barret and Le, Quoc V.},
  year = {2016},
  doi = {10.48550/arxiv.1611.01578},
  eprint = {1611.01578},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1611.01578}
}

References

66 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Learning to compose neural networks for question answering. In NAACL, 2016. find our card
  2. Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, and Nando de Freitas. Learning to learn by gradient descent by gradient descent. arXiv preprint arXiv:1606.04474, 2016. find
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015. find our card
  4. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. JMLR, 2003. find
  5. James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. JMLR, 2012. find
  6. James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-parameter optimization. In NIPS, 2011. find
  7. James Bergstra, Daniel Yamins, and David D Cox. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. ICML, 2013. find
  8. Alan W. Biermann. The inference of regular LISP programs from examples. IEEE transactions on Systems, Man, and Cybernetics, 1978. find
  9. Wei-Chen Cheng, Stanley Kok, Hoai Vu Pham, Hai Leong Chieu, and Kian Ming Adam Chai. Language modeling with sum-product networks. In INTERSPEECH, 2014. find
  10. Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In CVPR, 2005. find
  11. Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Andrew Senior, Paul Tucker, Ke Yang, Quoc V. Le, et al. Large scale distributed deep networks. In NIPS, 2012. find
  12. Dario Floreano, Peter Dürr, and Claudio Mattiussi. Neuroevolution: from architectures to learning. Evolutionary Intelligence, 2008. find
  13. Yarin Gal. A theoretically grounded application of dropout in recurrent neural networks. arXiv preprint arXiv:1512.05287, 2015. find our card
  14. David Ha, Andrew Dai, and Quoc V. Le. Hypernetworks. arXiv preprint arXiv:1609.09106, 2016. — [arXiv:1609.09106](https://arxiv.org/abs/1609.09106) find
  15. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016a. find
  16. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. arXiv preprint arXiv:1603.05027, 2016b. find
  17. Geoffrey Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazi… find
  18. Sepp Hochreiter and Juergen Schmidhuber. Long short-term memory. Neural Computation, 1997. find
  19. Gao Huang, Zhuang Liu, and Kilian Q. Weinberger. Densely connected convolutional networks. arXiv preprint arXiv:1608.06993, 2016a. — [arXiv:1608.06993](https://arxiv.org/abs/1608.06993) find
  20. Gao Huang, Zhuang Liu, Kilian Q. Weinberger, and Laurens van der Maaten. Densely connected convolutional networks. arXiv preprint arXiv:1608.06993, 2016b. — [arXiv:1608.06993](https://arxiv.org/abs/1608.06993) find
  21. Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger. Deep networks with stochastic depth. arXiv preprint arXiv:1603.09382, 2016c. find
  22. Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying word vectors and word classifiers: A loss framework for language modeling. arXiv preprint arXiv:1611.01462, 2016. find
  23. Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015. find
  24. Kevin Jarrett, Koray Kavukcuoglu, Yann Lecun, et al. What is the best multi-stage architecture for object recognition? In ICCV, 2009. find
  25. Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever. An empirical exploration of recurrent network architectures. In ICML, 2015. find
  26. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M. Rush. Character-aware neural language models. arXiv preprint arXiv:1508.06615, 2015. find our card
  27. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. find
  28. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012. find
  29. Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 2015. find
  30. Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. arXiv preprint arXiv:1605.07648, 2016. — [arXiv:1605.07648](https://arxiv.org/abs/1605.07648) find
  31. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998. find
  32. Chen-Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu. supervised nets. In AISTATS, 2015. find
  33. Ke Li and Jitendra Malik. Learning to optimize. arXiv preprint arXiv:1606.01885, 2016. — [arXiv:1606.01885](https://arxiv.org/abs/1606.01885) find
  34. Percy Liang, Michael I. Jordan, and Dan Klein. Learning programs: A hierarchical Bayesian approach. In ICML, 2010. find
  35. Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. In ICLR, 2013. find
  36. David G. Lowe. Object recognition from local scale-invariant features. In CVPR, 1999. find
  37. Hector Mendoza, Aaron Klein, Matthias Feurer, Jost Tobias Springenberg, and Frank Hutter. Towards automatically-tuned neural networks. In Proceedings of the 2016 Workshop on Automatic Machine Learning, pp. 58–65, 2016. find
  38. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016. find our card
  39. Tomas Mikolov and Geoffrey Zweig. Context dependent recurrent neural network language model. In SLT, pp. 234–239, 2012. find
  40. Deeply-Andriy Mnih and Geoffrey Hinton. Three new graphical models for statistical language modelling. In ICML, 2007. find
  41. Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted Boltzmann machines. In ICML, 2010. find
  42. Arvind Neelakantan, Quoc V. Le, and Ilya Sutskever. Neural programmer: Inducing latent programs with gradient descent. In ICLR, 2015. find
  43. Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. How to construct deep recurrent neural networks. arXiv preprint arXiv:1312.6026, 2013. find
  44. Ofir Press and Lior Wolf. Using the output embedding to improve language models. arXiv preprint arXiv:1608.05859, 2016. find our card
  45. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. arXiv preprint arXiv:1511.06732, 2015. find our card
  46. Scott Reed and Nando de Freitas. Neural programmer-interpreters. In ICLR, 2015. find
  47. Shreyas Saxena and Jakob Verbeek. Convolutional neural fabrics. In NIPS, 2016. find
  48. Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. Minimum risk training for neural machine translation. In ACL, 2016. find our card
  49. Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. find our card
  50. Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical Bayesian optimization of machine learning algorithms. In NIPS, 2012. find
  51. Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mostofa Ali, Ryan P. Adams, et al. Scalable bayesian optimization using deep neural networks. In ICML, 2015. find
  52. Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806, 2014. — [arXiv:1412.6806](https://arxiv.org/abs/1412.6806) find
  53. Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint arXiv:1505.00387, 2015. find our card
  54. Kenneth O. Stanley, David B. D’Ambrosio, and Jason Gauci. A hypercube-based encoding for evolving large-scale neural networks. Artificial Life, 2009. find
  55. Phillip D. Summers. A methodology for LISP program construction from examples. Journal of the ACM, 1977. find
  56. Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In ICML, 2013. find
  57. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. In NIPS, 2014. find
  58. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015. find
  59. Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 2012. find
  60. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In NIPS, 2015. find
  61. Daan Wierstra, Faustino J Gomez, and Jürgen Schmidhuber. Modeling systems with internal state using evolino. In GECCO, 2005. find
  62. Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. In Machine Learning, 1992. find
  63. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, et al. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016. find
  64. Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016. find
  65. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization. arXiv preprint arXiv:1409.2329, 2014. find our card
  66. Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutnı́k, and Jürgen Schmidhuber. Recurrent highway networks. arXiv preprint arXiv:1607.03474, 2016. find