Elamur Research⌕
☰
home / papers / 2016 / RL²: Fast Reinforcement Learning via Slow Reinforcement Learning

RL²: Fast Reinforcement Learning via Slow Reinforcement Learning

Paper · 2016 · Yan Duan, John Schulman, Xi Chen et al.

Summary

Deep RL agents usually need enormous amounts of experience: mastering one Atari game can take the equivalent of about 40 days of nonstop play, while a human needs roughly two hours. The authors attribute this gap to the lack of prior knowledge. Instead of hand-crafting a sample-efficient algorithm, RL$^2$ treats the learning procedure itself as something to be learned. A recurrent policy built from GRUs is trained with a conventional, slow RL optimizer (first-order TRPO) across many MDPs drawn from a task distribution. At each step the policy sees the observation, previous action, reward and termination flag. Its hidden state carries over between episodes of the same MDP and is reset only when a new MDP is sampled. The recurrent activations therefore act as the memory of a fast learner that adapts within a trial. On random multi-armed bandits and tabular MDPs, the trained network performs close to theoretically grounded algorithms. On a ViZDoom visual navigation task, the approach scales to high-dimensional inputs. This work is an early, influential example of meta-reinforcement learning.

Key points

  • Training objective: the expected discounted reward summed over an entire trial of n episodes, rather than over one episode. The authors note this is equivalent to minimizing cumulative pseudo-regret.
  • Bandit evaluation: Bernoulli arms with parameters sampled uniformly from [0, 1]. Baselines include Random, approximate Gittins index and UCB1.
  • Optimization setup: first-order TRPO with a GRU-based baseline and optional GAE. All experiments use 256 GRU units and discount 0.99. Batch size is 250,000 for bandits and tabular MDPs and 50,000 for navigation.
  • Visual navigation network: ViZDoom frames are downscaled to 40×30 RGB and passed through two conv layers (16 filters, 5×5, stride 2). In this task the policy and baseline share one network.
  • Implementation: built with TensorFlow and rllab. Reference algorithms come from the TabulaRL package.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢OpenAISan Francisco · United States18 works · 27,146 citations

Also on the byline: Berkeley College.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksDecision Sciences · 209 worksManagement Science and Operations Research · 108 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Machine learning · 51 works

Cite

Cite as Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever and Pieter Abbeel. 2016. RL²: Fast Reinforcement Learning via Slow Reinforcement Learning. arXiv:1611.02779.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1611.02779 · arXiv 1611.02779

@misc{duan2016fast,
  title = {{RL²: Fast Reinforcement Learning via Slow Reinforcement Learning}},
  author = {Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever and Pieter Abbeel},
  year = {2016},
  doi = {10.48550/arxiv.1611.02779},
  eprint = {1611.02779},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1611.02779}
}

References

58 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, and Nando de Freitas. Learning to learn by gradient descent by gradient descent. arXiv preprint arXiv:1606.04474, 2016. find
  2. Jean-Yves Audibert and Rémi Munos. Introduction to bandits: Algorithms and theory. ICML Tutorial on bandits, 2011. find
  3. Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research, 3(Nov):397–422, 2002. find
  4. Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157–166, 1994. find
  5. Sébastien Bubeck and Nicolo Cesa-Bianchi. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. arXiv preprint arXiv:1204.5721, 2012. — [arXiv:1204.5721](https://arxiv.org/abs/1204.5721) find
  6. Jhelum Chakravorty and Aditya Mahajan. Multi-armed bandits, gittins index, and its calculation. Methods and Applications of Statistics in Clinical Trials: Planning, Analysis, and Inferential Methods, 2:416–435, 2013. find
  7. Olivier Chapelle and Lihong Li. An empirical evaluation of thompson sampling. In Advances in neural information processing systems, pp. 2249–2257, 2011. find
  8. Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259, 2014.
  9. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014. find
  10. Marc Deisenroth and Carl E Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICML-11), pp. 465–472, 2011. find
  11. Coline Devin, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, and Sergey Levine. Learning modular neural network policies for multi-task and multi-robot transfer. arXiv preprint arXiv:1609.07088, 2016. find
  12. Li Fei-Fei, Rob Fergus, and Pietro Perona. One-shot learning of object categories. IEEE transactions on pattern analysis and machine intelligence, 28(4):594–611, 2006. find
  13. AA Feldbaum. Dual control theory. i. Avtomatika i Telemekhanika, 21(9):1240–1249, 1960. find
  14. Justin Fu, Sergey Levine, and Pieter Abbeel. One-shot learning of manipulation skills with online dynamics adaptation and neural network priors. arXiv preprint arXiv:1509.06841, 2015. — [arXiv:1509.06841](https://arxiv.org/abs/1509.06841) find
  15. Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, Aviv Tamar, et al. Bayesian reinforcement learning: a survey. World Scientific, 2015. find
  16. John Gittins, Kevin Glazebrook, and Richard Weber. Multi-armed bandit allocation indices. John Wiley & Sons, 2011. find
  17. John C Gittins. Bandit processes and dynamic allocation indices. Journal of the Royal Statistical Society. Series B (Methodological), pp. 148–177, 1979. find
  18. Xiaoxiao Guo, Satinder Singh, Honglak Lee, Richard L Lewis, and Xiaoshi Wang. Deep learning for real-time atari game play using offline monte-carlo tree search planning. In Advances in neural information processing systems, pp. 3338–3346, 2014. find
  19. Nicolas Heess, Gregory Wayne, David Silver, Tim Lillicrap, Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems, pp. 2944–2952, 2015. find
  20. Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to learn using gradient descent. In International Conference on Artificial Neural Networks, pp. 87–94. Springer, 2001. find
  21. Shin Ishii, Wako Yoshida, and Junichiro Yoshimoto. Control of exploitation–exploration meta-parameter in reinforcement learning. Neural networks, 15(4):665–687, 2002. find
  22. Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research, 11(Apr):1563–1600, 2010. find
  23. Rafal Józefowicz, Wojciech Zaremba, and Ilya Sutskever. An empirical exploration of recurrent network architectures. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, pp. 2342–2350, 2015. URL http://jmlr.org/proceedings/papers/v37/jo… ↗
  24. Michał Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaśkowski. Vizdoom: A doom-based ai research platform for visual reinforcement learning. arXiv preprint arXiv:1605.02097, 2016. — [arXiv:1605.02097](https://arxiv.org/abs/1605.02097) find
  25. Gregory Koch. Siamese neural networks for one-shot image recognition. PhD thesis, University of Toronto, 2015. find
  26. J Zico Kolter and Andrew Y Ng. Near-bayesian exploration in polynomial time. In Proceedings of the 26th Annual International Conference on Machine Learning, pp. 513–520. ACM, 2009. find
  27. Brenden M Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua B Tenenbaum. One shot learning of simple visual concepts. In Proceedings of the 33rd Annual Conference of the Cognitive Science Society, volume 172, pp. 2, 2011. find
  28. Hugo Larochelle, Dumitru Erhan, and Yoshua Bengio. Zero-data learning of new tasks. In AAAI, volume 1, pp. 3, 2008. find
  29. Sergey Levine and Pieter Abbeel. Learning neural network policies with guided policy search under unknown dynamics. In Advances in Neural Information Processing Systems, pp. 1071–1079, 2014. find
  30. Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39):1–40, 2016. find
  31. Ke Li and Jitendra Malik. Learning to optimize. arXiv preprint arXiv:1606.01885, 2016. — [arXiv:1606.01885](https://arxiv.org/abs/1606.01885) find
  32. Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015. find
  33. Benedict C May, Nathan Korda, Anthony Lee, and David S Leslie. Optimistic bayesian sampling in contextual-bandit problems. Journal of Machine Learning Research, 13(Jun):2069–2106, 2012. find
  34. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015. find
  35. Junhyuk Oh, Valliappa Chockalingam, Satinder Singh, and Honglak Lee. Control of memory, active perception, and action in minecraft. arXiv preprint arXiv:1605.09128, 2016. — [arXiv:1605.09128](https://arxiv.org/abs/1605.09128) find
  36. Sylvie CW Ong, Shao Wei Png, David Hsu, and Wee Sun Lee. Planning under uncertainty for robotic tasks with mixed observability. The International Journal of Robotics Research, 29(8): 1053–1068, 2010. find
  37. Ian Osband and Benjamin Van Roy. Why is posterior sampling better than optimism for reinforcement learning. arXiv preprint arXiv:1607.00215, 2016. — [arXiv:1607.00215](https://arxiv.org/abs/1607.00215) find
  38. Ian Osband, Dan Russo, and Benjamin Van Roy. (more) efficient reinforcement learning via posterior sampling. In Advances in Neural Information Processing Systems, pp. 3003–3011, 2013. find
  39. Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. arXiv preprint arXiv:1511.06342, 2015. find
  40. Theodore J Perkins, Doina Precup, et al. Using options for knowledge transfer in reinforcement learning. University of Massachusetts, Amherst, MA, USA, Tech. Rep, 1999. find
  41. Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. arXiv preprint arXiv:1511.06295, 2015. find
  42. Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04671, 2016a. find
  43. Andrei A Rusu, Matej Vecerik, Thomas Rothörl, Nicolas Heess, Razvan Pascanu, and Raia Hadsell. Sim-to-real robot learning from pixels with progressive nets. arXiv preprint arXiv:1610.04286, 2016b. — [arXiv:1610.04286](https://arxiv.org/abs/1610.04286) find
  44. Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. One-shot learning with memory-augmented neural networks. arXiv preprint arXiv:1605.06065, 2016. find
  45. John Schulman, Sergey Levine, Philipp Moritz, Michael I Jordan, and Pieter Abbeel. Trust region policy optimization. CoRR, abs/1502.05477, 2015. find
  46. John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In International Conference on Learning Representations (ICLR2016), 2016. find
  47. Nicolas Schweighofer and Kenji Doya. Meta-learning in reinforcement learning. Neural Networks, 16(1):5–9, 2003. find
  48. Satinder Pal Singh. Transfer of learning by composing solutions of elemental sequential tasks. Machine Learning, 8(3-4):323–339, 1992. find
  49. Malcolm Strens. A bayesian framework for reinforcement learning. In ICML, pp. 943–950, 2000. find
  50. Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(Jul):1633–1685, 2009. find
  51. William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285–294, 1933. find
  52. Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-learning. Artificial Intelligence Review, 18(2):77–95, 2002. find
  53. Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. arXiv preprint arXiv:1606.04080, 2016.
  54. Niklas Wahlström, Thomas B Schön, and Marc Peter Deisenroth. From pixels to torques: Policy learning with deep dynamical models. arXiv preprint arXiv:1502.02251, 2015. — [arXiv:1502.02251](https://arxiv.org/abs/1502.02251) find
  55. Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In Advances in Neural Information Processing Systems, pp. 2746–2754, 2015. find
  56. Peter Whittle. Optimization over time. John Wiley & Sons, Inc., 1982. find
  57. Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learning: a hierarchical bayesian approach. In Proceedings of the 24th international conference on Machine learning, pp. 1015–1022. ACM, 2007. find
  58. A Steven Younger, Sepp Hochreiter, and Peter R Conwell. Meta-learning with backpropagation. In Neural Networks, 2001. Proceedings. IJCNN’01. International Joint Conference on, volume 3. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/duan-2016-rl-2-fast-reinforcement-learning/