Elamur Research⌕
☰
home / papers / 2015 / Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models

Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models

Paper · 2015 · Bradly C. Stadie, Sergey Levine and Pieter Abbeel

Summary

Exploration in reinforcement learning with high-dimensional observations is still hard. Count-based bonus methods such as MBIE-EB and BEB come with theoretical guarantees, but they need to enumerate states. Large-scale agents therefore usually fall back on epsilon-greedy. The authors replace visitation counts with the "surprise" of a learned dynamics model.

Atari frames are first compressed by an autoencoder. A neural network then predicts the next encoded frame from the current encoding and the chosen action. The squared prediction error is normalized by the largest error seen so far, decayed over time, and added to the game reward as a novelty bonus. The scheme works with any online RL learner. Here it is combined with DQN on Arcade Learning Environment games and compared with Boltzman exploration and Thompson sampling.

Model-based bonuses gave the most consistent improvements, especially on games where humans far outperform earlier agents, and they also accelerated learning. To measure learning speed, the paper introduces AUC-100, a normalized area under the score curve over the first 100 epochs.

Key points

  • Novelty signal: the novelty of a transition is the normalized squared error of the dynamics predictor in the autoencoded space. Predicting raw pixels produced nearly uniform, uninformative bonuses.
  • Encoder design: an eight-layer autoencoder is used, taking its sixth layer as the state code. On Seaquest, this layer beat the fourth-layer bottleneck.
  • Training schedule: the policy updates every step, while the dynamics model is refit once per epoch (50,000 observations). The encoder is retrained every 5 epochs.
  • AUC-100 results: examples include Bowling (DQN 0.964, static-AE bonus 1.493, dynamic-AE bonus 1.836), Frostbite (0.573 vs 0.971 with static AE), and Seaquest (0.16 vs 0.265 with dynamic AE). Every method scored 0 on Montezuma.
  • Model learning: across all runs, the mean normalized prediction residual after 100 epochs fell to roughly half of its peak. This suggests the models genuinely learned game dynamics.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksDecision Sciences · 306 worksManagement Science and Operations Research · 145 works

Cite

Cite as Bradly C. Stadie, Sergey Levine and Pieter Abbeel. 2015. Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models. arXiv:1507.00814.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1507.00814 · arXiv 1507.00814

@misc{stadie2015incentivizing,
  title = {{Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models}},
  author = {Bradly C. Stadie, Sergey Levine and Pieter Abbeel},
  year = {2015},
  doi = {10.48550/arxiv.1507.00814},
  eprint = {1507.00814},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1507.00814}
}

References

25 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. L. Strehl and M. L. Littman, An Analysis of Model-Based Interval Estimation for Markov Decision Processes. Journal of Computer and System Sciences, 74, 12091331. find
  2. J. Z. Kolter and A. Y. Ng, Near-Bayesian Exploration in Polynomial Time Proceedings of the 26th Annual International Conference on Machine Learning, pp. 18, 2009. find
  3. V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, et al. Human-level Control Through Deep Reinforcement Learning. Nature, 518(7540):529533, 2015. find
  4. T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning. Springer series in statistics. Springer, New York, 2001. find
  5. M. Belkin, P. Niyogi and V. Sindhwani, Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples. JMLR, vol. 7, pp. 2399-2434, Nov. 2006. find
  6. G. E. Hinton and R. Salakhutdinov, Reducing the dimensionality of data with neural networks. Science, 313, 504–507, 2006. find
  7. R. I. Brafman and M. Tennenholtz, R-max, a general polynomial time algorithm for near-optimal reinforcement learning. Journal of Machine Learning Research, 2002. find
  8. M. Kearns and D. Koller, Efficient reinforcement learning in factored MDPs. Proc. IJCAI, 1999. find
  9. M. Kearns and S. Singh, Near-optimal reinforcement learning in polynomial time. Machine Learning Journal, 2002. find
  10. W. D. Smart and L. P. Kaelbling, Practical reinforcement learning in continuous spaces. Proc. ICML, 2000. find
  11. J. Sorg, S. Singh, R. L. Lewis, Variance-Based Rewards for Approximate Bayesian Reinforcement Learning. Proc. UAI, 2010. find
  12. M. Lopes, T. Lang, M. Toussaint and P.-Y. Oudeyer, Exploration in Model-based Reinforcement Learning by Empirically Estimating Learning Progress. NIPS, 2012. find
  13. M. Geist and O. Pietquin, Managing Uncertainty within Value Function Approximation in Reinforcement Learning. W. on Active Learning and Experimental Design, 2010. find
  14. M. Araya, O. Buffet, and V. Thomas. Near-optimal BRL Using Optimistic Local Transitions. (ICML-12), ser. ICML 12, J. Langford and J. Pineau, Eds. New York, NY, USA: Omnipress, Jul. 2012, pp. 97-104. find
  15. F. Doshi-Velez, D. Wingate, N. Roy, and J. Tenenbaum, Nonparametric Bayesian Policy Priors for Reinforcement Learning. NIPS, 2014. find
  16. T. Lang, M. Toussaint, K. Keristing, Exploration in relational domains for model-based reinforcement learning Proc. AAMAS, 2014. find
  17. Juergen Schmidhuber Developmental Robotics, Optimal Artificial Curiosity, Creativity, Music, and the Fine Arts. . Connection Science, vol. 18 (2), p 173-187. 2006. find
  18. J. Pazis and R. Parr, PAC Optimal Exploration in Continuous Space Markov Decision Processes. Proc. AAAI, 2013. find
  19. Kakade, S., Kearns, M., and Langford, J. (2003). Exploration in metric state spaces. Proc. ICML. find
  20. A. Guez, D. Silver, P. Dayan, Efficient Bayes-Adaptive Reinforcement Learning using Sample-Based Search. NIPS, 2014. find
  21. X. Guo, S. Singh, H. Lee, R. Lewis, X. Wang, Deep Learning for Real-Time Atari Game Play Using Offline Monte-Carlo Tree Search Planning. NIPS, 2014. find
  22. Y Gal, Z Ghahramani. Dropout as a Bayesian approximation: Insights and applications Deep Learning Workshop, ICML find
  23. David Carmel and Shaul Markovitch. Exploration Strategies for Model-based Learning in Multi-agent Systems Autonomous Agents and Multi-Agent Systems Volume 2, Issue 2 , pp 141-172 find
  24. M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling, The Arcade Learning Environment: An Evaluation Platform for General Agents. JAIR. Volume 47, p.235-279. June 2013.
  25. John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, Pieter Abbeel. Trust Region Policy Optimization Arxiv preprint 1502.05477. A PPENDIX 8.1 O N AUTO ENCODER LAYER SELECTION find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/stadie-2015-incentivizing-exploration-reinforcement-learning-deep/