Elamur Research⌕
☰
home / papers / 2020 / Agent57: Outperforming the Atari Human Benchmark

Agent57: Outperforming the Atari Human Benchmark

Paper · 2020 · Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski et al.

Summary

The suite of 57 Atari 2600 games (the ALE environment) has served for about ten years as the standard test of how general reinforcement learning (RL) algorithms are. Strong earlier agents such as MuZero and R2D2 achieved very high average scores. However, in several of the hardest games they learned almost nothing. These games require long exploration of the environment or linking a reward to actions taken long before it. The authors from DeepMind took the Never Give Up (NGU) agent as their basis and made three changes. First, intrinsic and extrinsic rewards are estimated by separate networks. Second, a meta-controller based on a non-stationary multi-armed bandit decides during training which policies get more resources. Each policy has its own degree of exploration and its own discount factor. Third, the backprop through time window is doubled. The new agent, Agent57, is the first to surpass the human benchmark in every one of the 57 games with a single set of hyperparameters. This shows that a single agent can combine exploration, exploitation of what it has found, and long-term credit assignment. The cost is a large amount of compute.

Key points

  • Before Agent57, MuZero exceeded 100% HNS in 51 games and R2D2 in 52. Agent57 exceeds this threshold in all 57 games.
  • The hardest games fall into two groups. In Skiing and Solaris the reward is heavily delayed. Montezuma's Revenge, Pitfall!, Private Eye and Venture require prolonged exploration, because hundreds of actions may pass before the first reward.
  • NGU's intrinsic reward combines two kinds of novelty. Within-episode novelty is computed via episodic memory, and long-term novelty via Random Network Distillation. The long-term novelty multiplier is capped at L = 5.
  • On a set of 10 hard games, the meta-controller alone improved R2D2 by about 20% CHNS. R2D2 with a fixed γ = 0.9999 beats humans in Skiing but becomes unstable on the full set of games. The bandit, in contrast, selects the discount separately for each game.
  • The authors acknowledge limitations: the agent is data-inefficient, and MuZero still has a higher average score. In some games, all current algorithms are still far from optimal play.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States533 works · 443,884 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,394 works

Cite

Cite as Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo and Charles Blundell. 2020. Agent57: Outperforming the Atari Human Benchmark. arXiv:2003.13350.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2003.13350 · arXiv 2003.13350

@misc{badia2020agent57,
  title = {{Agent57: Outperforming the Atari Human Benchmark}},
  author = {Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo and Charles Blundell},
  year = {2020},
  doi = {10.48550/arxiv.2003.13350},
  eprint = {2003.13350},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2003.13350}
}

References

46 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Achiam, J., Edwards, H., Amodei, D., and Abbeel, P. Variational option discovery algorithms. arXiv preprint arXiv:1807.10299, 2018. — [arXiv:1807.10299](https://arxiv.org/abs/1807.10299) find
  2. Andrychowicz, M., Baker, B., Chociej, M., Jozefowicz, R., McGrew, B., Pachocki, J., Petron, A., Plappert, M., Powell, G., Ray, A., et al. Learning dexterous in-hand manipulation. arXiv preprint arXiv:1808.00177, 2018. find
  3. Arjona-Medina, J. A., Gillhofer, M., Widrich, M., Unterthiner, T., Brandstetter, J., and Hochreiter, S. Rudder: Return decomposition for delayed rewards. In Advances in Neural Information Processing Systems, pp. 13544– 13555, 2019. find
  4. Aytar, Y., Pfaff, T., Budden, D., Paine, T., Wang, Z., and de Freitas, N. Playing hard exploration games by watching youtube. In Advances in Neural Information Processing Systems, pp. 2930–2941, 2018. find
  5. Barreto, A., Dabney, W., Munos, R., Hunt, J. J., Schaul, T., van Hasselt, H. P., and Silver, D. Successor features for transfer in reinforcement learning. In Advances in neural information processing systems, pp. 4055–4065, 2017. find
  6. Barto, A. G. Intrinsic motivation and reinforcement learning. In Intrinsically motivated learning in natural and artificial systems, pp. 17–47. Springer, 2013. find
  7. Bellemare, M., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., and Munos, R. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pp. 1471–1479, 2016. find
  8. Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253–279, 06 2013.
  9. Burda, Y., Edwards, H., Storkey, A., and Klimov, O. Exploration by random network distillation. arXiv preprint arXiv:1810.12894, 2018.
  10. Choi, J., Guo, Y., Moczulski, M., Oh, J., Wu, N., Norouzi, M., and Lee, H. Contingency-aware exploration in reinforcement learning. arXiv preprint arXiv:1811.01483, 2018. — [arXiv:1811.01483](https://arxiv.org/abs/1811.01483) find
  11. Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K. O., and Clune, J. Go-explore: a new approach for hard-exploration problems. arXiv preprint arXiv:1901.10995, 2019. find
  12. Eysenbach, B., Gupta, A., Ibarz, J., and Levine, S. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070, 2018.
  13. Ferret, J., Marinier, R., Geist, M., and Pietquin, O. Credit assignment as a proxy for transfer in reinforcement learning. arXiv preprint arXiv:1907.08027, 2019. — [arXiv:1907.08027](https://arxiv.org/abs/1907.08027) find
  14. Fortunato, M., Azar, M. G., Piot, B., Menick, J., Osband, I., Graves, A., Mnih, V., Munos, R., Hassabis, D., Pietquin, O., et al. Noisy networks for exploration. arXiv preprint arXiv:1706.10295, 2017. find
  15. Fortunato, M., Tan, M., Faulkner, R., Hansen, S., Badia, A. P., Buttimore, G., Deck, C., Leibo, J. Z., and Blundell, C. Generalization of reinforcement learners with working and episodic memory. In Advances in Neural Information Processing Systems, pp. 12448–12457, 2019. find
  16. Fu, J., Co-Reyes, J., and Levine, S. Ex2: Exploration with exemplar models for deep reinforcement learning. In Advances in neural information processing systems, pp. 2577–2587, 2017. find
  17. Garivier, A. and Moulines, E. On upper-confidence bound policies for non-stationary bandit problems, 2008. find
  18. Gregor, K., Rezende, D. J., and Wierstra, D. Variational intrinsic control. arXiv preprint arXiv:1611.07507, 2016. — [arXiv:1611.07507](https://arxiv.org/abs/1611.07507) find
  19. Harutyunyan, A., Dabney, W., Mesnard, T., Azar, M. G., Piot, B., Heess, N., van Hasselt, H. P., Wayne, G., Singh, S., Precup, D., et al. Hindsight credit assignment. In Advances in neural information processing systems, pp. 12467–12476, 2019. find
  20. Horgan, D., Quan, J., Budden, D., Barth-Maron, G., Hessel, M., Van Hasselt, H., and Silver, D. Distributed prioritized experience replay. arXiv preprint arXiv:1803.00933, 2018. — [arXiv:1803.00933](https://arxiv.org/abs/1803.00933) find
  21. Hung, C.-C., Lillicrap, T., Abramson, J., Wu, Y., Mirza, M., Carnevale, F., Ahuja, A., and Wayne, G. Optimizing agent behavior over long time scales by transporting value. Nature communications, 10(1):1–12, 2019. find
  22. Jaderberg, M., Dalibard, V., Osindero, S., Czarnecki, W. M., Donahue, J., Razavi, A., Vinyals, O., Green, T., Dunning, I., Simonyan, K., et al. Population based training of neural networks. arXiv preprint arXiv:1711.09846, 2017. find
  23. Kapturowski, S., Ostrovski, G., Quan, J., Munos, R., and Dabney, W. Recurrent experience replay in distributed reinforcement learning. In International Conference on Learning Representations, 2018. find
  24. Ke, N. R., Goyal, A., Bilaniuk, O., Binas, J., Charlin, L., Pal, C., and Bengio, Y. Sparse attentive backtracking: Long-range credit assignment in recurrent networks. arXiv preprint arXiv:1711.02326, 2017. — [arXiv:1711.02326](https://arxiv.org/abs/1711.02326) find
  25. Liu, Y., Luo, Y., Zhong, Y., Chen, X., Liu, Q., and Peng, J. Sequence modeling of temporal credit assignment for episodic reinforcement learning. arXiv preprint arXiv:1905.13420, 2019. — [arXiv:1905.13420](https://arxiv.org/abs/1905.13420) find
  26. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. Nature, 518 (7540):529, 2015. find
  27. Munos, R., Stepleton, T., Harutyunyan, A., and Bellemare, M. Safe and efficient off-policy reinforcement learning. In Advances in Neural Information Processing Systems, pp. 1046–1054, 2016. find
  28. Osband, I., Blundell, C., Pritzel, A., and Van Roy, B. Deep exploration via bootstrapped dqn. In Advances In Neural Information Processing Systems, pp. 4026–4034, 2016.
  29. Osband, I., Aslanides, J., and Cassirer, A. Randomized prior functions for deep reinforcement learning. In Advances in Neural Information Processing Systems, pp. 8617–8629, 2018. find
  30. Ostrovski, G., Bellemare, M. G., van den Oord, A., and Munos, R. Count-based exploration with neural density models. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 2721–2730. JMLR. org, 2017. find
  31. Oudeyer, P.-Y., Kaplan, F., and Hafner, V. V. Intrinsic motivation systems for autonomous mental development. IEEE transactions on evolutionary computation, 11(2): 265–286, 2007. find
  32. Plappert, M., Houthooft, R., Dhariwal, P., Sidor, S., Chen, R. Y., Chen, X., Asfour, T., Abbeel, P., and Andrychowicz, M. Parameter space noise for exploration. arXiv preprint arXiv:1706.01905, 2017. find
  33. Pohlen, T., Piot, B., Hester, T., Azar, M. G., Horgan, D., Budden, D., Barth-Maron, G., Van Hasselt, H., Quan, J., Večerík, M., et al. Observe and look further: Achieving consistent performance on atari. arXiv preprint arXiv:1805.11593, 2018. — [arXiv:1805.11593](https://arxiv.org/abs/1805.11593) find
  34. Puigdomènech Badia, A., Sprechmann, P., Vitvitskyi, A., Guo, D., Piot, B., Kapturowski, S., Tieleman, O., Arjovsky, M., Pritzel, A., Bolt, A., and Blundell, C. Never give up: Learning directed exploration strategies. In International Conference on Learning Representations, 2020. find
  35. Puterman, M. L. Markov decision processes. Handbooks in operations research and management science, 2:331– 434, 1990. find
  36. Salimans, T., Ho, J., Chen, X., Sidor, S., and Sutskever, I. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864, 2017. find
  37. Savinov, N., Raichuk, A., Marinier, R., Vincent, D., Pollefeys, M., Lillicrap, T., and Gelly, S. Episodic curiosity through reachability. arXiv preprint arXiv:1810.02274, 2018. — [arXiv:1810.02274](https://arxiv.org/abs/1810.02274) find
  38. Schaul, T., Borsa, D., Ding, D., Szepesvari, D., Ostrovski, G., Dabney, W., and Osindero, S. Adapting behaviour for learning progress, 2019. find
  39. Schmidhuber, J. A possibility for implementing curiosity and boredom in model-building neural controllers. In Proc. of the international conference on simulation of adaptive behavior: From animals to animats, pp. 222– 227, 1991. find
  40. Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., et al. Mastering atari, go, chess and shogi by planning with a learned model. arXiv preprint arXiv:1911.08265, 2019. find
  41. Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484–489, 2016. find
  42. Tang, H., Houthooft, R., Foote, D., Stooke, A., Chen, O. X., Duan, Y., Schulman, J., DeTurck, F., and Abbeel, P. \# exploration: A study of count-based exploration for deep reinforcement learning. In Advances in neural information processing systems, pp. 2753–2762, 2017. find
  43. Toromanoff, M., Wirbel, E., and Moutarde, F. Is deep reinforcement learning really superhuman on atari? arXiv preprint arXiv:1908.04683, 2019. — [arXiv:1908.04683](https://arxiv.org/abs/1908.04683) find
  44. Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350–354, 2019. find
  45. Weber, T., Heess, N., Buesing, L., and Silver, D. Credit assignment techniques in stochastic computation graphs. arXiv preprint arXiv:1901.01761, 2019. — [arXiv:1901.01761](https://arxiv.org/abs/1901.01761) find
  46. Xu, Z., van Hasselt, H. P., and Silver, D. Meta-gradient reinforcement learning. In Advances in neural information processing systems, pp. 2396–2407, 2018. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/badia-2020-agent57-outperforming-atari-human-benchmark/