Elamur Research⌕
☰
home / papers / 2017 / Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents

Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents

Paper · 2017 · Marlos C. Machado, Marc G. Bellemare, Erik Talvitie et al.

Summary

The Arcade Learning Environment (ALE) is a popular testbed for general AI agents that must play many Atari 2600 games without game-specific knowledge. As its use grew, research groups adopted different and often incompatible evaluation setups, which made published results hard to compare. This article reviews how the community has used the ALE, shows how evaluation protocols have diverged and been mixed up, and proposes best practices for the standard reinforcement learning setting. The authors also respond to criticism that the emulator is fully deterministic. They release a new ALE version with a stochasticity mechanism called sticky actions and support for alternative game modes and difficulty levels. Following their own recommendations, they report new benchmark results for Sarsa(λ) with Blob-PROST features (linear) and DQN (neural). Performance is reported at several stages of training to encourage attention to sample efficiency. Finally, they revisit the challenges from the original ALE paper and summarize the state of the art. They name five problems that remain open, offering guidance for future work on this and similar general-competency benchmarks.

Key points

  • In the ALE, an agent sees a 210 × 160 screen image and/or the 1024-bit RAM state. One frame lasts 1/60 s, and the underlying emulator is deterministic.
  • Common preprocessing steps include frame skipping (typically k = 4 or 5), color averaging or frame pooling, and frame stacking. Rewards are often normalized, for example by clipping them to their sign as in DQN.
  • The new ALE release adds sticky actions to inject randomness. It also adds "flavors," meaning mode–difficulty pairs such as the different modes of Freeway, which create many closely related environments suited to transfer learning.
  • Benchmarks compare linear Sarsa(λ) + Blob-PROST against DQN and report results at several points during learning, not only the final score.
  • The five open problems are representation learning, planning and model learning, exploration, transfer learning, and off-policy learning. For off-policy learning, the authors note that methods like Q-learning and GQ(λ) tend to be unstable in the ALE.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States378 works · 161,630 citations🏢Google DeepMindLondon · United Kingdom72 works · 84,326 citations🎓University of AlbertaEdmonton · Canada51 works · 5,017 citations

Also on the byline: Franklin & Marshall College.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

Cite

Cite as Marlos C. Machado, Marc G. Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht and Michael Bowling. 2017. Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents. arXiv:1709.06009.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1709.06009 · arXiv 1709.06009

@misc{machado2017revisiting,
  title = {{Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents}},
  author = {Marlos C. Machado, Marc G. Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht and Michael Bowling},
  year = {2017},
  doi = {10.48550/arxiv.1709.06009},
  eprint = {1709.06009},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1709.06009}
}

References

79 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Albrecht, S. V., L., J. C., Buckeridge, D. L., Botea, A., Caragea, C., Chi, C., Damoulas, T., Dilkina, B. N., Eaton, E., Fazli, P., Ganzfried, S., Lindauer, M. T., Machado, M. C., Malitsky, Y., Marcus, G., Meijer, S., Rossi, F., Shaban-Nejad, A., Thiebaux, S., Veloso, M. M., Walsh, T., Wang, C., Zha… find
  2. Barto, A. G. (2013). Intrinsic Motivation and Reinforcement Learning. In Intrinsically Motivated Learning in Natural and Artificial Systems, pp. 17–47. Springer. find
  3. Beattie, C., Leibo, J. Z., Teplyashin, D., Ward, T., Wainwright, M., Küttler, H., Lefrancq, A., Green, S., Valdés, V., Sadik, A., Schrittwieser, J., Anderson, K., York, S., Cant, M., Cain, A., Bolton, A., Gaffney, S., King, H., Hassabis, D., Legg, S., & Petersen, S. (2016). DeepMind Lab. CoRR, abs…
  4. Bellemare, M., Veness, J., & Bowling, M. (2013). Bayesian Learning of Recursively Factored Environments. In Proceedings of the International Conference on Machine Learning (ICML), pp. 1211–1219. find
  5. Bellemare, M. G., Dabney, W., & Munos, R. (2017). A Distributional Perspective on Reinforcement Learning. In Proceedings of the International Conference on Machine Learning (ICML). find
  6. Bellemare, M. G., Naddaf, Y., Veness, J., & Bowling, M. (2013). The Arcade Learning Environment: An Evaluation Platform for General Agents. Journal of Artificial Intelligence Research, 47, 253–279.
  7. Bellemare, M. G., Naddaf, Y., Veness, J., & Bowling, M. (2015). The Arcade Learning Environment: An Evaluation Platform for General Agents (Extended Abstract). In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pp. 4148–4152. find
  8. Bellemare, M. G., Ostrovski, G., Guez, A., Thomas, P. S., & Munos, R. (2016a). Increasing the Action Gap: New Operators for Reinforcement Learning. In Proceedings of the Conference on Artificial Intelligence (AAAI), pp. 1476–1483. find
  9. Bellemare, M. G., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., & Munos, R. (2016b). Unifying Count-Based Exploration and Intrinsic Motivation. In Advances in Neural Information Processing Systems (NIPS). find
  10. Bellemare, M. G., Veness, J., & Bowling, M. (2012a). Investigating Contingency Awareness using Atari 2600 Games. In Proceedings of the Conference on Artificial Intelligence (AAAI), pp. 864–871. find
  11. Bellemare, M. G., Veness, J., & Bowling, M. (2012b). Sketch-Based Linear Value Function Approximation. In Advances in Neural Information Processing Systems (NIPS), pp. 2222–2230. find
  12. Bellemare, M. G., Veness, J., & Talvitie, E. (2014). Skip Context Tree Switching. In Proceedings of the International Conference on Machine Learning (ICML), pp. 1458– 1466. find
  13. Bellman, R. E. (1957). Dynamic Programming. Princeton University Press, Princeton, NJ. find
  14. Brafman, R. I., & Tennenholtz, M. (2002). R-MAX - A General Polynomial Time Algorithm for Near-Optimal Reinforcement Learning. Journal of Machine Learning Research, 3, 213–231. find
  15. Braylan, A., Hollenbeck, M., Meyerson, E., & Miikkulainen, R. (2015). Frame Skip is a Powerful Parameter for Learning to Play Atari. In AAAI Workshop on Learning for General Competency in Video Games. find
  16. Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., & Zaremba, W. (2016). OpenAI Gym. CoRR, abs/1507.04296.
  17. Campbell, M., Jr., A. J. H., & Hsu, F. (2002). Deep Blue. Artificial Intelligence, 134 (1-2), 57–83. find
  18. Chiappa, S., Racaniere, S., Wierstra, D., & Mohamed, S. (2017). Recurrent Environment Simulators. In Proceedings of the International Conference on Learning Representations (ICLR). find
  19. Defazio, A., & Graepel, T. (2014). A Comparison of Learning Algorithms on the Arcade Learning Environment. CoRR, abs/1410.8620. — [arXiv:1410.8620](https://arxiv.org/abs/1410.8620) find
  20. Even-Dar, E., Kakade, S. M., & Mansour, Y. (2005). Reinforcement Learning in POMDPs Without Resets. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pp. 690–695. find
  21. Farahmand, A.-M., Barreto, A., & Nikovski, D. (2017). Value-Aware Loss Function for Model-based Reinforcement Learning. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS). find
  22. Guo, X., Singh, S., Lee, H., Lewis, R. L., & Wang, X. (2014). Deep Learning for Real-Time Atari Game Play Using Offline Monte-Carlo Tree Search Planning. In Advances in Neural Information Processing Systems (NIPS), pp. 3338–3346. find
  23. Harutyunyan, A., Bellemare, M. G., Stepleton, T., & Munos, R. (2016). Q(λ) with Off-Policy Corrections. In Proceedings of the International Conference on Algorithmic Learning Theory (ALT). find
  24. Hausknecht, M., & Stone, P. (2015). The Impact of Determinism on Learning Atari 2600 Games. In AAAI Workshop on Learning for General Competency in Video Games. find
  25. Hausknecht, M. J., Lehman, J., Miikkulainen, R., & Stone, P. (2014). A Neuroevolution Approach to General Atari Game Playing. IEEE Transactions on Computational Intelligence and AI in Games, 6 (4), 355–366. find
  26. Hester, T., Vecerik, M., Pietquin, O., Lanctot, M., Schaul, T., Piot, B., Sendonaris, A., Dulac-Arnold, G., Osband, I., Agapiou, J., et al. (2017). Learning from Demonstrations for Real World Reinforcement Learning. CoRR, abs/1704.03732. — [arXiv:1704.03732](https://arxiv.org/abs/1704.03732) find
  27. Hutter, M. (2005). Universal Artificial Intelligence: Sequential Decisions based on Algorithmic Probability. Springer. find
  28. Islam, R., Henderson, P., Gomrokchi, M., & Precup, D. (2017). Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control. In ICML Workshop on Reproducibility in Machine Learning. find
  29. Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., Leibo, J. Z., Silver, D., & Kavukcuoglu, K. (2017). Reinforcement Learning with Unsupervised Auxiliary Tasks. In Proceedings of the International Conference on Learning Representations (ICLR).
  30. Jinnai, Y., & Fukunaga, A. (2017). Learning to Prune Dominated Action Sequences in Online Black-Box Planning. In Proceedings of the Conference on Artificial Intelligence (AAAI), pp. 839–845. find
  31. Johnson, M., Hofmann, K., Hutton, T., & Bignell, D. (2016). The Malmo Platform for Artificial Intelligence Experimentation. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pp. 4246–4247. find
  32. Kearns, M. J., Mansour, Y., & Ng, A. Y. (1999). Approximate Planning in Large POMDPs via Reusable Trajectories. In Advances in Neural Information Processing Systems (NIPS), pp. 1001–1007. find
  33. Kearns, M. J., & Singh, S. P. (2002). Near-Optimal Reinforcement Learning in Polynomial Time. Machine Learning, 49 (2-3), 209–232. find
  34. Kelly, S., & Heywood, M. I. (2017). Emergent Tangled Graph Representations for Atari Game Playing Agents. In European Conference on Genetic Programming (EuroGP), pp. 64–79. find
  35. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems (NIPS), pp. 1106–1114.
  36. Levine, J., Congdon, C. B., Ebner, M., Kendall, G., Lucas, S. M., Miikkulainen, R., Schaul, T., & Thompson, T. (2013). General Video Game Playing. In Dagstuhl Follow-Ups. find
  37. Liang, Y., Machado, M. C., Talvitie, E., & Bowling, M. H. (2016). State of the Art Control of Atari Games Using Shallow Reinforcement Learning. In Proceedings of the International Conference on Autonomous Agents & Multiagent Systems (AAMAS), pp. 485–493. find
  38. Lin, L.-J. (1993). Reinforcement Learning for Robots Using Neural Networks. Tech. rep., Carnegie Mellon University, School of Computer Science. find
  39. Lipovetzky, N., Ramirez, M., & Geffner, H. (2015). Classical Planning with Simulators: Results on the Atari Video Games. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pp. 1610–1616. find
  40. Machado, M. C., Bellemare, M. G., & Bowling, M. (2017). A Laplacian Framework for Option Discovery in Reinforcement Learning. In Proceedings of the International Conference on Machine Learning (ICML). find
  41. Machado, M. C., Srinivasan, S., & Bowling, M. (2015). Domain-Independent Optimistic Initialization for Reinforcement Learning. In AAAI Workshop on Learning for General Competency in Video Games. find
  42. Maei, H. R., & Sutton, R. S. (2010). GQ(λ): A General Gradient Algorithm for Temporal-Difference Prediction Learning with Eligibility Traces. In Proceedings of the Conference on Artificial General Intelligence (AGI), pp. 719–726. find
  43. Martin, J., Sasikumar, S. N., Everitt, T., & Hutter, M. (2017). Count-Based Exploration in Feature Space for Reinforcement Learning. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI). find
  44. Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016). Asynchronous Methods for Deep Reinforcement Learning. In Proceedings of the International Conference on Machine Learning (ICML), pp. 1928–1937. find
  45. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., & Hassabis, D. (2015). Human-level Control through Deep… find
  46. Montfort, N., & Bogost, I. (2009). Racing the Beam: The Atari Video Computer System. MIT Press. find
  47. Munos, R., Stepleton, T., Harutyunyan, A., & Bellemare, M. G. (2016). Safe and Efficient Off-Policy Reinforcement Learning. In Advances in Neural Information Processing Systems (NIPS). find
  48. Naddaf, Y. (2010). Game-independent AI Agents for Playing Atari 2600 Console Games. Master’s thesis, University of Alberta. find
  49. Nair, A., Srinivasan, P., Blackwell, S., Alcicek, C., Fearon, R., Maria, A. D., Panneershelvam, V., Suleyman, M., Beattie, C., Petersen, S., Legg, S., Mnih, V., Kavukcuoglu, K., & Silver, D. (2015). Massively Parallel Methods for Deep Reinforcement Learning. In ICML Deep Learning Workshop.
  50. Oh, J., Guo, X., Lee, H., Lewis, R. L., & Singh, S. P. (2015). Action-Conditional Video Prediction using Deep Networks in Atari Games. In Advances in Neural Information Processing Systems (NIPS), pp. 2863–2871.
  51. Ontanon, S., Synnaeve, G., Uriarte, A., Richoux, F., Churchill, D., & Preuss, M. (2013). A Survey of Real-Time Strategy Game AI Research and Competition in StarCraft. IEEE Transactions on Computational Intelligence and AI in Games, 5, 293–311. find
  52. Osband, I., Blundell, C., Pritzel, A., & Roy, B. V. (2016). Deep Exploration via Bootstrapped DQN. In Advances in Neural Information Processing Systems (NIPS), pp. 4026–4034.
  53. Ostrovski, G., Bellemare, M. G., van den Oord, A., & Munos, R. (2017). Count-Based Exploration with Neural Density Models. In Proceedings of the International Conference on Machine Learning (ICML). find
  54. Oudeyer, P., Kaplan, F., & Hafner, V. (2007). Intrinsic Motivation Systems for Autonomous Mental Development. IEEE Transactions on Evolutionary Computation, 11 (2), 265– 286. find
  55. Parisotto, E., Ba, L. J., & Salakhutdinov, R. (2016). Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning. In Proceedings of the International Conference on Learning Representations (ICLR). find
  56. Ring, M. (1997). CHILD: A First Step Towards Continual Learning. Machine Learning, 28 (1), 77–104. find
  57. Rummery, G. A., & Niranjan, M. (1994). On-line Q-Learning using Connectionist Systems. CUED/F-INFENG/TR 166, Cambridge University Engineering Dept. find
  58. Rusu, A. A., Colmenarejo, S. G., Gucehre, C., Desjardins, G., Kirkpatrick, J., Pascanu, R., Mnih, V., Kavukcuoglu, K., & Hadsell, R. (2016). Policy Distillation. In Proceedings of the International Conference on Learning Representations (ICLR). find
  59. Schaeffer, J., Burch, N., Bjrnsson, Y., Kishimoto, A., Müller, M., Lake, R., Lu, P., & Sut-phen, S. (2007). Checkers is Solved. Science, 317 (5844), 1518–1522. find
  60. Schaul, T., Quan, J., Antonoglou, I., & Silver, D. (2016). Prioritized Experience Replay. In Proceedings of the International Conference on Learning Representations (ICLR). find
  61. Shleyfman, A., Tuisov, A., & Domshlak, C. (2016). Blind Search for Atari-Like Online Planning Revisited. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pp. 3251–3257. find
  62. Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T., Leach, M., Kavukcuoglu, K., Graepel, T., & Hassabis, D. (2016). M… find
  63. Singh, S., Barto, A. G., & Chentanez, N. (2004). Intrinsically Motivated Reinforcement Learning. In Advances in Neural Information Processing Systems (NIPS). find
  64. Stadie, B. C., Levine, S., & Abbeel, P. (2015). Incentivizing Exploration in Reinforcement Learning With Deep Predictive Models. CoRR, abs/1507.00814. find
  65. Strehl, A. L., & Littman, M. L. (2008). An Analysis of Model-Based Interval Estimation for Markov Decision Processes. Journal of Computer and System Sciences, 74 (8), 1309–1331. find
  66. Sutton, R. S., & Barto, A. G. (1998). Reinforcement Learning: An Introduction. MIT Press. find
  67. Sutton, R., Modayil, J., Delp, M., Degris, T., Pilarski, P., White, A., & Precup, D. (2011). Horde: A Scalable Real-Time Architecture for Learning Knowledge from Unsupervised Sensorimotor Interaction. In Proceedings of the International Conference on Autonomous Agents & Multiagent Systems (AAMAS), p… find
  68. Talvitie, E. (2014). Model Regularization for Stable Sample Rollouts. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI), pp. 780–789. find
  69. Talvitie, E. (2017). Self-correcting models for model-based reinforcement learning. In Proceedings of the Conference on Artificial Intelligence (AAAI), pp. 2597–2603. find
  70. Taylor, M. E., & Stone, P. (2009). Transfer Learning for Reinforcement Learning Domains: A Survey. Journal of Machine Learning Research, 10, 1633–1685. find
  71. Thrun, S., & Mitchell, T. M. (1993). find
  72. Lifelong Robot Learning. In Robotics and Autonomous Systems. van Hasselt, H., Guez, A., & Silver, D. (2016). Deep Reinforcement Learning with Double Q-Learning. In Proceedings of the Conference on Artificial Intelligence (AAAI), pp. 2094–2100. find
  73. Veness, J., Bellemare, M. G., Hutter, M., Chua, A., & Desjardins, G. (2015). Compress and Control. In Proceedings of the Conference on Artificial Intelligence (AAAI), pp. 3016–3023. find
  74. Venkatraman, A., Hebert, M., & Bagnell, J. A. (2015). Improving Multi-Step Prediction of Learned Time Series Models. In Proceedings of the Conference on Artificial Intelligence (AAAI). find
  75. Vezhnevets, A. S., Osindero, S., Schaul, T., Heess, N., Jaderberg, M., Silver, D., & Kavukcuoglu, K. (2017). FeUdal Networks for Hierarchical Reinforcement Learning. In Proceedings of the International Conference on Machine Learning (ICML). find
  76. Wang, Z., Schaul, T., Hessel, M., van Hasselt, H., Lanctot, M., & De Freitas, N. (2016). Dueling Network Architectures for Deep Reinforcement Learning. In Proceedings of the International Conference on Machine Learning (ICML). find
  77. Wilson, S. (1985). Knowledge Growth in an Artificial Animal. In Proceedings of the International Conference on Genetic Algorithms (ICGA). find
  78. Zahavy, T., Ben-Zrihem, N., & Mannor, S. (2016). Graying the Black Box: Understanding DQNs. In Proceedings of the International Conference on Machine Learning (ICML), pp. 1899–1908. find
  79. O 1 H 1 A 2 O 2 H 2 Figure 9: History tree representation of an environment. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/machado-2017-revisiting-arcade-learning-environment-evaluation/