Elamur Research⌕
☰
home / papers / 2012 / The Arcade Learning Environment: An Evaluation Platform for General Agents

The Arcade Learning Environment: An Evaluation Platform for General Agents

Paper · 2012 · Marc G. Bellemare, Yavar Naddaf, Joel Veness et al.

Summary

The paper tackles the problem of fairly assessing AI agents meant to be broadly competent rather than tuned to a single task. Testing on a few parametrized benchmarks invites method overfitting and hides how much expert effort goes into adapting an algorithm. The authors present the Arcade Learning Environment (ALE), an open-source framework built on the Stella emulator. It turns Atari 2600 games into standard reinforcement learning problems with screen or RAM observations, 18 joystick actions and score-based rewards. Because emulator states can be saved and restored, ALE also works as a generative model for planning. The authors propose a train/test protocol: agents are tuned on a few training games and then evaluated on unseen games. As baselines, they test SARSA(λ) with linear function approximation using five feature sets, and two search methods, Full Tree and UCT, across 55 games. UCT, which uses the emulator as a simulator, scored far higher than the learning agents on most titles. The learning agents often, but not always, beat simple baselines. ALE thus offers a diverse benchmark, free of experimenter bias, for general agents.

Key points

  • ALE wraps the Stella Atari 2600 emulator. Screens are 160×210 pixels with a 128-colour palette, RAM is 1024 bits, and there are 18 discrete actions. Emulation runs at up to about 6000 frames per second.
  • The RL baselines are SARSA(λ) agents with five feature sets: Basic, BASS, DISCO, LSH and RAM. BASS, DISCO and RAM come from Naddaf (2010); the others are new.
  • The training set has five games (Asterix, Beam Rider, Freeway, Seaquest, Space Invaders), and 50 further games are used only for testing. Results are compared against Random, Const and Perturb baselines.
  • UCT reached very high scores where the learners stayed modest, for example about 290,700 on Asterix versus under 1,000 for the best learner. It still struggled on games such as Montezuma's Revenge, where it scored 0.
  • The paper appeared in JAIR 47 (2013), pp. 253–279; the arXiv preprint 1207.4708 dates from 2012. The software is released under the GNU GPL at arcadelearningenvironment.org.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AlbertaEdmonton · Canada51 works · 5,017 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

Cite

Cite as Marc G. Bellemare, Yavar Naddaf, Joel Veness and Michael Bowling. 2012. The Arcade Learning Environment: An Evaluation Platform for General Agents. arXiv:1207.4708 · Journal of Artificial Intelligence Research 47, pages 253-279.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1613/jair.3912 · arXiv 1207.4708

@misc{bellemare2012arcade,
  title = {{The Arcade Learning Environment: An Evaluation Platform for General Agents}},
  author = {Marc G. Bellemare, Yavar Naddaf, Joel Veness and Michael Bowling},
  year = {2012},
  doi = {10.1613/jair.3912},
  eprint = {1207.4708},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1613/jair.3912}
}

References

33 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bellemare, M., Veness, J., & Bowling, M. (2012). Investigating contingency awareness using Atari 2600 games. In Proceedings of the the 26th Conference on Artificial Intelligence (AAAI). find
  2. Browne, C. B., Powley, E., Whitehouse, D., Lucas, S. M., Cowling, P. I., Rohlfshagen, P., Tavener, S., Perez, D., Samothrakis, S., & Colton, S. (2012). A survey of Monte Carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in Games, 4 (1), 1 –43. find
  3. Cobo, L. C., Zang, P., Isbell, C. L., & Thomaz, A. L. (2011). Automatic state abstraction from demonstration. In Proceedings of the 22nd Second International Joint Conference on Articial Intelligence (IJCAI). find
  4. Coles, A., Coles, A., Olaya, A., Jiménez, S., López, C., Sanner, S., & Yoon, S. (2012). A survey of the seventh international planning competition. AI Magazine, 33 (1), 83–88. find
  5. Diuk, C., Cohen, A., & Littman, M. L. (2008). An object-oriented representation for efficient reinforcement learning. In Proceedings of the 25th International Conference on Machine learning (ICML). find
  6. Dowe, D. L., & Hajek, A. R. (1998). A non-behavioural, computational extension to the Turing Test. In Proceedings of the International Conference on Computational Intelligence and Multimedia Applications (ICCIMA). find
  7. Genesereth, M. R., Love, N., & Pell, B. (2005). General Game Playing: Overview of the AAAI competition. AI Magazine, 26 (2), 62–72. find
  8. Gionis, A., Indyk, P., & Motwani, R. (1999). Similarity search in high dimensions via hashing. In Proceedings of the International Conference on Very Large Databases. find
  9. Hausknecht, M., Khandelwal, P., Miikkulainen, R., & Stone, P. (2012). HyperNEAT-GGP: A HyperNEAT-based Atari general game player. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO). find
  10. Hernández-Orallo, J., & Dowe, D. L. (2010). Measuring universal intelligence: Towards an anytime intelligence test. Artificial Intelligence, 174 (18), 1508 – 1539. find
  11. Hernández-Orallo, J., & Minaya-Collado, N. (1998). A formal definition of intelligence based on an intensional variant of Kolmogorov complexity. In Proceedings of the International Symposium of Engineering of Intelligent Systems (EIS). find
  12. Hutter, M. (2005). Universal Artificial Intelligence: Sequential Decisions based on Algorithmic Probability. Springer, Berlin. find
  13. Kanerva, P. (1988). Sparse Distributed Memory. The MIT Press. find
  14. Kocsis, L., & Szepesvári, C. (2006). Bandit based Monte-Carlo planning. In Proceedings of the 15th European Conference on Machine Learning (ECML). find
  15. Legg, S. (2008). Machine Super Intelligence. Ph.D. thesis, University of Lugano. find
  16. Legg, S., & Veness, J. (2011). An approximation of the universal intelligence measure. In Proceedings of the Ray Solomonoff Memorial Conference. find
  17. Mohan, S., & Laird, J. E. (2009). Learning to play Mario. Tech. rep. CCA-TR-2009-03, Center for Cognitive Architecture, University of Michigan. find
  18. Monroy, G. A., Stanley, K. O., & Miikkulainen, R. (2006). Coevolution of neural networks using a layered pareto archive. In Proceedings of the 8th Genetic and Evolutionary Computation Conference (GECCO). find
  19. Montfort, N., & Bogost, I. (2009). Racing the Beam: The Atari Video Computer System. MIT Press. find
  20. Naddaf, Y. (2010). Game-Independent AI Agents for Playing Atari 2600 Console Games. Master’s thesis, University of Alberta. find
  21. Pell, B. (1993). Strategy Generation and Evaluation for Meta-Game Playing. Ph.D. thesis, University of Cambridge. find
  22. Pierce, D., & Kuipers, B. (1997). Map learning with uninterpreted sensors and effectors. Artificial Intelligence, 92 (1-2), 169–227. find
  23. Russell, S. J. (1997). Rationality and intelligence. Artificial intelligence, 94 (1), 57–77. find
  24. Schaul, T., Togelius, J., & Schmidhuber, J. (2011). Measuring intelligence through games. CoRR, abs/1109.1314. — [arXiv:1109.1314](https://arxiv.org/abs/1109.1314) find
  25. Schweitzer, P. J., & Seidmann, A. (1985). Generalized polynomial approximations in Markovian decision processes. Journal of mathematical analysis and applications, 110 (2), 568–582. find
  26. Stober, J., & Kuipers, B. (2008). From pixels to policies: A bootstrapping agent. In Proceedings of the 7th IEEE International Conference on Development and Learning (ICDL). find
  27. Sutton, R. S., & Barto, A. G. (1998). Reinforcement Learning: An Introduction. The MIT Press. find
  28. Sutton, R., Modayil, J., Delp, M., Degris, T., Pilarski, P., White, A., & Precup, D. (2011). Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In Proceedings of the 10th International Conference on Autonomous Agents and Multiagents Systems (A… find
  29. Thrun, S., & Mitchell, T. M. (1995). Lifelong robot learning. Robotics and Autonomous Systems, 15 (1), 25–46. find
  30. Watkins, C., & Dayan, P. (1992). Q-learning. Machine Learning, 8, 279–292. find
  31. Whiteson, S., Tanner, B., Taylor, M. E., & Stone, P. (2011). Protecting against evaluation overfitting in empirical reinforcement learning. In Proceedings of the IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL). find
  32. Whiteson, S., Tanner, B., & White, A. (2010). The reinforcement learning competitions. AI Magazine, 31 (2), 81–94. find
  33. Wintermute, S. (2010). Using imagery to simplify perceptual abstraction in reinforcement learning agents. In Proceedings of the the 24th Conference on Artificial Intelligence (AAAI). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bellemare-2012-arcade-learning-environment-evaluation-platform/