Elamur Research⌕
☰
home / papers / 2024 / Gymnasium: A Standard Interface for Reinforcement Learning Environments

Gymnasium: A Standard Interface for Reinforcement Learning Environments

Paper · 2024 · Mark Towers, Ariel Kwiatkowski, Jordan Terry et al.

Summary

Progress in reinforcement learning has been slowed by inconsistent environment and algorithm implementations, which makes results hard to compare and reuse. OpenAI Gym once filled the role of a common interface, but OpenAI stopped maintaining it and released no updates after October 2022. This NeurIPS 2025 paper presents Gymnasium, the Farama Foundation's maintained successor. Its core abstractions are:

  • Env, a loose analogue of a POMDP driven by reset and step.
  • VectorEnv, which runs many sub-environments in parallel and batches their outputs.
  • Spaces, which describe observation and action structure.
  • Registry, a versioned catalogue that makes environment creation reproducible.

The library also ships wrappers and built-in environments ranging from toy MDPs to robotic simulators. Additions beyond Gym include a functional environment API suited to search methods and hardware acceleration, broader vectorization support with configurable autoreset, and richer spaces for multi-modal agents. Gymnasium is widely adopted and integrates with major training libraries. A stable shared interface cuts engineering effort and improves reproducibility. Its limits are that it is written in Python and does not train agents itself.

Key points

  • Adoption: the authors report more than 18 million downloads up to May 2025, over a million in April 2025 alone, roughly 500 GitHub issues, 800 pull requests, and over 40 contributors.
  • Compatible ecosystem: Stable Baselines3, CleanRL, Tianshou, Ray RLlib, and Dopamine on the training side; Minari, ALE, Metaworld, and Gymnasium Robotics as related tooling.
  • Autoreset modes: vector environments support three. Next-step is the default and fastest; same-step was Gym's only mode; disabled requires a masked explicit reset.
  • New spaces: Text, Sequence, Graph, and OneOf (disjoint union) are unique to Gymnasium. They sit alongside Box, Discrete, MultiBinary, MultiDiscrete, Dict, and Tuple.
  • Custom vectorization: a custom vectorized CartPole built on tensor operations runs roughly ten times more steps per second than the Sync or Async vectorizers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Université de MontréalMontreal · Canada87 works · 91,597 citations🔬Mila - Quebec Artificial Intelligence InstituteMontreal · Canada65 works · 14,425 citations🎓University of SouthamptonSouthampton · United Kingdom13 works · 2,608 citations

Also on the byline: Technical University of Crete.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

Cite

Cite as Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U. Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulao, Andreas Kallinteris, Markus Krimmel, Arjun KG, Rodrigo Perez-Vicente, Andrea Pierre, Sander Schulhoff, Jun Jet Tai, Hannah Tan and Omar G. Younis. 2024. Gymnasium: A Standard Interface for Reinforcement Learning Environments. arXiv:2407.17032.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2407.17032 · arXiv 2407.17032

@misc{towers2024gymnasium,
  title = {{Gymnasium: A Standard Interface for Reinforcement Learning Environments}},
  author = {Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U. Balis, Gianluca {De Cola}, Tristan Deleu, Manuel Goulao, Andreas Kallinteris, Markus Krimmel, Arjun KG, Rodrigo Perez-Vicente, Andrea Pierre, Sander Schulhoff, Jun Jet Tai, Hannah Tan and Omar G. Younis},
  year = {2024},
  doi = {10.48550/arxiv.2407.17032},
  eprint = {2407.17032},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.17032}
}

References

57 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lucas N. Alegre. SUMO-RL. https://github.com/LucasAlegre/sumo-rl, 2019. ↗
  2. Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay, 2018. URL https://arxiv.org/abs/1707.01495. ↗
  3. Charles Beattie, Thomas Köppe, Edgar A. Duéñez-Guzmán, and Joel Z. Leibo. Deepmind lab2d, 2020. URL https://arxiv.org/abs/2011.07027. — [arXiv:2011.07027](https://arxiv.org/abs/2011.07027) ↗
  4. M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253–279, jun 2013.
  5. Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019. find
  6. Clément Bonnet, Daniel Luo, Donal Byrne, Shikha Surana, Sasha Abramowitz, Paul Duckworth, Vincent Coyette, Laurence I. Midgley, Elshadai Tegegn, Tristan Kalloniatis, Omayma Mahjoub, Matthew Macfarlane, Andries P. Smit, Nathan Grinsztajn, Raphael Boige, Cemlyn N. Waters, Mohamed A. Mimouni, Ulrich A.… find
  7. Felix Book, Arne Traue, Maximilian Schenke, Barnabas Haucke-Korber, and Oliver Wallscheid. Gym-electric-motor (gem) control: An automated open-source controller design suite for drives. In 2023 IEEE International Electric Machines & Drives Conference (IEMDC), pages 1–7, 2023. doi: 10.1109/IEMDC55163… — [doi:10.1109/iemdc55163.2023.10239044](https://doi.org/10.1109/iemdc55163.2023.10239044) find
  8. Yann Bouteiller, Edouard GEZE, GobeX, Stefan Kuhn, and pius. trackmania-rl/tmrl: Release 0.7.1, May 2025. URL https://doi.org/10.5281/zenodo.15344778. — [doi:10.5281/zenodo.15344778](https://doi.org/10.5281/zenodo.15344778) ↗
  9. James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax. ↗
  10. Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016. find
  11. Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G. Bellemare. Dopamine: A Research Framework for Deep Reinforcement Learning. 2018. URL http://arxiv.org/abs/1812.06110. — [arXiv:1812.06110](https://arxiv.org/abs/1812.06110) ↗
  12. Maxime Chevalier-Boisvert, Bolun Dai, Mark Towers, Rodrigo Perez-Vicente, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. In Advances in Neural Information Process… find
  13. Kartik Choudhary, Dhawal Gupta, and Philip S. Thomas. ICU-Sepsis: A benchmark MDP built from real medical data. In Reinforcement Learning Conference, 2024. URL https://arxiv.org/ abs/2406.05646. — [arXiv:2406.05646](https://arxiv.org/abs/2406.05646) ↗
  14. Steven Dalton, Iuri Frosio, and Michael Garland. Accelerating reinforcement learning through gpu atari emulation, 2020. URL https://arxiv.org/abs/1907.08467. ↗
  15. Rodrigo de Lazcano, Kallinteris Andreas, Jun Jet Tai, Seungjae Ryan Lee, and Jordan Terry. Gymnasium robotics, 2023. URL http://github.com/Farama-Foundation/ Gymnasium-Robotics. ↗
  16. Gustavo de Moura Souza and Claudio Fabiano Motta Toledo. Monte carlo continuous action space solver for chance constrained markov decision processes. Master’s thesis, Universidade de São Paulo, 2024. find
  17. Quentin Delfosse, Jannis Blüml, Bjarne Gregori, Sebastian Sztwiertnia, and Kristian Kersting. Ocatari: Object-centric atari 2600 reinforcement learning environments, 2024. URL https://arxiv.org/abs/2306.08649. — [arXiv:2306.08649](https://arxiv.org/abs/2306.08649) ↗
  18. Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https://github.com/openai/baselines, 2017. ↗
  19. Florian Felten, Lucas N. Alegre, Ann Nowé, Ana L. C. Bazzan, El Ghazali Talbi, Grégoire Danoy, and Bruno C. da Silva. A toolkit for reliable benchmarking and research in multi-objective reinforcement learning. In Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS 20… find
  20. C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax - a differentiable physics engine for large scale rigid body simulation, 2021. URL http://github.com/google/brax. ↗
  21. Xudong Gong, Dawei Feng, kele Xu, Weijia Wang, Zhangjun Sun, Xing Zhou, Bo Ding, and Huaimin Wang. Vvc-gym: A fixed-wing uav reinforcement learning environment for multi-goal long-horizon problems. In International Conference on Learning Representations, 2025. find
  22. DJ Groot, G Leto, A Vlaskin, A Moec, and J Ellerbroek. Bluesky-gym: Reinforcement learning environments for air traffic applications, 2024. find
  23. Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. arXiv:1801.01290 [cs, stat], January 2018. URL http://arxiv.org/abs/1801.01290. arXiv: 1801.01290 version: 1. ↗
  24. Matthew W. Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Nikola Momchev, Danila Sinopalnikov, Piotr Stańczyk, Sabela Ramos, Anton Raichuk, Damien Vincent, Léonard Hussenot, Robert Dadashi, Gabriel Dulac-Arnold, Manu Orsini, Alexis Jacq, Johan Ferret, Nino Vieillard, Seyed Kamyar Se… — [arXiv:2006.00979](https://arxiv.org/abs/2006.00979) find
  25. Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1–18, 2022. URL http://jmlr.org/papers/… ↗
  26. Shengyi Huang, Quentin Gallouédec, Florian Felten, Antonin Raffin, Rousslan Fernand Julien Dossa, Yanxiao Zhao, Ryan Sullivan, Viktor Makoviychuk, Denys Makoviichuk, Mohamad H. Danesh, Cyril Roumégous, Jiayi Weng, Chufan Chen, Md Masudur Rahman, João G. M. Araújo, Guorui Quan, Daniel Tan, Timo Klein… — [arXiv:2402.03046](https://arxiv.org/abs/2402.03046) find
  27. Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1):99–134, May 1998. ISSN 0004-3702. doi: 10.1016/S0004-3702(98)00023-X. find
  28. Saman Kazemkhani, Aarav Pandya, Daphne Cornelisse, Brennan Shacklett, and Eugene Vinitsky. Gpudrive: Data-driven, multi-agent driving simulation at 1 million fps. In Proceedings of the International Conference on Learning Representations (ICLR), 2025. URL https://arxiv.org/abs/2408.01584. ↗
  29. Michał Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaśkowski. ViZ-Doom: A Doom-based AI research platform for visual reinforcement learning. In IEEE Conference on Computational Intelligence and Games, pages 341–348, Santorini, Greece, Sep 2016. IEEE. URL http://arxiv.org/abs/16… — [arXiv:1605.02097](https://arxiv.org/abs/1605.02097) ↗
  30. Sotetsu Koyamada, Shinri Okano, Soichiro Nishimori, Yu Murata, Keigo Habara, Haruka Kita, and Shin Ishii. Pgx: Hardware-accelerated parallel game simulators for reinforcement learning. In Advances in Neural Information Processing Systems, 2023. find
  31. Edouard Leurent. An environment for autonomous driving decision-making. https://github.com/eleurent/highway-env, 2018. ↗
  32. Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael Jordan, and Ion Stoica. Rllib: Abstractions for distributed reinforcement learning. In International conference on machine learning, pages 3053–3062. PMLR, 2018. find
  33. Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement learning on web interfaces using workflow-guided exploration. In International Conference on Learning Representations (ICLR), 2018. URL https://arxiv.org/abs/1802.08802. ↗
  34. Mikel Malagón, Josu Ceberio, and Jose A. Lozano. Craftium: An extensible framework for creating reinforcement learning environments. arXiv preprint arXiv:2407.03969, 2024. — [arXiv:2407.03969](https://arxiv.org/abs/2407.03969) find
  35. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013. find
  36. Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis H… find
  37. Alistair Muldal, Yotam Doron, John Aslanides, Tim Harley, Tom Ward, and Siqi Liu. dm_env: A python interface for reinforcement learning environments, 2019. URL http://github.com/ deepmind/dm_env. ↗
  38. Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvari, Satinder Singh, Benjamin Van Roy, Richard Sutton, David Silver, and Hado Van Hasselt. Behaviour suite for reinforcement learning, 2020. URL https://arxiv.org/abs/19… — [arXiv:1908.03568](https://arxiv.org/abs/1908.03568) ↗
  39. Mathieu Poliquin. Stable retro, a maintained fork of openai’s gym-retro. https://github.com/ Farama-Foundation/stable-retro, 2025. ↗
  40. Martin L Puterman. Markov decision processes. Handbooks in operations research and management science, 2:331–434, 1990. find
  41. Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1–8, 2021. URL http://jmlr.org/papers/v22/20-1364.html. ↗
  42. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv:1707.06347 [cs], August 2017. find
  43. Yang Shao, Quan Kong, Tadayuki Matsumura, Taiki Fuji, Kiyoto Ito, and Hiroyuki Mizuno. Mask atari for deep reinforcement learning as pomdp benchmarks, 2022. URL https://arxiv.org/ abs/2203.16777. — [arXiv:2203.16777](https://arxiv.org/abs/2203.16777) ↗
  44. David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354–359, 2017. find
  45. Joseph Suarez. Pufferlib: Making reinforcement learning libraries and environments play nice, 2024. URL https://arxiv.org/abs/2406.12905. — [arXiv:2406.12905](https://arxiv.org/abs/2406.12905) ↗
  46. Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Proceedings of the 12th International Conference on Neural Information Processing Systems, NIPS’99, pages 1057–1063, Cambridge, MA, USA, Novembe… find
  47. Jun Jet Tai, Jim Wong, Mauro Innocente, Nadjim Horri, James Brusey, and Swee King Phang. Pyflyt– uav simulation environments for reinforcement learning research. arXiv preprint arXiv:2304.01305, 2023. — [arXiv:2304.01305](https://arxiv.org/abs/2304.01305) find
  48. J Terry, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sullivan, Luis S Santos, Clemens Dieffendahl, Caroline Horsch, Rodrigo Perez-Vicente, et al. Pettingzoo: Gym for multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34: 15032–15043, 2021… find
  49. J. K. Terry, Benjamin Black, and Luis Santos. Multiplayer support for the arcade learning environment, 2021b. URL https://arxiv.org/abs/2009.09341. — [arXiv:2009.09341](https://arxiv.org/abs/2009.09341) ↗
  50. Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033. find
  51. Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020. ISSN 2665-9638. doi: https://doi.org/ 10.1016/j.simpa.2020.10… — [doi:10.1016/j.simpa.2020.100022](https://doi.org/10.1016/j.simpa.2020.100022) find
  52. Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350–354, 2019. find
  53. Jiayi Weng, Huayu Chen, Dong Yan, Kaichao You, Alexis Duburcq, Minghao Zhang, Hang Su, and Jun Zhu. Tianshou: a Highly Modularized Deep Reinforcement Learning Library. arXiv:2107.14171 [cs], July 2021. URL http://arxiv.org/abs/2107.14171. arXiv: 2107.14171. — [arXiv:2107.14171](https://arxiv.org/abs/2107.14171) ↗
  54. Jiayi Weng, Min Lin, Shengyi Huang, Bo Liu, Denys Makoviichuk, Viktor Makoviychuk, Zichen Liu, Yufan Song, Ting Luo, Yukun Jiang, Zhongwen Xu, and Shuicheng Yan. En-vPool: A highly parallel reinforcement learning environment execution engine. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho… find
  55. Kenny Young and Tian Tian. Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments, 2019. URL https://arxiv.org/abs/1903.03176. — [arXiv:1903.03176](https://arxiv.org/abs/1903.03176) ↗
  56. Omar G. Younis, Rodrigo Perez-Vicente, John U. Balis, Will Dudley, Alex Davey, and Jordan K Terry. Minari, September 2024. URL https://doi.org/10.5281/zenodo.13767625. — [doi:10.5281/zenodo.13767625](https://doi.org/10.5281/zenodo.13767625) ↗
  57. Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning (CoRL), 2019. URL https://arxiv.org/abs/1910.10897. — [arXiv:1910.10897](https://arxiv.org/abs/1910.10897) ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/towers-2024-gymnasium-standard-interface-reinforcement-learning/