Elamur Research⌕
☰
home / papers / 2024 / OMNI-EPIC: Open-endedness via Models of human Notions of Interestingness with Environments Programmed in Code

OMNI-EPIC: Open-endedness via Models of human Notions of Interestingness with Environments Programmed in Code

Paper · 2024 · Maxence Faldor, Jenny Zhang, Antoine Cully et al.

Summary

Open-ended learning systems need an unbounded supply of training tasks. Earlier environment generators, including the original OMNI, could only vary parameters inside a hand-designed family of worlds. OMNI-EPIC removes this limit by having foundation models write executable Python code for each new task, covering the simulated world, the reward function, and the termination condition. A task-generator LLM consults an archive of previously mastered and failed tasks and proposes a next challenge that should be neither trivial nor out of reach. A model of interestingness (MoI) then rejects proposals that are not meaningfully novel compared with similar archived tasks. RL agents train on the generated environments, and a success detector judges completion automatically, feeding results back into the archive. In this work, generated code targets a single simulator, PyBullet. The authors report a fast-diversifying stream of creative tasks and show that the curriculum adjusts to the agent's learning progress. The work matters as a step toward Darwin Completeness, the ability in principle to create any learning environment, which AI-generating algorithms are argued to require.

Key points

  • The framework pairs OMNI, an approach to open-endedness that relies on models of what people consider interesting, with EPIC (Environments Programmed in Code). Foundation models choose tasks that are both learnable and interesting, then write the corresponding environment code.
  • Each generated task includes world construction, reward, termination and success logic. The same world can therefore support different tasks just by swapping the reward, for example hitting a target versus avoiding it.
  • The pipeline has four parts: a task archive, an LLM task generator conditioned on the most similar learned and failed tasks, an MoI novelty check, and an automatic success detector.
  • The current implementation is limited to PyBullet. The main agent is an R2D2 robot. Generation starts from hand-written seed task descriptions: crossing a pride-coloured bridge with gaps, climbing a 10-step staircase, and kicking a ball into a goal.
  • The method is not tied to one robot body. An Ant quadruped with a continuous 8-dimensional action space (values in [-1, 1]) also solved generated tasks, such as crossing lava on a boat.
  • The authors note that training a full generalist agent on OMNI-EPIC tasks was beyond their lab's compute and remains future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 works

Cite

Cite as Maxence Faldor, Jenny Zhang, Antoine Cully and Jeff Clune. 2024. OMNI-EPIC: Open-endedness via Models of human Notions of Interestingness with Environments Programmed in Code. arXiv:2405.15568.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2405.15568 · arXiv 2405.15568

@misc{faldor2024omni,
  title = {{OMNI-EPIC: Open-endedness via Models of human Notions of Interestingness with Environments Programmed in Code}},
  author = {Maxence Faldor, Jenny Zhang, Antoine Cully and Jeff Clune},
  year = {2024},
  doi = {10.48550/arxiv.2405.15568},
  eprint = {2405.15568},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.15568}
}

References

58 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmieg, Michael Chang, Natalie Clay, Adrian Collister, et al. Human-timescale adaptation in an open-ended task space. arXiv preprint arXiv:2301.07608, 2023.
  2. Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik’s cube with a robot hand. arXiv preprint arXiv:1910.07113, 2019. find
  3. Adrien Baranes and Pierre-Yves Oudeyer. Active learning of inverse models with intrinsically motivated goal exploration in robots. Robotics and Autonomous Systems, 61(1):49–73, 2013.
  4. Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Yuval Noah Harari, Ya-Qin Zhang, Lan Xue, Shai Shalev-Shwartz, Gillian Hadfield, et al. Managing AI risks in an era of rapid progress. arXiv preprint arXiv:2310.17688, pp. 18, 2023. find
  5. Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
  6. Herbie Bradley, Andrew Dai, Hannah Teufel, Jenny Zhang, Koen Oostermeijer, Marco Bellagente, Jeff Clune, Kenneth Stanley, Grégory Schott, and Joel Lehman. Quality-diversity through AI feedback. arXiv preprint arXiv:2310.13032, 2023.
  7. Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.
  8. Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning, 2024. find
  9. Jeff Clune. AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence. arXiv preprint arXiv:1905.10985, 2019. — [arXiv:1905.10985](https://arxiv.org/abs/1905.10985) find
  10. Cédric Colas, Pierre Fournier, Mohamed Chetouani, Olivier Sigaud, and Pierre-Yves Oudeyer. Curious: intrinsically motivated modular multi-goal reinforcement learning. In International conference on machine learning, pp. 1331–1340. PMLR, 2019. find
  11. Cédric Colas, Tristan Karch, Olivier Sigaud, and Pierre-Yves Oudeyer. Autotelic agents with intrinsically motivated goal-conditioned reinforcement learning: a short survey. Journal of Artificial Intelligence Research, 74:1159–1199, 2022. find
  12. Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. 2016. find
  13. Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in neural information processing systems, 33:13049–13061, 2020. find
  14. Li Ding, Jenny Zhang, Jeff Clune, Lee Spector, and Joel Lehman. Quality diversity through human feedback. arXiv preprint arXiv:2310.12103, 2023. — [arXiv:2310.12103](https://arxiv.org/abs/2310.12103) find
  15. Yuqing Du, Ksenia Konyushkova, Misha Denil, Akhil Raju, Jessica Landon, Felix Hill, Nando de Freitas, and Serkan Cabi. Vision-language models as success detectors. arXiv preprint arXiv:2303.07280, 2023. — [arXiv:2303.07280](https://arxiv.org/abs/2303.07280) find
  16. Adrien Ecoffet, Jeff Clune, and Joel Lehman. Open questions in creating safe open-ended AI: tensions between control and creativity. In Artificial Life Conference Proceedings 32, pp. 27–35. MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info . . . , 2020. find
  17. Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023. find
  18. Nicolas Heess, Dhruva Tb, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, SM Eslami, et al. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv:1707.02286, 2017. — [arXiv:1707.02286](https://arxiv.org/abs/1707.02286) find
  19. JE Hopcroft. Introduction to Automata Theory, Languages, and Computation, 2001. find
  20. Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107.14795, 2021. find
  21. Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In International Conference on Machine Learning, pp. 4940–4950. PMLR, 2021. find
  22. Minqi Jiang, Tim Rocktäschel, and Edward Grefenstette. General intelligence requires rethinking exploration. Royal Society Open Science, 10(6):230539, 2023. find
  23. Arthur Juliani, Ahmed Khalifa, Vincent-Pierre Berges, Jonathan Harper, Ervin Teng, Hunter Henry, Adam Crespi, Julian Togelius, and Danny Lange. Obstacle tower: A generalization challenge in vision, control, and planning. arXiv preprint arXiv:1902.01378, 2019. — [arXiv:1902.01378](https://arxiv.org/abs/1902.01378) find
  24. Niels Justesen, Ruben Rodriguez Torrado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, and Sebastian Risi. Illuminating generalization in deep reinforcement learning through procedural level generation. arXiv preprint arXiv:1806.10729, 2018. — [arXiv:1806.10729](https://arxiv.org/abs/1806.10729) find
  25. Ingmar Kanitscheider, Joost Huizinga, David Farhi, William Hebgen Guss, Brandon Houghton, Raul Sampedro, Peter Zhokhov, Bowen Baker, Adrien Ecoffet, Jie Tang, et al. Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraft. arXiv preprint arXiv:2106.14876, 2021. — [arXiv:2106.14876](https://arxiv.org/abs/2106.14876) find
  26. Victoria Krakovna, Jonathan Uesato, Vladimirand Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, and Shane Legg. Specification gaming: the flip side of AI ingenuity. https://deepmind.google/discover/blog/ specification-gaming-the-flip-side-of-ai-ingenuity/, 2020. ↗
  27. Joel Lehman, Jonathan Gordon, Shawn Jain, Kamal Ndousse, Cathy Yeh, and Kenneth O Stanley. Evolution through large models. In Handbook of Evolutionary Machine Learning, pp. 331–366. Springer, 2023. find
  28. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33: 9459–9474, 2020.
  29. Bryan Lim, Manon Flageat, and Antoine Cully. Large Language Models as In-context AI Generators for Quality-Diversity. arXiv preprint arXiv:2404.15794, 2024. — [arXiv:2404.15794](https://arxiv.org/abs/2404.15794) find
  30. Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv:2310.12931, 2023.
  31. Andrzej Maćkiewicz and Waldemar Ratajczak. Principal components analysis (PCA). Computers & Geosciences, 19(3):303–342, 1993. find
  32. Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. arXiv preprint arXiv:1504.04909, 2015. — [arXiv:1504.04909](https://arxiv.org/abs/1504.04909) find
  33. OpenAI. Text embedding 3 small. https://platform.openai.com/docs/guides/ embeddings/embedding-models, 2024. Accessed: 17 May 2024. ↗
  34. Pierre-Yves Oudeyer and Frederic Kaplan. What is intrinsic motivation? A typology of computational approaches. Frontiers in neurorobotics, 1:108, 2007. find
  35. Pierre-Yves Oudeyer, Frdric Kaplan, and Verena V Hafner. Intrinsic motivation systems for autonomous mental development. IEEE transactions on evolutionary computation, 11(2):265–286, 2007. find
  36. Jack Parker-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Evolving curricula with regret-based environment design. In International Conference on Machine Learning, pp. 17473–17498. PMLR, 2022. find
  37. Justin K Pugh, Lisa B Soros, and Kenneth O Stanley. Quality diversity: A new frontier for evolutionary computation. Frontiers in Robotics and AI, 3:202845, 2016. find
  38. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. find
  39. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748–8763. PMLR, 2021…
  40. Mikayel Samvelyan, Akbir Khan, Michael Dennis, Minqi Jiang, Jack Parker-Holder, Jakob Foerster, Roberta Raileanu, and Tim Rocktäschel. MAESTRO: Open-ended environment design for multi-agent reinforcement learning. arXiv preprint arXiv:2303.03376, 2023. — [arXiv:2303.03376](https://arxiv.org/abs/2303.03376) find
  41. Noor Shaker, Julian Togelius, and Mark J Nelson. Procedural content generation in games. 2016. find
  42. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of Go with deep neural networks and tree search. nature, 529(7587):484–489, 2016. find
  43. Kenneth O Stanley and Joel Lehman. Why greatness cannot be planned: The myth of the objective. Springer, 2015. find
  44. Kenneth O Stanley, Joel Lehman, and Lisa Soros. Open-endedness: The last grand challenge you’ve never heard of. While open-endedness could be a force for discovering intelligence, it could also be a component of AI itself, 2017. find
  45. Shyam Sudhakaran, Miguel González-Duque, Matthias Freiberger, Claire Glanois, Elias Najarro, and Sebastian Risi. Mariogpt: Open-ended text2level generation through large language models. Advances in Neural Information Processing Systems, 36, 2024. find
  46. Richard S Sutton. Reinforcement learning: An introduction. A Bradford Book, 2018. find
  47. Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS). find
  48. Graham Todd, Sam Earle, Muhammad Umair Nasir, Michael Cerny Green, and Julian Togelius. Level generation through large language models. In Proceedings of the 18th International Conference on the Foundations of Digital Games, pp. 1–8, 2023. find
  49. Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulao, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032, 2024.
  50. Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research, 9(11), 2008. find
  51. A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
  52. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023a. find
  53. Lirui Wang, Yiyang Ling, Zhecheng Yuan, Mohit Shridhar, Chen Bao, Yuzhe Qin, Bailin Wang, Huazhe Xu, and Xiaolong Wang. Gensim: Generating robotic simulation tasks via large language models. arXiv preprint arXiv:2310.01361, 2023b. — [arXiv:2310.01361](https://arxiv.org/abs/2310.01361) find
  54. Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley. Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint arXiv:1901.01753, 2019.
  55. Rui Wang, Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jeffrey Clune, and Kenneth Stanley. Enhanced poet: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. In International conference on machine learning, pp. 9940–9951. PMLR, 2020. find
  56. Yufei Wang, Zhou Xian, Feng Chen, Tsun-Hsuan Wang, Yian Wang, Katerina Fragkiadaki, Zackory Erickson, David Held, and Chuang Gan. Robogen: Towards unleashing infinite data for automated robot learning via generative simulation. arXiv preprint arXiv:2311.01455, 2023c. — [arXiv:2311.01455](https://arxiv.org/abs/2311.01455) find
  57. Abhay Zala, Jaemin Cho, Han Lin, Jaehong Yoon, and Mohit Bansal. EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agents. arXiv preprint arXiv:2403.12014, 2024. find
  58. Jenny Zhang, Joel Lehman, Kenneth Stanley, and Jeff Clune. Omni: Open-endedness via models of human notions of interestingness. arXiv preprint arXiv:2306.01711, 2023. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/faldor-2024-omni-epic-open-endedness-models/