Elamur Research⌕
☰
home / papers / 2024 / Open-Endedness is Essential for Artificial Superhuman Intelligence

Open-Endedness is Essential for Artificial Superhuman Intelligence

Paper · 2024 · Edward Hughes, Michael Dennis, Jack Parker-Holder et al.

Summary

This position paper from Google DeepMind (ICML 2024) asks what an AI system would need in order to keep generating new knowledge by itself. It argues that this capacity, open-endedness, is a prerequisite for artificial superhuman intelligence (ASI). No dataset, model or benchmark is released, so the contribution is conceptual. The authors propose a formal definition that depends on the observer. A system counts as open-ended for a given observer if its stream of artifacts keeps producing items that the observer's predictive model fails to anticipate (novelty), while a longer history still improves those predictions (learnability). The authors then argue that foundation models make practical open-ended systems feasible, because they encode human ideas of what is interesting and can serve as general-purpose generators of variations. They add that current foundation models are not yet open-ended. Since high-quality static web data is limited, they expect progress toward ASI to depend on systems that create and refine their own knowledge. The paper also discusses the safety and societal risks this raises.

Key points

  • Formalism: a system S emits artifacts X_t, and an observer O predicts future artifacts with a model X̂_t scored by a loss ℓ(t, T).
    • Novelty: for any fixed t, some later artifact always has higher expected loss.
    • Learnability: for a fixed target, expected loss falls as more history is observed.
  • Illustrations:
    • A noisy TV with uniform noise is learnable but stops being novel once the model converges.
    • A TV whose channel is switched at random is novel but not learnable.
    • A figure shows that aircraft designs look different to a mouse, an aerospace student and a superintelligent alien.
  • Prior definitions: the definition generalises the four necessary conditions of Soros and Stanley (2014) without requiring an evolutionary process. It overlaps substantially with Sigaud et al. (2023).
  • Motivation and directions: the authors cite possible exhaustion of training data (Villalobos et al., 2022). Earlier open-ended successes they mention include Go, adaptation to 3D tasks (Bauer et al., 2023), the Minecraft tech tree (Wang et al., 2023a) and new mathematical results (Romera-Paredes et al., 2024). Proposed research directions pair foundation models with evolutionary algorithms and reinforcement learning.
  • Safety (Section 4):
    • Humans must stay able to understand the generated artifacts, which makes explainability central.
    • Society needs to prepare for feedback-loop tipping points such as flash crashes.
    • Open-ended search should be steered toward artifacts that are useful, not merely safe.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksSocial Sciences · 2,009 worksSafety Research · 665 works

Cite

Cite as Edward Hughes, Michael Dennis, Jack Parker-Holder, Feryal Behbahani, Aditi Mavalankar, Yuge Shi, Tom Schaul and Tim Rocktaschel. 2024. Open-Endedness is Essential for Artificial Superhuman Intelligence. arXiv:2406.04268.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2406.04268 · arXiv 2406.04268

@misc{hughes2024open,
  title = {{Open-Endedness is Essential for Artificial Superhuman Intelligence}},
  author = {Edward Hughes, Michael Dennis, Jack Parker-Holder, Feryal Behbahani, Aditi Mavalankar, Yuge Shi, Tom Schaul and Tim Rocktaschel},
  year = {2024},
  doi = {10.48550/arxiv.2406.04268},
  eprint = {2406.04268},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.04268}
}

References

141 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. D. Abel, A. Barreto, B. Van Roy, D. Precup, H. van Hasselt, and S. Singh. A definition of continual reinforcement learning. ArXiv preprint, abs/2307.11046, 2023. URL https://arxiv.org/abs/2307.11046. — [arXiv:2307.11046](https://arxiv.org/abs/2307.11046) ↗
  2. M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Y. Lu, L. Luu,… — Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
  3. K. Baumli, S. Baveja, F. Behbahani, H. Chan, G. Comanici, S. Flennerhag, M. Gazeau, K. Holsheimer, D. Horgan, M. Laskin, et al. Vision-language models as a source of rewards. ArXiv preprint, abs/2312.09187, 2023. URL https://arxiv.org/abs/2312.09187. — [arXiv:2312.09187](https://arxiv.org/abs/2312.09187) ↗
  4. M. Bedau. Measurement of evolutionary activity, teleology, and life. 1992. find
  5. E. M. Aldrich, J. Grundfest, and G. Laughlin. The flash crash: A new deconstruction. Available at SSRN 2721922, 2017. find
  6. M. A. Bedau, E. Snyder, C. T. Brown, N. H. Packard, et al. A comparison of evolutionary activity in artificial evolving D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schul-systems and in the biosphere. In Proceedings of the fourth European conference on artificial life, pages 125–134. MIt Pre… find
  7. N. Heess, L. Gonzalez, S. Osindero, S. Ozair, S. Reed, J. Zhang, K. Zolna, J. Clune, N. de Freitas, S. Singh, and T. Rocktäschel. Genie: Generative Interactive Environments, Feb. 2024. find
  8. M. A. Bedau, E. Snyder, and N. H. Packard. A classification of long-term evolutionary dynamics. Artificial Life: The Proceedings..., page 228, 1998. find
  9. Y. Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by Random Network Distillation, Oct. 2018.
  10. M. C. Campi and S. Garatti. Compression, generalization and learning. ArXiv preprint, abs/2301.12767, 2023. URL https://arxiv.org/abs/2301.12767. — [arXiv:2301.12767](https://arxiv.org/abs/2301.12767) ↗
  11. M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016. find
  12. R. Carey and T. Everitt. Human control: Definitions and algorithms. ArXiv preprint, abs/2305.19861, 2023. URL https://arxiv.org/abs/2305.19861. — [arXiv:2305.19861](https://arxiv.org/abs/2305.19861) ↗
  13. C. Berner, G. Brockman, B. Chan, V. Cheung, P. Debiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, C. Hesse, et al. Dota 2 with large scale deep reinforcement learning. ArXiv preprint, abs/1912.06680, 2019. URL https://arxiv.org/abs/1912.06680. ↗
  14. H. Chan, V. Mnih, F. Behbahani, M. Laskin, L. Wang, F. Pardo, M. Gazeau, H. Sahni, D. Horgan, K. Baumli, Y. Schroecker, S. Spencer, R. Steigerwald, J. Quan, G. Comanici, S. Flennerhag, A. Neitz, L. M. Zhang, T. Schaul, S. Singh, C. Lyle, T. Rocktäschel, J. Parker-Holder, and K. Holsheimer. Vision-l… find
  15. S. Bills, N. Cammarata, D. Mossing, H. Tillman, L. Gao, G. Goh, I. Sutskever, J. Leike, J. Wu, and W. Saunders. Language models can explain neurons in language models. URL https://openaipublic.blob. core. windows. net/neuron-explainer/paper/index. html.(Date accessed: 14.05. 2023), 2023. ↗
  16. A. Chen, D. M. Dohan, and D. R. So. EvoPrompting: Language Models for Code-Level Neural Architecture Search, Feb. 2023a. find
  17. R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
  18. X. Chen, M. Lin, N. Schärli, and D. Zhou. Teaching Large Language Models to Self-Debug, Apr. 2023b. find
  19. P. Christiano, A. Cotra, and M. Xu. Eliciting latent knowledge: How to tell if your eyes deceive you, 2021. find
  20. S. R. Bowman, J. Hyun, E. Perez, E. Chen, C. Pettit, S. Heiner, K. Lukošiūtė, A. Askell, A. Jones, A. Chen, et al. Measuring progress on scalable oversight for large language models. ArXiv preprint, abs/2211.03540, 2022. URL https://arxiv.org/abs/2211.03540. ↗
  21. J. Clark and D. Amodei. Faulty reward functions in the wild. Internet: https://blog.openai. com/faulty-reward-functions, 2016. ↗
  22. J. Clune. AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence, Jan. 2020. find
  23. H. Bradley, A. Dai, H. Teufel, J. Zhang, K. Oostermeijer, M. Bellagente, J. Clune, K. Stanley, G. Schott, and J. Lehman. Quality-Diversity through AI Feedback, Oct. 2023.
  24. J. Clune. Ai will go farterh if it stands on the shoulders of giant human data sets. Dec. 2022. find
  25. J. C. Brant and K. O. Stanley. Minimal criterion coevolution: a new approach to open-ended search. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 67–74, 2017. find
  26. C. Colas, T. Karch, O. Sigaud, and P.-Y. Oudeyer. Autotelic agents with intrinsically motivated goal-conditioned reinforcement learning: a short survey. Journal of Artificial Intelligence Research, 74:1159–1199, 2022. find
  27. T. Brooks, B. Peebles, C. Holmes, W. DePue, Y. Guo, L. Jing, D. Schnurr, J. Taylor, T. Luhman, E. Luh-D. Collingridge. The Social Control of Technology. man, C. Ng, R. Wang, and A. Ramesh. Video St. Martin’s Press, 1980. ISBN 9780312731687. generation models as world simulators. 2024. URL https://bo… — [arXiv:2006.04948](https://arxiv.org/abs/2006.04948) find
  28. A. Dafoe, E. Hughes, Y. Bachrach, T. Collins, K. R. McKee, J. Z. Leibo, K. Larson, and T. Graepel. Open Problems in Cooperative AI, Dec. 2020. find
  29. Y. Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch. Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325, 2023c.
  30. O. David, S. Moran, and A. Yehudayoff. On statistical learning via the lens of compression. ArXiv preprint, abs/1610.03592, 2016. URL https://arxiv.org/ abs/1610.03592. — [arXiv:1610.03592](https://arxiv.org/abs/1610.03592) ↗
  31. S. Earle, J. Togelius, and L. B. Soros. Video games as a testbed for open-ended phenomena. In 2021 IEEE Conference on Games (CoG), pages 1–9. find
  32. G. Delétang, A. Ruoss, P.-A. Duquenne, E. Catt, T. Genewein, C. Mattern, J. Grau-Moya, L. K. Wenliang, M. Aitchison, L. Orseau, M. Hutter, and J. Veness. Language Modeling Is Compression, Sept. 2023.
  33. A. Ecoffet, J. Clune, and J. Lehman. Open Questions in Creating Safe Open-ended AI: Tensions Between Control and Creativity, June 2020. find
  34. M. Faldor, J. Zhang, A. Cully, and J. Clune. Omni-epic: Open-endedness via models of human notions of interestingness with environments programmed in code. arXiv preprint arXiv:2405.15568, 2024.
  35. M. Dennis, N. Jaques, E. Vinitsky, A. M. Bayen, S. Russell, A. Critch, and S. Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual… find
  36. C. Fernando, D. Banarse, H. Michalewski, S. Osindero, and T. Rocktäschel. Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution, Sept. 2023.
  37. K. Gandhi, D. Sadigh, and N. D. Goodman. Strategic Reasoning with Language Models, May 2023.
  38. J. Derbyshire. Potential surprise theory as a theoretical foundation for scenario planning. Technological Forecasting and Social Change, 124:77–87, 2017. find
  39. J. Garcıa and F. Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437–1480, 2015. find
  40. D. Deutsch. The beginning of infinity: Explanations that transform the world. Penguin UK, 2011. find
  41. L. L. di Langosco, J. Koch, L. D. Sharkey, J. Pfau, and D. Krueger. Goal misgeneralization in deep reinforcement learning. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryla… find
  42. PMLR, 2022. URL https://proceedings.mlr.press/v162/langosco22a.html. ↗
  43. Z. Gou, Z. Shao, Y. Gong, Y. Shen, Y. Yang, N. Duan, and W. Chen. Critic: Large language models can self-correct with tool-interactive critiquing. ArXiv preprint, abs/2305.11738, 2023. URL https://arxiv.org/ abs/2305.11738. ↗
  44. Q. Guo, R. Wang, J. Guo, B. Li, K. Song, X. Tan, G. Liu, J. Bian, and Y. Yang. Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers, Sept. 2023. find
  45. I. Gur, N. Jaques, Y. Miao, J. Choi, M. Tiwari, H. Lee, and A. Faust. Environment generation for zero-shot compositional reinforcement learning. Advances in Neural Information Processing Systems, 34:4157–4169, 2021. find
  46. E. L. Dolson, A. E. Vostinar, M. J. Wiser, and C. Ofria. The modes toolbox: Measurements of open-ended dynamics in evolving systems. Artificial life, 25(1):50–73, 2019. find
  47. Y. Du, K. Konyushkova, M. Denil, A. Raju, J. Landon, F. Hill, N. de Freitas, and S. Cabi. Vision-language models as success detectors. In Proceedings of The 2nd Conference on Lifelong Learning Agents, pages 120–136, 2023a. find
  48. W. Gurnee and M. Tegmark. Language Models Represent Space and Time, Oct. 2023. find
  49. D. Hadfield-Menell, S. J. Russell, P. Abbeel, and A. D. Dragan. Cooperative inverse reinforcement learning. In D. D. Lee, M. Sugiyama, U. von Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2… find
  50. Y. Du, E. Kosoy, A. Dayan, M. Rufova, P. Abbeel, and A. Gopnik. What can ai learn from human exploration? intrinsically-motivated humans and agents in open-world exploration. In NeurIPS 2023 workshop: Information-Theoretic Principles in Cognitive Systems, 2023b. find
  51. D. Hadfield-Menell, A. D. Dragan, P. Abbeel, and S. J. Russell. The off-switch game. In C. Sierra, editor, Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pages 220–227. ijcai.org, 2017. doi: 10.24963/ij… — [doi:10.24963/ijcai.2017/32](https://doi.org/10.24963/ijcai.2017/32) find
  52. M. Klissarov, P. D’Oro, S. Sodhani, R. Raileanu, P.-L. Bacon, P. Vincent, A. Zhang, and M. Henaff. Motif: Intrinsic Motivation from Artificial Intelligence Feedback, Sept. 2023. find
  53. F. H. Knight. Risk, uncertainty and profit, volume 31. Houghton Mifflin, 1921. find
  54. E. Hazan and S. Kale. Extracting certainty from uncertainty: Regret bounded by variation in costs. Machine learning, 80:165–188, 2010. find
  55. V. Krakovna, L. Orseau, R. Kumar, M. Martic, and S. Legg. Penalizing side effects using stepwise relative reachability. ArXiv preprint, abs/1806.01186, 2018. URL https://arxiv.org/abs/1806.01186. — [arXiv:1806.01186](https://arxiv.org/abs/1806.01186) ↗
  56. M. Henaff, R. Raileanu, M. Jiang, and T. Rocktäschel. Exploration via Elliptical Episodic Bonuses, Jan. 2023. find
  57. S. Legg and M. Hutter. Universal Intelligence: A Definition of Machine Intelligence, Dec. 2007. find
  58. J. H. Holland. Adaptation in natural and artificial systems: an introductory analysis with applications to biology, control, and artificial intelligence. MIT press, 1992. find
  59. J. Lehman and K. O. Stanley. Abandoning Objectives: Evolution Through the Search for Novelty Alone. Evolutionary Computation, 19(2):189–223, June 2011. ISSN 1063-6560. doi: 10.1162/EVCO a 00025. — [doi:10.1162/evco](https://doi.org/10.1162/evco) find
  60. A. Hu, L. Russell, H. Yeo, Z. Murez, G. Fedoseev, A. Kendall, J. Shotton, and G. Corrado. GAIA-1: A Generative World Model for Autonomous Driving, Sept. 2023. find
  61. J. Lehman, J. Gordon, S. Jain, K. Ndousse, C. Yeh, and K. O. Stanley. Evolution through Large Models, June 2022. find
  62. J. Huang, S. S. Gu, L. Hou, Y. Wu, X. Wang, H. Yu, and J. Han. Large Language Models Can Self-Improve, Oct. 2022.
  63. J. Z. Leibo, E. Hughes, M. Lanctot, and T. Graepel. Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research, Mar. 2019. find
  64. S. Lifshitz, K. Paster, H. Chan, J. Ba, and S. McIlraith. STEVE-1: A Generative Model for Text-to-Behavior in Minecraft, June 2023. find
  65. P. Huang, X. Zhang, Z. Cao, S. Liu, M. Xu, W. Ding, J. Francis, B. Chen, and D. Zhao. What went wrong? closing the sim-to-real gap via differentiable causal discovery. In Conference on Robot Learning, pages 734–760. PMLR, 2023. find
  66. S. Liu, C. Chen, X. Qu, K. Tang, and Y.-S. Ong. Large language models as evolutionary optimizers. arXiv preprint arXiv:2310.19046, 2023a. — [arXiv:2310.19046](https://arxiv.org/abs/2310.19046) find
  67. M. Hutter. Universal artificial intelligence: Sequential decisions based on algorithmic probability. Springer Science & Business Media, 2004. find
  68. X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, S. Zhang, X. Deng, A. Zeng, Z. Du, C. Zhang, S. Shen, T. Zhang, Y. Su, H. Sun, M. Huang, Y. Dong, and J. Tang. AgentBench: Evaluating LLMs as Agents, Aug. 2023b.
  69. G. Irving, P. Christiano, and D. Amodei. AI safety via debate, Oct. 2018.
  70. M. Jiang, E. Grefenstette, and T. Rocktäschel. Prioritized Level Replay, June 2021. find
  71. Y. J. Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, D. Jayaraman, Y. Zhu, L. Fan, and A. Anandkumar. Eureka: Human-Level Reward Design via Coding Large Language Models, Oct. 2023.
  72. M. Jiang, T. Rocktäschel, and E. Grefenstette. General Intelligence Requires Rethinking Exploration, Nov. 2022. find
  73. A. N. Mavor-Parker, K. A. Young, C. Barry, and L. D. Griffin. How to stay curious while avoiding noisy tvs using aleatoric uncertainty estimation. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July… find
  74. PMLR, 2022. URL https://proceedings.mlr.press/ v162/mavor-parker22a.html. ↗
  75. M. B. Johanson, E. Hughes, F. Timbers, and J. Z. Leibo. Emergent bartering behaviour in multi-agent reinforcement learning. ArXiv preprint, abs/2205.06760, 2022. URL https://arxiv.org/abs/2205.06760. — [arXiv:2205.06760](https://arxiv.org/abs/2205.06760) ↗
  76. N. Justesen, R. R. Torrado, P. Bontrager, A. Khalifa, J. Togelius, and S. Risi. Illuminating generalization in deep reinforcement learning through procedural level generation. arXiv preprint arXiv:1806.10729, 2018. — [arXiv:1806.10729](https://arxiv.org/abs/1806.10729) find
  77. D. W. McShea. Perspective metazoan complexity and evolution: is there a trend? Evolution, 50(2):477–492, 1996. and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 174… find
  78. PMLR, 2022. URL https://proceedings.mlr.press/ v162/parker-holder22a.html. ↗
  79. E. Meyerson, M. J. Nelson, H. Bradley, A. Moradi, A. K. Hoover, and J. Lehman. Language Model Crossover: Variation through Few-Shot Prompting, Feb. 2023. find
  80. S. Mirchandani, F. Xia, P. Florence, B. Ichter, D. Driess, M. G. Arenas, K. Rao, D. Sadigh, and A. Zeng. Large Language Models as General Pattern Machines, July 2023.
  81. D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self-supervised prediction. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of… find
  82. PMLR, 2017. URL http://proceedings.mlr.press/ v70/pathak17a.html. ↗
  83. I. Momennejad, H. Hasanbeig, F. Vieira Frujeri, H. Sharma, N. Jojic, H. Palangi, R. Ness, and J. Larson. Evaluating cognitive maps and planning in large language models with cogeval. Advances in Neural Information Processing Systems, 36, 2024. find
  84. J. Perolat, B. De Vylder, D. Hennes, E. Tarassov, F. Strub, V. de Boer, P. Muller, J. T. Connor, N. Burch, T. Anthony, et al. Mastering the game of stratego with model-free multiagent reinforcement learning. Science, 378(6623): 990–996, 2022. find
  85. M. R. Morris, J. Sohl-dickstein, N. Fiedel, T. Warkentin, A. Dafoe, A. Faust, C. Farabet, and S. Legg. Levels of AGI: Operationalizing Progress on the Path to AGI, Nov. 2023. find
  86. J.-B. Mouret and J. Clune. Illuminating search spaces by mapping elites. ArXiv preprint, abs/1504.04909, 2015. URL https://arxiv.org/abs/1504.04909. — [arXiv:1504.04909](https://arxiv.org/abs/1504.04909) ↗
  87. J. K. Pugh, L. B. Soros, and K. O. Stanley. Quality diversity: A new frontier for evolutionary computation. Frontiers in Robotics and AI, 3:40, 2016. ISSN 2296-9144. doi: 10.3389/frobt.2016.00040. — [doi:10.3389/frobt.2016.00040](https://doi.org/10.3389/frobt.2016.00040) find
  88. OEL Team, A. Stooke, A. Mahajan, C. Barros, C. Deck, J. Bauer, J. Sygnowski, M. Trebacz, M. Jaderberg, M. Mathieu, N. McAleese, N. Bradley-Schmieg, N. Wong, N. Porcel, R. Raileanu, S. Hughes-Fitt, V. Dalibard, and W. M. Czarnecki. Open-ended learning leads to generally capable agents. ArXiv preprint… — [arXiv:2107.12808](https://arxiv.org/abs/2107.12808) find
  89. R. Raileanu and T. Rocktäschel. RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments, Feb. 2020. find
  90. P. J. Richerson, R. Boyd, et al. Institutional evolution in the holocene: the rise of complex societies. In Proceedings-British Academy, volume 110, pages 197–234. Oxford University Press Inc., 2001. find
  91. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback, Ma…
  92. B. Romera-Paredes, M. Barekatain, A. Novikov, M. Balog, M. P. Kumar, E. Dupont, F. J. R. Ruiz, J. S. Ellenberg, P. Wang, O. Fawzi, P. Kohli, and A. Fawzi. Mathematical discoveries from program search with large language models. Nature, 625(7995):468–475, Jan. 2024. ISSN 1476-4687. doi: 10.1038/s4158…
  93. V. Pallagani, B. Muppasani, K. Murugesan, F. Rossi, B. Srivastava, L. Horesh, F. Fabiano, and A. Loreggia. Understanding the capabilities of large language models for automated planning. arXiv preprint arXiv:2305.16151, 2023. — [arXiv:2305.16151](https://arxiv.org/abs/2305.16151) find
  94. A. L. Samuel. Some studies in machine learning using the game of checkers. IBM Journal of research and development, 3(3):210–229, 1959. find
  95. M. Samvelyan, A. Khan, M. Dennis, M. Jiang, J. Parker-Holder, J. Foerster, R. Raileanu, and T. Rocktäschel. MAESTRO: Open-Ended Environment Design for Multi-Agent Reinforcement Learning, Mar. 2023. find
  96. J. S. Park, J. C. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein. Generative Agents: Interactive Simulacra of Human Behavior, Apr. 2023. find
  97. M. Samvelyan, S. C. Raparthy, A. Lupu, E. Hambro, A. H. Markosyan, M. Bhatt, Y. Mao, M. Jiang, J. Parker-Holder, J. Foerster, T. Rocktäschel, and R. Raileanu. Rainbow teaming: Open-ended generation of diverse adversarial prompts, 2024. find
  98. J. Parker-Holder, M. Jiang, M. Dennis, M. Samvelyan, J. N. Foerster, E. Grefenstette, and T. Rocktäschel. Evolving curricula with regret-based environment design. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, W. Saunders, C. Yeh, J. Wu, S. Bills, L. Ouyang, J. Ward, and J. Leike. Se… — Self-critiquing models for assisting human evaluators
  99. I. Shumailov, Z. Shumaylov, Y. Zhao, Y. Gal, N. Papernot, and R. Anderson. Model dementia: Generated data makes models forget. arXiv e-prints, pages arXiv–2305, 2023. find
  100. P. Shyam, W. Jaskowski, and F. Gomez. Model-based active exploration. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pa… find
  101. PMLR, 2019. URL http://proceedings.mlr.press/ v97/shyam19a.html. ↗
  102. T. Schaul, D. Borsa, D. Ding, D. Szepesvari, G. Ostrovski, W. Dabney, and S. Osindero. Adapting behaviour for learning progress. arXiv preprint arXiv:1912.06910, 2019. — [arXiv:1912.06910](https://arxiv.org/abs/1912.06910) find
  103. T. Schick, J. Dwivedi-Yu, R. Dessı̀, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36, 2024.
  104. O. Sigaud, G. Baldassarre, C. Colas, S. Doncieux, R. Duro, N. Perrin-Gilbert, and V.-G. Santucci. A definition of open-ended learning problems for goal-conditioned agents. ArXiv preprint, abs/2311.00344, 2023. URL https://arxiv.org/abs/2311.00344. — [arXiv:2311.00344](https://arxiv.org/abs/2311.00344) ↗
  105. J. Schmidhuber. Adaptive confidence and adaptive curiosity. Inst. für Informatik, 1991a. find
  106. D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis. Mastering the game of Go w… find
  107. J. Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural controllers. In Proc. of the international conference on simulation of adaptive behavior: From animals to animats, pages 222– 227, 1991b. find
  108. J. Secretan, N. Beato, D. B. D Ambrosio, A. Rodriguez, A. Campbell, and K. O. Stanley. Picbreeder: Evolving pictures collaboratively online. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’08, pages 1759–1768, New York, NY, USA, Apr. 2008. Association for Computin… — [doi:10.1145/1357054.1357328](https://doi.org/10.1145/1357054.1357328) find
  109. D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, and D. Hassabis. Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm, Dec. 2017.
  110. G. Shackle. Expectation in Economics. Cambridge University Press, 1949. ISBN 9781107629141. URL https://books.google.co.uk/books? id=zEb47udAsOcC. ↗
  111. R. J. Solomonoff. A preliminary report on a general theory of inductive inference. Citeseer, 1960. find
  112. L. Soros and K. Stanley. Identifying necessary conditions for open-ended evolution through the artificial life world of chromaria. In ALIFE 14: The Fourteenth International Conference on the Synthesis and Simulation of Living Systems, ALIFE 2023: Ghost in the Machine: Proceedings of the 2023 Artific… — [doi:10.1162/978-0-262-32621-6-ch128](https://doi.org/10.1162/978-0-262-32621-6-ch128) find
  113. R. Shah, V. Varma, R. Kumar, M. Phuong, V. Krakovna, J. Uesato, and Z. Kenton. Goal misgeneralization: Why correct specifications aren’t enough for correct goals. ArXiv preprint, abs/2210.01790, 2022. URL https://arxiv.org/abs/2210.01790. — [arXiv:2210.01790](https://arxiv.org/abs/2210.01790) ↗
  114. A. Sharma, D. Czégel, M. Lachmann, C. P. Kempes, S. I. Walker, and L. Cronin. Assembly theory explains and quantifies selection and evolution. Nature, 622(7982): 321–328, Oct. 2023. ISSN 1476-4687. doi: 10.1038/ s41586-023-06600-9. find
  115. K. Stanley and J. Lehman. Why Greatness Cannot Be Planned: The Myth of the Objective. Springer International Publishing, 2015. ISBN 9783319155241. URL https://books.google.co.uk/books? id=Llb1CAAAQBAJ. ↗
  116. M. Shin, J. Kim, B. van Opheusden, and T. L. Griffiths. Superhuman Artificial Intelligence Can Improve Human Decision Making by Increasing Novelty. Proceedings of the National Academy of Sciences, 120(12):e2214840120, Mar. 2023. ISSN 0027-8424, 1091-6490. doi: 10.1073/ pnas.2214840120. find
  117. K. O. Stanley. Why open-endedness matters. Artificial Life, 25(3):232–235, 2019. ISSN 1064-5462. doi: 10.1162/ artl a 00294. URL https://doi.org/10.1162/ artl_a_00294. find
  118. K. O. Stanley and L. Soros. The role of subjectivity in the evaluation of open-endedness. In Presentation delivered in OEE2: The Second Workshop on Open-Ended Evolution, at ALIFE 2016, 2016. find
  119. T. Lillicrap, K. Kavukcuoglu, D. Hassabis, C. Apps, and D. Silver. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575 (7782):350–354, Nov. 2019. ISSN 1476-4687. doi: 10.1038/s41586-019-1724-z. find
  120. K. O. Stanley, J. Lehman, and L. Soros. Open-endedness: The last grand challenge you’ve never heard of. While open-endedness could be a force for discovering intelligence, it could also be a component of AI itself, 2017. find
  121. L. S. Vygotsky and M. Cole. Mind in society: Development of higher psychological processes. Harvard university press, 1978. find
  122. S. Stepney and S. Hickinbotham. On the open-endedness of detecting open-endedness. Artificial Life, pages 1–26, 2023. find
  123. C. H. Waddington. Paradigm for an evolutionary process. Biological Theory, 3:258–266, 2008. find
  124. R. J. Sternberg and J. E. Davidson. The nature of insight. The MIT Press, 1995. find
  125. G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar. Voyager: An Open-Ended Embodied Agent with Large Language Models, May 2023a. find
  126. N. N. Taleb. Antifragile: Things that gain from disorder, volume 3. Random House Trade Paperbacks, 2014. find
  127. R. Wang, J. Lehman, J. Clune, and K. O. Stanley. Paired open-ended trailblazer (POET): endlessly generating increasingly complex and diverse learning environments and their solutions. ArXiv preprint, abs/1901.01753, 2019. URL https://arxiv.org/abs/1901.01753. ↗
  128. X. Tang, A. Zou, Z. Zhang, Y. Zhao, X. Zhang, A. Cohan, and M. Gerstein. Medagents: Large language models as collaborators for zero-shot medical reasoning. arXiv preprint arXiv:2311.10537, 2023.
  129. T. Taylor. Requirements for open-ended evolution in natural and artificial systems. arXiv preprint arXiv:1507.07403, 2015. — [arXiv:1507.07403](https://arxiv.org/abs/1507.07403) find
  130. R. Wang, J. Lehman, A. Rawal, J. Zhi, Y. Li, J. Clune, and K. O. Stanley. Enhanced POET: open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virt… find
  131. PMLR, 2020. URL http://proceedings.mlr.press/v119/wang20l.html. ↗
  132. T. Taylor. Routes to open-endedness in evolutionary systems. arXiv preprint arXiv:1806.01883, 2018. — [arXiv:1806.01883](https://arxiv.org/abs/1806.01883) find
  133. A. M. Turner, D. Hadfield-Menell, and P. Tadepalli. Conservative agency via attainable utility preservation. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pages 385–391, 2020. find
  134. T. T. Wang, A. Gleave, T. Tseng, K. Pelrine, N. Belrose, J. Miller, M. D. Dennis, Y. Duan, V. Pogrebniak, S. Levine, et al. Adversarial policies beat superhuman go ais. In International Conference on Machine Learning, pages 35655–35739. PMLR, 2023b. find
  135. F. Urbina, F. Lentzos, C. Invernizzi, and S. Ekins. Dual use of artificial-intelligence-powered drug discovery. Nature Machine Intelligence, 4(3):189–191, 2022. find
  136. K. Valmeekam, M. Marquez, S. Sreedharan, and S. Kambhampati. On the planning abilities of large language models-a critical investigation. Advances in Neural Information Processing Systems, 36:75993–76005, 2023. — On the Planning Abilities of Large Language Models : A Critical Investigation
  137. Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi. Self-instruct: Aligning language model with self generated instructions. ArXiv preprint, abs/2212.10560, 2022. URL https://arxiv.org/abs/2212.10560. ↗
  138. P. Villalobos, J. Sevilla, L. Heim, T. Besiroglu, M. Hobbhahn, and A. Ho. Will we run out of data? An analysis of the limits of scaling datasets in Machine Learning, Oct. 2022. find
  139. Z. Wang, S. Cai, A. Liu, Y. Jin, J. Hou, B. Zhang, H. Lin, Z. He, Z. Zheng, Y. Yang, X. Ma, and Y. Liang. JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models, Nov. 2023c.
  140. O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, J. Oh, D. Horgan, M. Kroiss, I. Danihelka, A. Huang, L. Sifre, T. Cai, J. P. Agapiou, M. Jaderberg, A. S. Vezhnevets, R. Leblond, T. Pohlen, V. Dalibard, D. Budden, Y. Sulsky,… find
  141. A. Illustrating Open-Endedness M. L. Wong, C. E. Cleland, D. Arend Jr, S. Bartlett, H. J. Cleaves, H. Demarest, A. Prabhu, J. I. Lunine, and R. M. Hazen. On the roles of function and selection in evolving systems. Proceedings of the National Academy of Sciences, 120(43):e2310223120, 2023b. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hughes-2024-open-endedness-essential-artificial-superhuman/