Elamur Research⌕
☰
home / papers / 2025 / AlphaGo Moment for Model Architecture Discovery

AlphaGo Moment for Model Architecture Discovery

Paper · 2025 · Yixiu Liu, Yang Nan, Weixian Xu et al.

Summary

The paper addresses a bottleneck in AI progress: new model architectures are still invented through slow, human-led research. The authors present ASI-Arch, an autonomous multi-agent system. Classic Neural Architecture Search only selects among building blocks people have already defined. ASI-Arch instead tries to invent new designs. A Researcher agent proposes an architecture and writes it as code. An Engineer agent trains and evaluates it in a real environment. An Analyst agent interprets the outcomes. A Cognition module adds knowledge drawn from earlier papers, and a shared database of results steers each next iteration. Starting from DeltaNet, the system explored linear attention variants and reported 106 designs that outperform human-made baselines. The authors also find that the number of such discoveries rises roughly in proportion to compute, which they call a scaling law for scientific discovery. If this holds, architecture research could be scaled with hardware rather than expert time.

Key points

  • Scale of the run: 1,773 autonomous experiments over roughly 20,000 GPU hours produced 106 linear attention architectures that the authors label SOTA.
  • Framework structure: Four modules (Researcher, Engineer, Analyst, Cognition) form a closed evolutionary loop. Each proposal is checked for novelty and validity before training.
  • Search tree: The exploration tree is rooted at DeltaNet, and each node is scored by performance.
  • Scaling claim: The cumulative count of SOTA architectures grows linearly with GPU hours. The authors contrast this with an estimated ~2,000 human hours per hand-designed model.
  • Open release: The authors release the framework, the discovered architectures and the agents' reasoning traces as open source, through the SII-GAIR/ASI-Arch project.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Shanghai Jiao Tong UniversityShanghai · China232 works · 20,305 citations🎓Shanghai Innovation InstituteShanghai · China35 works · 333 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksComputer Vision and Pattern Recognition · 3,621 works

Cite

Cite as Yixiu Liu, Yang Nan, Weixian Xu, Xiangkun Hu, Lyumanshan Ye, Zhen Qin and Pengfei Liu. 2025. AlphaGo Moment for Model Architecture Discovery. arXiv:2507.18074.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2507.18074 · arXiv 2507.18074

@misc{liu2025alphago,
  title = {{AlphaGo Moment for Model Architecture Discovery}},
  author = {Yixiu Liu, Yang Nan, Weixian Xu, Xiangkun Hu, Lyumanshan Ye, Zhen Qin and Pengfei Liu},
  year = {2025},
  doi = {10.48550/arxiv.2507.18074},
  eprint = {2507.18074},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2507.18074}
}

References

44 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ajay Agrawal, Joshua Gans, and Avi Goldfarb. 2018. Prediction Machines: The Simple Economics of Artificial Intelligence. Harvard Business Press. find
  2. N’Daye Ahmed, Maliha Wahed, and N. C. Thompson. 2022. Modeling the ai-research ecosystem: A study of the circulation of scientific knowledge and talent. Research Policy, 51(5):104505. find
  3. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. 2016. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565.
  4. Eric B. Baum. 2004. What is Thought? The MIT Press. find
  5. Daniil A Boiko, Robert MacKnight, Gabe Gomes, and Adam Funke. 2023. Autonomous chemical research with large language models. Nature, 624(7992):570–576.
  6. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
  7. Erik Brynjolfsson and Tom Mitchell. 2017. What can machine learning do? workforce implications. Science, 358(6370):1530–1534. find
  8. Shidong Chen, Zhaofei Li, Boyu Du, and Hu Li. 2023. Llmatic: A generative llm for neural architecture search. arXiv preprint arXiv:2312.01633. — [arXiv:2312.01633](https://arxiv.org/abs/2312.01633) find
  9. Junyan Cheng, Peter Clark, and Kyle Richardson. 2025. Language modeling by language models. arXiv preprint arXiv:2506.20249.
  10. Yuri Chervonyi, Trieu H. Trinh, Miroslav Olšák, Xiaomeng Yang, Hoang Nguyen, Marcelo Menegali, Junehyuk Jung, Vikas Verma, Quoc V. Le, and Thang Luong. 2025. Gold-medalist performance in solving olympiad geometry with alphageometry2.
  11. Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. 2020. Rethinking attention with performers. arXiv preprint arXiv:2009.14794.
  12. Tri Dao and Albert Gu. 2024. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060. find
  13. DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, H… find
  14. Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. 2019. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1–21. find
  15. Albert Gu and Tri Dao. 2023. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752. find
  16. Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. 2020. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pages 5156–5165. PMLR. find
  17. D. Kokotajlo, S. Alexander, T. Larsen, E. Lifland, and R. Dean. 2025. Ai 2027. find
  18. Yann LeCun, Yoshua Bengio, et al. 1995. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361(10):1995. find
  19. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Pogodin, Oriol Vinyals, et al. 2022. Competition-level code generation with alphacode. Science, 378(6624):1092–1097.
  20. Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. 2024. Jamba: A hybrid transformer-mamba language model. arXiv preprint arXiv:2403.19887.
  21. MiniMax, :, Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, Chengjun Xiao, Chengyu Du, Chi Zhang, Chu Qiao, Chunhao Zhang, Chunhui Du, Congchao Guo, Da Chen, Deming Ding, Dianjun Sun, Dong Li, Enwei Jiao, Haigang Zhou, Haimo Zhang, H… — MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention
  22. Alexander Novikov, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco JR Ruiz, Abbas Mehrabian, et al. 2025. Alphaevolve: A coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.13131.
  23. OpenAI. 2023. Gpt-4 technical report. techreport arXiv:2303.08774, OpenAI.
  24. Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. 2023. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048. find
  25. Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong. 2024a. Lightning attention-2: A free lunch for handling unlimited sequence lengths in large language models. arXiv preprint arXiv:2401.04658.
  26. Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong. 2022. cosformer: Rethinking softmax in attention. arXiv preprint arXiv:2202.08791. — [arXiv:2202.08791](https://arxiv.org/abs/2202.08791) find
  27. Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, and Yiran Zhong. 2024b. Hgrn2: Gated linear rnns with state expansion. arXiv preprint arXiv:2404.07904. — [arXiv:2404.07904](https://arxiv.org/abs/2404.07904) find
  28. Zhen Qin, Songlin Yang, and Yiran Zhong. 2023. Hierarchically gated recurrent neural network for sequence modeling. Advances in Neural Information Processing Systems, 36:33202–33221.
  29. Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka L Suematsu, Jie Tan, Quoc V Le, and Alex Kurakin. 2017. Large-scale evolution of image classifiers. In International conference on machine learning, pages 2902–2911. PMLR. find
  30. Stuart J. Russell and Peter Norvig. 2010. Artificial intelligence: a modern approach. Prentice Hall. find
  31. Jürgen Schmidhuber. 1997. A computer scientist’s view of life, the universe, and everything. Lecture Notes in Computer Science, 1337:201–208. find
  32. Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, and Pablo Villalobos. 2022. Compute trends across three eras of machine learning. arXiv preprint arXiv:2202.05924. — [arXiv:2202.05924](https://arxiv.org/abs/2202.05924) find
  33. Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2022. Efficient transformers: A survey. ACM Computing Surveys (CSUR), 55(6):1–28.
  34. The White House. 2023. Ai talent: A report on the workforce needs for a booming artificial intelligence industry. Technical report, The White House Office of Science and Technology Policy. find
  35. Trieu H. Trinh, Yuhuai Wu, Quoc V. Le, He He, and Thang Luong. 2024. Solving olympiad geometry without human demonstrations. Nature, 625(7995):476–482.
  36. Vahe Tshitoyan, John Dagdelen, Leigh Weston, Alexander Dunn, Ziqin Rong, Olga Kononova, Kristin A Persson, Gerbrand Ceder, and Anubhav Jain. 2019. Unsupervised word embeddings capture latent knowledge from materials science literature. Nature, 571(7763):95–98. find
  37. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  38. Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768. find
  39. Songlin Yang, Jan Kautz, and Ali Hatamizadeh. 2024a. Gated delta networks: Improving mamba2 with delta rule. arXiv preprint arXiv:2412.06464. find
  40. Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. 2024b. Parallelizing linear transformers with the delta rule over sequence length. Advances in neural information processing systems, 37:115491–115522. find
  41. Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y. X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. 2025. Native sparse attention: Hardware-aligned and natively trainable sparse attention.
  42. Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. 2025. Darwin godel machine: Open-ended evolution of self-improving agents. ArXiv, abs/2505.22954.
  43. Ruocheng Zhang, Jiaxin Li, Zhaoning Liu, James A. Evans, Jeff Clune, and Diyi Ho. 2024. Large language models for science: A study on the state of the art. arXiv preprint arXiv:2402.16912. — [arXiv:2402.16912](https://arxiv.org/abs/2402.16912) find
  44. Barret Zoph and Quoc V Le. 2016. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2025-alphago-moment-model-architecture/