Elamur Research⌕
☰
home / papers / 2025 / Language Modeling by Language Models

Language Modeling by Language Models

Paper · 2025 · Junyan Cheng, Peter Clark and Kyle Richardson

Summary

The paper asks whether LLM agents can carry out the whole research cycle for inventing new language model architectures. Unlike open-ended automated science, this task has a concrete target (runnable code) and measurable success. The authors build LMADE, an environment with two parts. A knowledge engine offers a curated library of LM papers plus literature and web search tools. A verification engine checks code validity, pre-trains candidates on a filtered SmolLM corpus and scores them on LM-Eval tasks. On top of it runs Genesys, a multi-agent system. Designer agents draft and critique proposals and write code as generalized autoregressive blocks (GAB), which break down into trees and allow genetic-programming-style evolution. Verifier agents choose which designs get trained. A Ladder of Scales spends a shrinking training budget on progressively larger models (14M–350M). Over a thousand designs were fully pre-trained, and the strongest ones rival human-made baselines such as GPT and Mamba2. The work shows that LLM-driven discovery can produce competitive results in a costly, verifiable ML domain. The system and its artifacts are publicly released.

Key points

  • Scale of the experiment: 1,162 new designs were generated and 1,062 were fully verified through pre-training. The run involved over 1B tokens, about 2.76M lines of code and roughly 86K agent interactions.
  • The genetic programming backbone over GAB trees raised the rate of successful design generation by about 86 percentage points compared with direct prompting.
  • LMADE's reference library holds 297 LM papers with code in a vector store. It also connects to ArXiv, Semantic Scholar and Perplexity.ai. Designs are checked by a symbolic checker that tests causality, differentiability and training stability, then evaluated on 29 LM-Eval benchmarks.
  • Five showcased designs (VQH, HMamba, Geogate, HippoVQ, SRN) were compared with GPT, Mamba2, RWKV7, RetNet and TTT. They matched or beat the baselines on 7/9 benchmarks at 125M and 6/9 at 350M. At 350M, Geogate had the top average (61.81 vs 61.78 for GPT).
  • Resources: a live console at genesys.allen.ai and code at github.com/allenai/genesys. Stated limitations are the lack of hardware-level efficiency work (e.g., FlashAttention) and no discovery at the billion-parameter scale.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Allen Institute for Artificial IntelligenceSeattle · United States317 works · 106,633 citations🎓Dartmouth CollegeHanover · United States42 works · 42,728 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Junyan Cheng, Peter Clark and Kyle Richardson. 2025. Language Modeling by Language Models. arXiv:2506.20249.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2506.20249 · arXiv 2506.20249

@misc{cheng2025language,
  title = {{Language Modeling by Language Models}},
  author = {Junyan Cheng, Peter Clark and Kyle Richardson},
  year = {2025},
  doi = {10.48550/arxiv.2506.20249},
  eprint = {2506.20249},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2506.20249}
}

References

69 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., et al. Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL https://arxiv.org/abs/2404.14219. ↗
  2. Allal, L. B., Lozhkov, A., and Bakouch, E. Smollm - blazingly fast and remarkably powerful. Hugging Face Blogs, 2024. URL https://huggingface.co/blog/smollm. ↗
  3. Allal, L. B., Lozhkov, A., Bakouch, E., Blázquez, G. M., Penedo, G., Tunstall, L., Marafioti, A., Kydlíček, H., Lajarín, A. P., Srivastav, V., et al. Smollm2: When smol goes big–data-centric training of a small language model. arXiv preprint arXiv:2502.02737, 2025.
  4. Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M. K., Klambauer, G., Brandstetter, J., and Hochreiter, S. xLSTM: Extended long short-term memory. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=ARAxPPIAh… ↗
  5. Behrouz, A., Zhong, P., and Mirrokni, V. Titans: Learning to memorize at test time. arXiv preprint arXiv:2501.00663, 2024. find
  6. Bhirangi, R., Wang, C., Pattabiraman, V., Majidi, C., Gupta, A., Hellebrekers, T., and Pinto, L. Hierarchical state space models for continuous sequence-to-sequence modeling. arXiv preprint arXiv:2402.10211, 2024. — [arXiv:2402.10211](https://arxiv.org/abs/2402.10211) find
  7. Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397–2430. PMLR, 20… find
  8. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  9. Chen, A., Dohan, D., and So, D. Evoprompting: Language models for code-level neural architecture search. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 7787–7817. Curran Associates, Inc., 2023. find
  10. Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Pham, H., Dong, X., Luong, T., Hsieh, C.-J., Lu, Y., et al. Symbolic discovery of optimization algorithms. Advances in neural information processing systems, 36, 2024. find
  11. Cheng, J., Novati, G., Pan, J., Bycroft, C., Žemgulytė, A., Applebaum, T., Pritzel, A., Wong, L. H., Zielinski, M., Sargeant, T., et al. Accurate proteome-wide missense variant effect prediction with alphamissense. Science, 381(6664):eadg7492, 2023. find
  12. Chitty-Venkata, K. T., Emani, M., Vishwanath, V., and Somani, A. K. Neural architecture search for transformers: A survey. IEEE Access, 10:108374–108412, 2022. find
  13. Dao, T. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/ forum?id=mZn2Xyh9Ec. ↗
  14. Dao, T. and Gu, A. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. In International conference on machine learning. PMLR, 2024. find
  15. Eldan, R. and Li, Y. Tinystories: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.07759, 2023.
  16. Elsken, T., Metzen, J. H., and Hutter, F. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1–21, 2019. find
  17. Feng, L., Tung, F., Ahmed, M. O., Bengio, Y., and Hajimirsadegh, H. Were rnns all we needed? arXiv preprint arXiv:2410.01201, 2024. — [arXiv:2410.01201](https://arxiv.org/abs/2410.01201) find
  18. Fourrier, C., Habib, N., Lozovskaya, A., Szafer, K., and Wolf, T. Open llm leaderboard v2. https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard, 2024. ↗
  19. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. The pile: An 800gb dataset of diverse text for language modeling, 2020. URL https://arxiv.org/abs/2101.00027. ↗
  20. Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., et al. A framework for few-shot language model evaluation, 07 2024. URL https://zenodo.org/ records/12608602. ↗
  21. Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y., et al. Olmo: Accelerating the Science of Language Models. Proceedings of ACL, 2024. find
  22. Gu, A., Dao, T., Ermon, S., Rudra, A., and Ré, C. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33:1474–1487, 2020a. find
  23. Gu, A., Goel, K., and Ré, C. Efficiently modeling long sequences with structured state spaces. In The International Conference on Learning Representations (ICLR), 2022. find
  24. Gu, S., Kelly, B., and Xiu, D. Empirical asset pricing via machine learning. The Review of Financial Studies, 33(5):2223–2273, 2020b. find
  25. Hemberg, E., Moskal, S., and O’Reilly, U.-M. Evolving code with a large language model. Genetic Programming and Evolvable Machines, 25(2):21, 2024. find
  26. Hu, S., Tu, Y., Han, X., Cui, G., He, C., Zhao, W., Long, X., Zheng, Z., Fang, Y., Huang, Y., Zhang, X., Thai, Z. L., Wang, C., Yao, Y., Zhao, C., Zhou, J., Cai, J., Zhai, Z., Ding, N., Jia, C., Zeng, G., dahai li, Liu, Z., and Sun, M. MiniCPM: Unveiling the potential of small language models with s… — MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
  27. Jansen, P., Tafjord, O., Radensky, M., Siangliulue, P., Hope, T., Mishra, B. D., Majumder, B. P., Weld, D. S., and Clark, P. Codescientist: End-to-end semi-automated scientific discovery with code-based experimentation, 2025a. URL https://arxiv.org/abs/2503.22708. ↗
  28. Jansen, P., Tafjord, O., Radensky, M., Siangliulue, P., Hope, T., Mishra, B. D., Majumder, B. P., Weld, D. S., and Clark, P. Codescientist: End-to-end semi-automated scientific discovery with code-based experimentation. arXiv preprint arXiv:2503.22708, 2025b.
  29. Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. Highly accurate protein structure prediction with alphafold. nature, 596(7873):583–589, 2021.
  30. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
  31. Kidger, P. torchtyping: Type annotations and runtime type checking of tensor shapes (and dtypes, ...). https://github.com/patrick-kidger/torchtyping, 2021. ↗
  32. Kinney, R., Anastasiades, C., Authur, R., Beltagy, I., Bragg, J., Buraczynski, A., Cachola, I., Candra, S., Chandrasekhar, Y., Cohan, A., et al. The semantic scholar open data platform. arXiv preprint arXiv:2301.10140, 2023. find
  33. Koza, J. R. Genetic programming as a means for programming computers by natural selection. Statistics and computing, 4:87–112, 1994. find
  34. Langley, P. Scientific discovery: Computational explorations of the creative processes. MIT press, 1987. find
  35. Lee, K.-H., Fischer, I., Wu, Y.-H., Marwood, D., Baluja, S., Schuurmans, D., and Chen, X. Evolving deeper llm thinking. arXiv preprint arXiv:2501.09891, 2025.
  36. Liu, Z., Liu, K., Zhu, Y., Lei, X., Yang, Z., Zhang, Z., Li, P., and Liu, Y. Aigs: Generating science from ai-powered automated falsification. arXiv:2411.11910, 2024a. — [arXiv:2411.11910](https://arxiv.org/abs/2411.11910) find
  37. Liu, Z., Liu, K., Zhu, Y., Lei, X., Yang, Z., Zhang, Z., Li, P., and Liu, Y. Aigs: Generating science from ai-powered automated falsification. arXiv preprint arXiv:2411.11910, 2024b. — [arXiv:2411.11910](https://arxiv.org/abs/2411.11910) find
  38. Lu, C., Lu, C., Lange, R. T., Foerster, J., Clune, J., and Ha, D. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292, 2024a.
  39. Lu, Z., Li, X., Cai, D., Yi, R., Liu, F., Zhang, X., Lane, N. D., and Xu, M. Small language models: Survey, measurements, and insights. arXiv preprint arXiv:2409.15790, 2024b.
  40. Merchant, A., Batzner, S., Schoenholz, S. S., Aykol, M., Cheon, G., and Cubuk, E. D. Scaling deep learning for materials discovery. Nature, 624(7990):80–85, 2023. find
  41. Nearing, G., Cohen, D., Dube, V., Gauch, M., Gilon, O., Harrigan, S., Hassidim, A., Klotz, D., Kratzert, F., Metzger, A., et al. Global prediction of extreme floods in ungauged watersheds. Nature, 627(8004):559–563, 2024. find
  42. Park, H., Yan, X., Zhu, R., Huerta, E. A., Chaudhuri, S., Cooper, D., Foster, I., and Tajkhorshid, E. A generative artificial intelligence framework based on a molecular diffusion model for the design of metal-organic frameworks for carbon capture. Communications Chemistry, 7(1):21, 2024. find
  43. Paster, K., Santos, M. D., Azerbayev, Z., and Ba, J. Openwebmath: An open dataset of high-quality mathematical web text. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=jKHmjlpViu. ↗
  44. Penedo, G., Kydlíček, H., allal, L. B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L. V., and Wolf, T. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URL https://arxiv.org/abs/2406.17557. ↗
  45. Peng, B., Goldstein, D., Anthony, Q. G., Albalak, A., Alcaide, E., Biderman, S., Cheah, E., Ferdinan, T., GV, K. K., Hou, H., Krishna, S., Jr., R. M., Muennighoff, N., Obeid, F., Saito, A., Song, G., Tu, H., Zhang, R., Zhao, B., Zhao, Q., Zhu, J., and Zhu, R.-J. Eagle and finch: RWKV with matrix-val… find
  46. Qin, Z., Yang, S., and Zhong, Y. Hierarchically gated recurrent neural network for sequence modeling. Advances in Neural Information Processing Systems, 36:33202–33221, 2023.
  47. Real, E., Liang, C., So, D., and Le, Q. Automl-zero: Evolving machine learning algorithms from scratch. In International conference on machine learning, pp. 8007–8019. PMLR, 2020. find
  48. Romera-Paredes, B., Barekatain, M., Novikov, A., Balog, M., Kumar, M. P., Dupont, E., Ruiz, F. J., Ellenberg, J. S., Wang, P., Fawzi, O., et al. Mathematical discoveries from program search with large language models. Nature, 625(7995):468–475, 2024.
  49. Saxton, D., Grefenstette, E., Hill, F., and Kohli, P. Analysing mathematical reasoning abilities of neural models. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=H1gR5iR5FX. ↗
  50. Schmidgall, S., Su, Y., Wang, Z., Sun, X., Wu, J., Yu, X., Liu, J., Liu, Z., and Barsoum, E. Agent laboratory: Using llm agents as research assistants. arXiv:2501.04227, 2025a.
  51. Schmidgall, S., Su, Y., Wang, Z., Sun, X., Wu, J., Yu, X., Liu, J., Liu, Z., and Barsoum, E. Agent laboratory: Using llm agents as research assistants. arXiv preprint arXiv:2501.04227, 2025b.
  52. Sharpe, W. F. The sharpe ratio. Journal of portfolio management, 21(1):49–58, 1994. find
  53. Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., and Wei, F. Retentive network: A successor to transformer for large language models, 2023.
  54. Sun, Y., Li, X., Dalal, K., Xu, J., Vikram, A., Zhang, G., Dubois, Y., Chen, X., Wang, X., Koyejo, S., et al. Learning to (learn at test time): Rnns with expressive hidden states. arXiv preprint arXiv:2407.04620, 2024.
  55. Tang, Y., Han, K., Liu, F., Ni, Y., Tian, Y., Bai, Z., Hu, Y.-Q., Liu, S., Jui, S., and Wang, Y. Rethinking optimization and architecture for tiny language models. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024. find
  56. Tay, Y., Dehghani, M., Abnar, S., Chung, H. W., Fedus, W., Rao, J., Narang, S., Tran, V. Q., Yogatama, D., and Metzler, D. Scaling laws vs model architectures: How does inductive bias influence scaling? arXiv preprint arXiv:2207.10551, 2022a.
  57. Tay, Y., Dehghani, M., Bahri, D., and Metzler, D. Efficient transformers: A survey, 2022b. URL https://arxiv.org/abs/2009.06732. ↗
  58. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  59. Viterbi, A. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE transactions on Information Theory, 13(2):260–269, 1967. find
  60. Wan, Z., Wang, X., Liu, C., Alam, S., Zheng, Y., Liu, J., Qu, Z., Yan, S., Zhu, Y., Zhang, Q., Chowdhury, M., and Zhang, M. Efficient large language models: A survey. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum? id=bsCCJHbO8A. Survey Certificatio… ↗
  61. Wang, H., Fu, T., Du, Y., Gao, W., Huang, K., Liu, Z., Chandak, P., Liu, S., Van Katwyk, P., Deac, A., et al. Scientific discovery in the age of artificial intelligence. Nature, 620(7972):47–60, 2023. find
  62. White, C., Safari, M., Sukthanker, R., Ru, B., Elsken, T., Zela, A., Dey, D., and Hutter, F. Neural architecture search: Insights from 1000 papers. arXiv preprint arXiv:2301.08727, 2023. — [arXiv:2301.08727](https://arxiv.org/abs/2301.08727) find
  63. Wong, F., Zheng, E. J., Valeri, J. A., Donghia, N. M., Anahtar, M. N., Omori, S., Li, A., Cubillos-Ruiz, A., Krishnan, A., Jin, W., et al. Discovery of a structural class of antibiotics with explainable deep learning. Nature, 626(7997):177–185, 2024. find
  64. Wu, L.-L., Luesukprasert, L., and Lee, L. Research and the long tail: A large-scale citation analysis. In 2009 42nd Hawaii International Conference on System Sciences, pp. 1–10. find
  65. Wu, Y.-F., Lee, M., and Ahn, S. Neural language of thought models. arXiv preprint arXiv:2402.01203, 2024. — [arXiv:2402.01203](https://arxiv.org/abs/2402.01203) find
  66. Xiao, D., Meng, Q., Li, S., and Yuan, X. Improving transformers with dynamically composable multi-head attention. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024. find
  67. Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023. find
  68. Yamada, Y., Lange, R. T., Lu, C., Hu, S., Lu, C., Foerster, J., Clune, J., and Ha, D. The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search. arXiv preprint arXiv:2504.08066, 2025.
  69. Ye, T., Dong, L., Xia, Y., Sun, Y., Zhu, Y., Huang, G., and Wei, F. Differential transformer. arXiv preprint arXiv:2410.05258, 2024. — [arXiv:2410.05258](https://arxiv.org/abs/2410.05258) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cheng-2025-language-modeling-language-models/