Elamur Research⌕
☰
home / papers / 2026 / RepoGenesis: Benchmarking End-to-End Microservice Generation from Readme to Repository

RepoGenesis: Benchmarking End-to-End Microservice Generation from Readme to Repository

Paper · 2026 · Zhiyuan Peng, Xin Yin, Pu Zhao et al.

Summary

Most code-generation benchmarks test single functions or classes, or ask models to edit existing codebases. Few check whether an agent can build a complete, deployable web microservice from scratch. RepoGenesis fills this gap. It is a Python/Java benchmark where a system gets a README describing endpoints, schemas, authentication and constraints, and must produce a whole repository. Black-box tests then judge the result.

The collection has 106 repositories: 30 in the evaluation set (RepoGenesis Verified) and 76 for training. They mix real GitHub projects with expert-supervised builds. Test quality is checked through a "review-rebuttal" loop in which LLM reviewers critique the tests and a human acts as arbiter. Systems are scored on three measures: functional correctness (Pass@1), how much of the required API is implemented (AC), and whether the service actually deploys (DSR).

Open-source agents and commercial IDEs often deploy and cover many endpoints, yet rarely behave correctly. Most failures come from inconsistencies across files. A small model fine-tuned on the training split rivals GPT-5 mini, which suggests the data is useful for improving models.

Key points

  • Scale: 106 repositories (60 Python, 46 Java), 18 domains, 11 frameworks (e.g., Flask, FastAPI, Django, Spring Boot, Javalin, Quarkus), 1,258 API endpoints and 2,335 test cases.
  • Splits: RepoGenesis (Verified) has 30 repositories, 6 from GitHub and 24 expert-supervised, graded Easy, Medium or Hard. RepoGenesis (Train) has 76 repositories.
  • Systems evaluated: DeepCode, MetaGPT, MS-Agent, Qwen-Agent, plus the IDEs Antigravity, Cursor and Copilot.
  • Results: Pass@1 peaks at 23.67% for Python and 21.45% for Java, whereas the highest AC score is 73.91% and the highest DSR hits 100%.
  • Failure breakdown: cross-file consistency 50.2%, architectural coherence 26.0%, dependency management 23.8%. The fine-tuned GenesisAgent-8B performs about as well as GPT-5 mini.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Shanghai Jiao Tong UniversityShanghai · China149 works · 16,161 citations🎓Zhejiang UniversityHangzhou · China172 works · 12,988 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

Cite

Cite as Zhiyuan Peng, Xin Yin, Pu Zhao, Fangkai Yang, Lu Wang, Ran Jia, Xu Chen, Qingwei Lin, Saravan Rajmohan and Dongmei Zhang. 2026. RepoGenesis: Benchmarking End-to-End Microservice Generation from Readme to Repository. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.415

@inproceedings{peng2026repogenesis,
  title = {{RepoGenesis: Benchmarking End-to-End Microservice Generation from Readme to Repository}},
  author = {Zhiyuan Peng, Xin Yin, Pu Zhao, Fangkai Yang, Lu Wang, Ran Jia, Xu Chen, Qingwei Lin, Saravan Rajmohan and Dongmei Zhang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.415},
  url = {https://doi.org/10.18653/v1/2026.acl-long.415}
}

References

51 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anysphere. 2024. Cursor: The ai code editor. https://cursor.com. ↗
  2. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and 1 others. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.
  3. Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, and 1 others. 2023. Multipl-e: a scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software En… find
  4. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  5. Etienne Daspe, Mathis Durand, Julien Hatin, and Salma Bradai. 2024. Benchmarking large language models for ethereum smart contract development. In 2024 6th Conference on Blockchain Research & Applications for Innovative Networks and Services (BRAINS), pages 1–4. find
  6. Jingzhe Ding, Shengda Long, Changxin Pu, Huan Zhou, Hongwan Gao, Xiang Gao, Chao He, Yue Hou, Fei Hu, Zhaojian Li, and 1 others. 2025. Nl2repobench: Towards long-horizon repository generation evaluation of coding agents. arXiv preprint arXiv:2512.12730. — [arXiv:2512.12730](https://arxiv.org/abs/2512.12730) find
  7. Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2023. Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation. arXiv preprint arXiv:2308.01861. — [arXiv:2308.01861](https://arxiv.org/abs/2308.01861) find
  8. Roy Thomas Fielding. 2000. Architectural styles and the design of network-based software architectures. University of California, Irvine. find
  9. Jeff Forcier, Paul Bissex, and Wesley J Chun. 2008. Python web development with Django. Addison-Wesley Professional. find
  10. Martin Fowler and James Lewis. 2015. Microservices: Nur ein weiteres konzept in der softwarearchitektur oder mehr. Objektspektrum, 1(2015):14–20. find
  11. GitHub. 2021. Github copilot: Your ai pair programmer. https://github.com/features/ copilot. ↗
  12. Google. 2025. Antigravity: The ai-first ide. https://antigravity.google. ↗
  13. Miguel Grinberg. 2018. Flask web development. " O’Reilly Media, Inc.". find
  14. Ziqi Guan, Xin Yin, Zhiyuan Peng, and Chao Ni. 2025. Repotransagent: Multi-agent llm framework for repository-aware code translation. arXiv preprint arXiv:2508.17720. — [arXiv:2508.17720](https://arxiv.org/abs/2508.17720) find
  15. Yiyang Hao, Ge Li, Yongqiang Liu, Xiaowei Miao, He Zong, Siyuan Jiang, Yang Liu, and He Wei. 2022. Aixbench: A code generation benchmark dataset. arXiv preprint arXiv:2206.13179. — [arXiv:2206.13179](https://arxiv.org/abs/2206.13179) find
  16. Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zhang, Yuheng Wang, Jinlin Wang, Ceyao Chen, Zewen Huang, Qiuyuan Wu, Liang Qian, and 1 others. 2023. Metagpt: Meta programming for a multi-agent collaborative framework. arXiv preprint arXiv:2308.00352.
  17. Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018. Mapping language to code in programmatic context. arXiv preprint arXiv:1808.09588. find
  18. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024a. Swe-bench: Can language models resolve real-world github issues? In ICLR.
  19. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024b. Swe-bench pro: Can ai agents solve long-horizon software engineering tasks? arXiv preprint arXiv:2404.07223. — [arXiv:2404.07223](https://arxiv.org/abs/2404.07223) find
  20. Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. 2019. Spoc: Search-based pseudocode to code. Advances in Neural Information Processing Systems, 32. find
  21. Data Intelligence Lab. 2025. Deepcode: Open agentic coding. find
  22. Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2023. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pages 18319–18345. PMLR. find
  23. J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159–174. find
  24. Chenliang Li, He He, Yiqing Li, Yizhang Li, Zekun Li, Fei Wang, and 1 others. 2024a. Modelscope-agent: Building your customizable agent system with open-source large language models. arXiv preprint arXiv:2404.07051. find
  25. Jia Li, Ge Li, Xuanming Zhang, Yihong Dong, and Zhi Jin. 2024b. Evocodebench: An evolving code generation benchmark aligned with real-world code repositories. arXiv preprint arXiv:2404.00599. — [arXiv:2404.00599](https://arxiv.org/abs/2404.00599) find
  26. Wei Li, Xin Zhang, Zhongxin Guo, Shaoguang Mao, Wen Luo, Guangyue Peng, Yangyu Huang, Houfeng Wang, and Scarlett Li. 2025. Fea-bench: A benchmark for evaluating repository-level code generation for feature implementation. arXiv preprint arXiv:2503.06680. find
  27. Dianshu Liao, Shidong Pan, Xiaoyu Sun, Xiaoxue Ren, Qing Huang, Zhenchang Xing, Huan Jin, and Qinying Li. 2024. A3-codgen: A repository-level code generation framework for code reuse with local-aware, global-aware, and third-party-library-aware. IEEE Transactions on Software Engineering. find
  28. Tianyang Liu, Canwen Xu, and Julian McAuley. 2023. Repobench: Benchmarking repository-level code auto-completion systems. arXiv preprint arXiv:2306.03091. find
  29. Jane Luo, Xin Zhang, Steven Liu, Jie Wu, Jianfeng Liu, Yiming Huang, Yangyu Huang, Chengyu Yin, Ying Xin, Yuefeng Zhan, and 1 others. 2025. Rpg: A repository planning graph for unified and scalable codebase generation. arXiv preprint arXiv:2509.16198. — [arXiv:2509.16198](https://arxiv.org/abs/2509.16198) find
  30. Mark Masse. 2011. REST API design rulebook: designing consistent RESTful web service interfaces. " O’Reilly Media, Inc.". find
  31. Harshitha Menon, Daniel Nichols, Abhinav Bhatele, and Todd Gamblin. 2024. Learning to predict and improve build successes in package ecosystems. In Proceedings of the 21st International Conference on Mining Software Repositories, pages 315–327. find
  32. Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Griffiths, Bertie O’Connor, Caleb Lee, and 1 others. 2023. Gaia: A benchmark for general ai assistants. arXiv preprint arXiv:2311.12983.
  33. Le Hai Nam, Dung Manh Nguyen, and Nghi DQ Bui. 2024. Repoexec: Automated repository-level code generation evaluation. arXiv preprint arXiv:2406.11927. — [arXiv:2406.11927](https://arxiv.org/abs/2406.11927) find
  34. Andy Neumann, Nuno Laranjeiro, and Jorge Bernardino. 2018. An analysis of public rest web service apis. IEEE Transactions on Services Computing, 14(4):957–970. find
  35. Sam Newman. 2021. Building microservices: designing fine-grained systems. " O’Reilly Media, Inc.". find
  36. Ziyi Ni, Huacan Wang, Shuo Zhang, Shuo Lu, Ziyang He, Wang You, Zhenheng Tang, Yuntao Du, Bill Sun, Hongzhang Liu, and 1 others. 2025. Gittaskbench: A benchmark for code agents solving real-world tasks through code repository leveraging. arXiv preprint arXiv:2508.18993. — [arXiv:2508.18993](https://arxiv.org/abs/2508.18993) find
  37. Zhiyuan Peng, Xin Yin, Rui Qian, Peiqin Lin, Yongkang Liu, Hao Zhang, Chenhao Ying, and Yuan Luo. 2025a. Soleval: Benchmarking large language models for repository-level solidity code generation. arXiv preprint arXiv:2502.18793. — [arXiv:2502.18793](https://arxiv.org/abs/2502.18793) find
  38. Zhiyuan Peng, Xin Yin, Chenhao Ying, Chao Ni, and Yuan Luo. 2025b. A preference-driven methodology for high-quality solidity code generation. arXiv preprint arXiv:2506.03006. — [arXiv:2506.03006](https://arxiv.org/abs/2506.03006) find
  39. Chris Richardson. 2018. Microservices patterns: with examples in Java. Simon and Schuster. find
  40. Sijie Shen, Xiang Zhu, Yihong Dong, Qizhi Guo, Yankun Zhen, and Ge Li. 2023. Catcoder: Repository-level code generation with relevant code and type context. arXiv preprint arXiv:2310.15077. — [arXiv:2310.15077](https://arxiv.org/abs/2310.15077) find
  41. John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528– 50652.
  42. Xin Yin, Chao Ni, Tien N Nguyen, Shaohua Wang, and Xiaohu Yang. 2024a. Rectifier: Code translation with corrector via llms. arXiv preprint arXiv:2407.07472. — [arXiv:2407.07472](https://arxiv.org/abs/2407.07472) find
  43. Xin Yin, Chao Ni, Shaohua Wang, Zhenhao Li, Limin Zeng, and Xiaohu Yang. 2024b. Thinkrepair: Self-directed automated program repair. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 1274–1286. find
  44. Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. 2024. Codereval: A benchmark of pragmatic code generation with generative pre-trained models. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, page… find
  45. Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, and 1 others. 2025. Multi-swe-bench: A multilingual benchmark for issue resolving. arXiv preprint arXiv:2504.02605. — [arXiv:2504.02605](https://arxiv.org/abs/2504.02605) find
  46. Daoguang Zan, Ailun Yu, Wei Liu, Dong Chen, Bo Shen, Yafen Yao, Wei Li, Xiaolin Chen, Yongshun Gong, Bei Guan, and 1 others. 2024. Codes: Natural language to code repository via multi-layer sketch. ACM Transactions on Software Engineering and Methodology.
  47. Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. Repocoder: Repository-level code completion through iterative retrieval and generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. find
  48. Wei Tao, Yucheng Zhou, Yanlin Wang, Wenqiang Zhang, Hongyu Zhang, and Yu Cheng. 2024. Magis: Llm-based multi-agent framework for github issue resolution. Advances in Neural Information Processing Systems, 37:51963–51993. find
  49. Qwen Team. 2024. Qwen-agent: A framework for rag and agent applications. https://github.com/ QwenLM/Qwen-Agent. ↗
  50. Xingyao Wang and 1 others. 2024. Openhands: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741.
  51. Boyang Yang, Luyao Ren, Xin Yin, Jiadong Ren, Haoye Tian, and Shunfu Jin. 2025. Input reduction enhanced llm-based program repair. arXiv preprint arXiv:2507.15251. — [arXiv:2507.15251](https://arxiv.org/abs/2507.15251) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/peng-2026-repogenesis-benchmarking-end-end-microservice/