Elamur Research⌕
☰
home / papers / 2025 / 基于大语言模型智能体的代码生成综述

基于大语言模型智能体的代码生成综述

Summary

This Chinese-language survey from Peking University reviews code generation agents built on large language models (LLMs). Plain LLMs answer in a single pass. They cannot break tasks down on their own, interact with real development environments, verify their output or keep correcting it. Agents fill these gaps by wrapping the model in loops of planning, acting, observing and refining. The authors argue that agents differ from earlier code generation in three ways: they work autonomously, they cover most of the software development life cycle, and research attention has moved from model algorithms to engineering concerns such as workflow management, reliability and tool integration. Using a systematic literature search, they organize the field by technique rather than by application scenario, which is how prior English-language surveys grouped it. They also map applications across the development cycle, catalogue benchmarks and metrics, and compare commercial tools. The result is a method-centred reference that ends with a five-part research agenda on open problems.

Key points

  • Corpus: The search covered January 2022 to June 2025 across ACM DL, IEEE Xplore, SpringerLink, Google Scholar, DBLP and CNKI, limited to CCF A/B venues plus high-impact arXiv preprints. It produced 447 candidates, filtered to 100 core papers. arXiv supplied 44 of them, ACL 12, ICML 11 and ICLR 9. No relevant papers appeared in 2022.

  • Taxonomy: Single-agent techniques fall into three groups:

    • planning and reasoning (e.g., Self-planning, CodeAct, CodePlan);
    • tool integration and retrieval augmentation;
    • reflection and self-improvement.

    Multi-agent systems are treated separately. The survey names Self-collaboration (April 2023) as the earliest LLM multi-agent software development framework, followed by work such as ChatDev, MetaGPT and SWE-agent.

  • Tools compared: GitHub Copilot, Devin, Cursor, Tongyi Lingma and Claude Code (200k-token context). They are grouped into three roles: programming assistant, collaborative partner and autonomous development team. Devin is singled out for low real-world reliability.

  • Challenges: The authors list five areas:

    • core capability limits, including context engineering failures (poisoned, distracting, confusing or conflicting context);
    • error cascades in multi-agent pipelines and continual learning;
    • tool safety, high running cost and edge deployment;
    • hallucination, malicious code and copyright;
    • evaluation that relies too heavily on pass@k.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Natural language processing

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.13328/j.cnki.jos.007593

@article{2025,
  title = {{基于大语言模型智能体的代码生成综述}},
  author = {董益宏 and 姜雪 and 钱家如 and 王天 and 张克驰 and 金芝 and 李戈},
  journal = {软件学报 (Journal of Software)},
  year = {2025},
  doi = {10.13328/j.cnki.jos.007593},
  url = {https://doi.org/10.13328/j.cnki.jos.007593}
}

References

169 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kitzelmann E. Inductive programming: A survey of program synthesis techniques. In: Proc. of the 3rd Int’l Workshop on Approaches and Applications of Inductive Programming (AAIP). Edinburgh: Springer, 2010. 50–73. [doi: 10.1007/978-3-642-11931-6_3] — [doi:10.1007/978-3-642-11931-6_3](https://doi.org/10.1007/978-3-642-11931-6_3) find
  2. Ling W, Grefenstette E, Hermann KM, Kočiský T, Senior A, Wang FM, Blunsom P. Latent predictor networks for code generation. In: Proc. of the 54th Annual Meeting of the Association for Computational Linguistics (Vol. 1: Long Papers). Berlin: ACL, 2016. 599–609. [doi: 10.18653/v1/P16-1057] find
  3. Yin PC, Neubig G. A syntactic neural model for general-purpose code generation. In: Proc. of the 55th Annual Meeting of the Association for Computational Linguistics (Vol. 1: Long Papers). Vancouver: ACL, 2017. 440–450. [doi: 10.18653/v1/P17-1041] find our card
  4. Touvron H, Lavril T, Izacard G, Martinet X, Lachaux MA, Lacroix T, Rozière B, Goyal N, Hambro E, Azhar F, Rodriguez A, Joulin A, Grave E, Lample G. LLaMA: Open and efficient foundation language models. arXiv:2302.13971, 2023. find
  5. Touvron H, Martin L, Stone K, et al. LLaMA 2: Open foundation and fine-tuned chat models. arXiv:2307.09288, 2023. find
  6. Ouyang L, Wu J, Jiang X, Almeida D, Wainwright CL, Mishkin P, Zhang C, Agarwal S, Slama K, Ray A, Schulman J, Hilton J, Kelton F, Miller L, Simens M, Askell A, Welinder P, Christiano P, Leike J, Lowe R. Training language models to follow instructions with human feedback. In: Proc. of the 36th Int’l… find
  7. Rozière B, Gehring J, Gloeckle F, et al. Code Llama: Open foundation models for code. arXiv:2308.12950, 2024. find our card
  8. Wang Y, Wang WS, Joty S, Hoi SCH. CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In: Proc. of the 2021 Conf. on Empirical Methods in Natural Language Processing (EMNLP). Punta Cana: ACL, 2021. 8696–8708. [doi: 10.18653/v1/2021.emnlp-main.68… find
  9. Li YJ, Choi D, Chung J, et al. Competition-level code generation with AlphaCode. Science, 2022, 378(6624): 1092–1097. [doi: 10.1126/science.abq1158] find our card
  10. Jiang X, Dong YH, Wang LC, Fang Z, Shang QW, Li G, Jin Z, Jiao WP. Self-planning code generation with large language models. ACM Trans. on Software Engineering and Methodology, 2024, 33(7): 182. [doi: 10.1145/3672456] find
  11. Le H, Chen HL, Saha A, Gokul A, Sahoo D, Joty S. CodeChain: Towards modular code generation through chain of self-revisions with representative sub-modules. In: Proc. of the 12th Int’l Conf. on Learning Representations (ICLR). Vienna: OpenReview.net, 2024. 1–31. find
  12. Pan RW, Zhang HY, Liu C. CodeCoR: An LLM-based self-reflective multi-agent framework for code generation. arXiv:2501.07811, 2025. — [arXiv:2501.07811](https://arxiv.org/abs/2501.07811) find
  13. Rasheed Z, Waseem M, Saari M, Systä K, Abrahamsson P. CodePori: Large scale model for autonomous software development by using multi-agents. arXiv:2402.01411, 2024. — [arXiv:2402.01411](https://arxiv.org/abs/2402.01411) find
  14. Manish S. An autonomous multi-agent LLM framework for agile software development. Int’l Journal of Trend in Scientific Research and Development, 2024, 8(5): 892–898. [doi: 10.5281/zenodo.14179129] — [doi:10.5281/zenodo.14179129](https://doi.org/10.5281/zenodo.14179129) find
  15. Dong YH, Jiang X, Jin Z, Li G. Self-collaboration code generation via ChatGPT. ACM Trans. on Software Engineering and Methodology, 2024, 33(7): 189. [doi: 10.1145/3672459] find
  16. Qian C, Liu W, Liu HZ, Chen N, Dang YF, Li JH, Yang C, Chen WZ, Su YS, Cong X, Xu JY, Li DH, Liu ZY, Sun MS. ChatDev: Communicative agents for software development. In: Proc. of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol. 1: Long Papers). Bangkok: ACL, 2024. 15174–… find our card
  17. Hong SR, Zhuge MC, Chen J, Zheng XW, Cheng YH, Wang JL, Zhang CY, Wang ZL, Yau SKS, Lin ZJ, Zhou LY, Ran CY, Xiao LF, Wu CL, Schmidhuber J. MetaGPT: Meta programming for a multi-agent collaborative framework. In: Proc. of the 12th Int’l Conf. on Learning Representations (ICLR). Vienna: OpenReview.ne… find our card
  18. Lu S, Guo DY, Ren S, Huang JJ, Svyatkovskiy A, Blanco A, Clement CB, Drain D, Jiang DX, Tang DY, Li G, Zhou LD, Shou LJ, Zhou L, Tufano M, Gong M, Zhou M, Duan N, Sundaresan N, Deng SK, Fu SY, Liu SJ. CodeXGLUE: A machine learning benchmark dataset for code understanding and generation. In: Proc. of… find
  19. Mu FW, Shi L, Wang S, Yu ZH, Zhang BQ, Wang CX, Liu SC, Wang Q. ClarifyGPT: Empowering LLM-based code generation with intention clarification. arXiv:2310.10996, 2023. — [arXiv:2310.10996](https://arxiv.org/abs/2310.10996) find
  20. Wang ZT, Wang W, Li ZR, Wang L, Yi C, Xu XJ, Cao LY, Su HJ, Chen SZ, Zhou J. XUAT-Copilot: Multi-agent collaborative system for automated user acceptance testing with large language model. arXiv:2401.02705, 2024. find
  21. Zhang K, Zhang CX, Wang C, Zhang C, Wu YC, Xing ZC, Liu Y, Li QS, Peng X. LogiAgent: Automated logical testing for REST systems with LLM-based multi-agents. arXiv:2503.15079, 2025. — [arXiv:2503.15079](https://arxiv.org/abs/2503.15079) find
  22. Baumgartner N, Iyenghar P, Schoemaker T, Pulvermüller E. AI-driven refactoring: A pipeline for identifying and correcting data clumps in Git repositories. Electronics, 2024, 13(9): 1644. [doi: 10.3390/electronics13091644] — [doi:10.3390/electronics13091644](https://doi.org/10.3390/electronics13091644) find
  23. Rabinovich M, Stern M, Klein D. Abstract syntax networks for code generation and semantic parsing. In: Proc. of the 55th Annual Meeting of the Association for Computational Linguistics (Vol. 1: Long Papers). Vancouver: ACL, 2017. 1139–1149. [doi: 10.18653/v1/P17-1105] find
  24. Guo DY, Ren S, Lu S, Feng ZY, Tang DY, Liu SJ, Zhou L, Duan N, Svyatkovskiy A, Fu SY, Tufano M, Deng SK, Clement C, Drain D, Sundaresan N, Yin J, Jiang DX, Zhou M. GraphCodeBERT: Pre-training code representations with data flow. In: Proc. of the 9th Int’l Conf. on Learning Representations (ICLR). Op… find our card
  25. Guo DY, Lu S, Duan N, Wang YL, Zhou M, Yin J. UniXcoder: Unified cross-modal pre-training for code representation. In: Proc. of the 60th Annual Meeting of the Association for Computational Linguistics (Vol. 1: Long Papers). Dublin: ACL, 2022. 7212–7225. [doi: 10.18653/v1/2022.acl-long.499] find our card
  26. Hu YJ, Zhou Q, Chen QH, Li XP, Liu LB, Zhang DJ, Kachroo A, Oz T, Tripp O. QualityFlow: An agentic workflow for program synthesis controlled by LLM quality checks. arXiv:2501.17167, 2025. — [arXiv:2501.17167](https://arxiv.org/abs/2501.17167) find
  27. Ishibashi Y, Nishimura Y. Self-organized agents: A LLM multi-agent framework toward ultra large-scale code generation and optimization. arXiv:2404.02183, 2024. — [arXiv:2404.02183](https://arxiv.org/abs/2404.02183) find
  28. Zhang KC, Li J, Li G, Shi XJ, Jin Z. CodeAgent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. In: Proc. of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol. 1: Long Papers). Bangkok: ACL, 2024. 13643–13658. [doi… find
  29. Wang RX, Han XD, Ji L, Wang S, Baldwin T, Li HN. ToolGen: Unified tool retrieval and calling via generation. In: Proc. of the 13th Int’l Conf. on Learning Representations (ICLR). Singapore: OpenReview.net, 2025. 1–26. find
  30. Sapkota R, Roumeliotis KI, Karkee M. Vibe coding vs. agentic coding: Fundamentals and practical implications of agentic AI. arXiv:2505.19443, 2025. — [arXiv:2505.19443](https://arxiv.org/abs/2505.19443) find
  31. Schäfer M, Nadi S, Eghbali A, Tip F. Adaptive test generation using a large language model. arXiv:2302.06527, 2023. — [arXiv:2302.06527](https://arxiv.org/abs/2302.06527) find
  32. Xu QH, Wang GC, Briand L, Liu K. A multi-agent LLM-based JUit test generation with strong oracles. arXiv:2506.02943, 2025. — [arXiv:2506.02943](https://arxiv.org/abs/2506.02943) find
  33. Dearing MT, Tao YH, Wu XF, Lan ZL, Taylor V. Leveraging LLMs to automate energy-aware refactoring of parallel scientific codes. arXiv:2505.02184, 2025. find
  34. Peng HY, Gupte A, Hasler R, Eliopoulos NJ, Ho CC, Mantri R, Deng L, Läufer K, Thiruvathukal GK, Davis JC. SysLLMatic: Large language models are software system optimizers. arXiv:2506.01249, 2025. — [arXiv:2506.01249](https://arxiv.org/abs/2506.01249) find
  35. Shi WX, Zhang YH, Xing XY, Xu J. Harnessing large language models for seed generation in greybox fuzzing. arXiv:2411.18143, 2024. — [arXiv:2411.18143](https://arxiv.org/abs/2411.18143) find
  36. Foster C, Gulati A, Harman M, Harper I, Mao K, Ritchey J, Robert H, Sengupta S. Mutation-guided LLM-based test generation at meta. In: Proc. of the 33rd ACM Int’l Conf. on the Foundations of Software Engineering. Trondheim: ACM, 2025. 180–191. [doi: 10.1145/3696630.3728544] — [doi:10.1145/3696630.3728544](https://doi.org/10.1145/3696630.3728544) find
  37. Jin HL, Huang LH, Cai HP, Yan J, Li B, Chen HM. From LLMs to LLM-based agents for software engineering: A survey of current, challenges and future. arXiv:2408.02479, 2024. — [arXiv:2408.02479](https://arxiv.org/abs/2408.02479) find
  38. Liu JW, Wang KX, Chen YX, Peng X, Chen ZP, Zhang LM, Lou YL. Large language model-based agents for software engineering: A survey. arXiv:2409.02977, 2024. — [arXiv:2409.02977](https://arxiv.org/abs/2409.02977) find
  39. He JD, Treude C, Lo D. LLM-based multi-agent systems for software engineering: Literature review, vision, and the road ahead. ACM Trans. on Software Engineering and Methodology, 2025, 34(5): 124. [doi: 10.1145/3712003] — [doi:10.1145/3712003](https://doi.org/10.1145/3712003) find
  40. Wang YL, Zhong WJ, Huang YX, Shi ES, Yang M, Chen JC, Li H, Ma YC, Wang QX, Zheng ZB. Agents in software engineering: Survey, landscape, and vision. Automated Software Engineering, 2025, 32(2): 70. [doi: 10.1007/s10515-025-00544-2] — [doi:10.1007/s10515-025-00544-2](https://doi.org/10.1007/s10515-025-00544-2) find
  41. Svyatkovskiy A, Deng SK, Fu SY, Sundaresan N. Intellicode compose: Code generation using Transformer. In: Proc. of the 28th ACM Joint Meeting on European Software Engineering Conf. and Symp. on the Foundations of Software Engineering (ESEC/FSE). ACM, 2020. 1433–1443. [doi: 10.1145/3368089.3417058] find
  42. Herrington J. Code Generation in Action. Greenwich: Manning Publications Co., 2003. find
  43. Becker BA, Denny P, Finnie-Ansley J, Luxton-Reilly A, Prather J, Santos EA. Programming is hard-or at least it used to be: Educational opportunities and challenges of AI code generation. In: Proc. of the 54th ACM Technical Symp. on Computer Science Education V.1 (SIGCSE). Toronto: ACM, 2023. 500–506… find
  44. Xu FF, Vasilescu B, Neubig G. In-IDE code generation from natural language: Promise and challenges. ACM Trans. on Software Engineering and Methodology, 2022, 31(2): 29. [doi: 10.1145/3487569] — [doi:10.1145/3487569](https://doi.org/10.1145/3487569) find
  45. Huynh N, Lin BY. Large language models for code generation: A comprehensive survey of challenges, techniques, evaluation, and applications. arXiv:2503.01245, 2025. — [arXiv:2503.01245](https://arxiv.org/abs/2503.01245) find
  46. Kasneci E, Sessler K, Küchemann S, et al. ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 2023, 103: 102274. [doi: 10.1016/j.lindif.2023.102274] find our card
  47. Chang YP, Wang X, Wang JD, Wu Y, Yang LY, Zhu KJ, Chen H, Yi XY, Wang CX, Wang YD, Ye W, Zhang Y, Chang Y, Yu PS, Yang Q, Xie X. A survey on evaluation of large language models. ACM Trans. on Intelligent Systems and Technology, 2024, 15(3): 39. [doi: 10.1145/3641289] find our card
  48. Zhao WX, Zhou K, Li JY, Tang T, Wang X, Hou Y, Min Y, Zhang B, Zhang J, Dong Z, Du Y, Yang C, Chen Y, Chen Z, Jiang J, Ren R, Li Y, Tang X, Liu Z, Liu P, Nie JY, Wen JR. A survey of large language models. arXiv:2303.18223, 2023. find our card
  49. Naveed H, Khan AU, Qiu S, Saqib M, Anwar S, Usman M, Akhtar N, Barnes N, Mian A. A comprehensive overview of large language models. ACM Trans. on Intelligent Systems and Technology, 2025, 16(5): 106. [doi: 10.1145/3744746] find our card
  50. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I. Attention is all you need. In: Proc. of the 31st Int’l Conf. on Neural Information Processing Systems (NeurIPS). Long Beach: Curran Associates Inc., 2017. 6000–6010. find
  51. Chen M, Tworek J, Jun H, et al. Evaluating large language models trained on code. arXiv:2107.03374, 2021. find our card
  52. Guo DY, Zhu QH, Yang DJ, Xie ZD, Dong K, Zhang WT, Chen GT, Bi X, Wu Y, Li YK, Luo FL, Xiong YF, Liang WF. DeepSeek-Coder: When the large language model meets programming—The rise of code intelligence. arXiv:2401.14196, 2024. find
  53. Hui BY, Yang J, Cui ZY, Yang JX, Liu DH, Zhang L, Liu TY, Zhang JJ, Yu BW, Lu KM, Dang K, Fan Y, Zhang YC, Yang A, Men R, Huang F, Zheng B, Miao YB, Quan SHR, Feng YL, Ren XZ, Ren XC, Zhou JR, Lin JY. Qwen2.5-Coder technical report. arXiv:2409.12186, 2024. find our card
  54. Wei J, Wang XZ, Schuurmans D, Bosma M, Ichter B, Xia F, Chi EH, Le QV, Zhou D. Chain-of-thought prompting elicits reasoning in large language models. In: Proc. of the 36th Int’l Conf. on Neural Information Processing Systems (NeurIPS). New Orleans: Curran Associates Inc., 2022. 1800. find
  55. Wang EZ, Cassano F, Wu C, Bai YF, Song W, Nath V, Han ZW, Hendryx SM, Yue S, Zhang H. Planning in natural language improves LLM search for code generation. In: Proc. of the 13th Int’l Conf. on Learning Representations (ICLR). Singapore: OpenReview.net, 2025. 1–47. find
  56. Nakano R, Hilton J, Balaji S, Wu J, Ouyang L, Kim C, Hesse C, Jain S, Kosaraju V, Saunders W, Jiang X, Cobbe K, Eloundou T, Krueger G, Button K, Knight M, Chess B, Schulman J. WebGPT: Browser-assisted question-answering with human feedback. arXiv:2112.09332, 2022. find our card
  57. Schick T, Dwivedi-Yu J, Dessì R, Raileanu R, Lomeli M, Hambro E, Zettlemoyer L, Cancedda N, Scialom T. Toolformer: Language models can teach themselves to use tools. In: Proc. of the 37th Int’l Conf. on Neural Information Processing Systems (NeurIPS). New Orleans: Curran Associates Inc., 2023. 2997… find our card
  58. Gao LY, Madaan A, Zhou SY, Alon U, Liu PF, Yang YM, Callan J, Neubig G. PAL: Program-aided language models. In: Proc. of the 40th Int’l Conf. on Machine Learning (ICML). Honolulu: JMLR.org, 2023. 435. find
  59. Park JS, O’Brien J, Cai CJ, Morris MR, Liang P, Bernstein MS. Generative agents: Interactive simulacra of human behavior. In: Proc. of the 36th Annual ACM Symp. on User Interface Software and Technology (UIST). San Francisco: ACM, 2023. 2. [doi: 10.1145/3586183.3606763] find
  60. Mehta N, Teruel M, Sanz PF, Deng X, Awadallah AH, Kiseleva J. Improving grounded language understanding in a collaborative environment by interacting with agents through help feedback. In: Findings of the Association for Computational Linguistics: EACL 2024. St. Julian’s: ACL, 2024. 1306–1321. find
  61. Xi ZH, Chen WX, Guo X, et al. The rise and potential of large language model based agents: A survey. Science China Information Sciences, 2025, 68(2): 121101. [doi: 10.1007/s11432-024-4222-0] find our card
  62. Wang L, Ma C, Feng XY, Zhang ZY, Yang H, Zhang JS, Chen ZY, Tang JK, Chen X, Lin YK, Zhao WX, Wei ZW, Wen JR. A survey on large language model based autonomous agents. Frontiers of Computer Science, 2024, 18(6): 186345. [doi: 10.1007/s11704-024-40231-1] find our card
  63. Guo TC, Chen XY, Wang YQ, Chang RD, Pei SC, Chawla NV, Wiest O, Zhang XL. Large language model based multi-agents: A survey of progress and challenges. In: Proc. of the 33rd Int’l Joint Conf. on Artificial Intelligence. Jeju: ijcai.org, 2024. 8048–8057. find our card
  64. Cheng YH, Zhang CY, Zhang ZW, Meng XR, Hong SR, Li WH, Wang ZH, Wang ZK, Yin F, Zhao JH, He XQ. Exploring large language model based intelligent agents: Definitions, methods, and prospects. arXiv:2401.03428, 2024. — [arXiv:2401.03428](https://arxiv.org/abs/2401.03428) find
  65. Giray L. Prompt engineering with ChatGPT: A guide for academic writers. Annals of Biomedical Engineering, 2023, 51(12): 2629–2633. [doi: 10.1007/s10439-023-03272-4] — [doi:10.1007/s10439-023-03272-4](https://doi.org/10.1007/s10439-023-03272-4) find
  66. White J, Fu QC, Hays S, Sandborn M, Olea C, Gilbert H, Elnashar A, Spencer-Smith J, Schmidt DC. A prompt pattern catalog to enhance prompt engineering with ChatGPT. arXiv:2302.11382, 2023. find our card
  67. Gao YF, Xiong Y, Gao XY, Jia KX, Pan JL, Bi YX, Dai Y, Sun JW, Wang M, Wang HF. Retrieval-augmented generation for large language models: A survey. arXiv:2312.10997, 2023. find our card
  68. Lewis P, Perez E, Piktus A, Petroni F, Karpukhin V, Goyal N, Küttler H, Lewis M, Yih WT, Rocktäschel T, Riedel S, Kiela D. Retrieval-augmented generation for knowledge-intensive NLP tasks. In: Proc. of the 34th Int’l Conf. on Neural Information Processing Systems (NeurIPS). Vancouver: Curran Associa… find
  69. Li J, Tao CY, Li J, Li G, Jin Z, Zhang HZ, Fang Z, Liu F. Large language model-aware in-context learning for code generation. ACM Trans. on Software Engineering and Methodology, 2025, 34(7): 190. [doi: 10.1145/3715908] — [doi:10.1145/3715908](https://doi.org/10.1145/3715908) find
  70. Wei J, Wei J, Tay Y, Tran D, Webson A, Lu YF, Chen XY, Liu HX, Huang D, Zhou D, Ma TY. Larger language models do in-context learning differently. arXiv:2303.03846, 2023. find our card
  71. Huang D, Zhang JM, Luck M, Bu QW, Qing YH, Cui HM. AgentCoder: Multi-agent-based code generation with iterative testing and optimisation. arXiv:2312.13010, 2023. find
  72. Phan HN, Nguyen TN, Nguyen PX, Bui NDQ. HyperAgent: Generalist software engineering agents to solve coding tasks at scale. arXiv:2409.16299, 2024. — [arXiv:2409.16299](https://arxiv.org/abs/2409.16299) find
  73. Zhang KC, Zhang HZ, Li G, Li J, Li Z, Jin Z. ToolCoder: Teach code generation models to use API search tools. arXiv:2305.04032, 2023. — [arXiv:2305.04032](https://arxiv.org/abs/2305.04032) find
  74. Phan HN, Phan HN, Nguyen TN, Bui NDQ. RepoHyper: Better context retrieval is all you need for repository-level code completion. arXiv:2403.06095, 2024. — [arXiv:2403.06095](https://arxiv.org/abs/2403.06095) find
  75. Madaan A, Tandon N, Gupta P, Hallinan S, Gao LY, Wiegreffe S, Alon U, Dziri N, Prabhumoye S, Yang YM, Gupta S, Majumder BP, Hermann K, Welleck S, Yazdanbakhsh A, Clark P. Self-Refine: Iterative refinement with self-feedback. In: Proc. of the 37th Int’l Conf. on Neural Information Processing Systems… find
  76. Zhang KC, Li Z, Li J, Li G, Jin Z. Self-Edit: Fault-aware code editor for code generation. In: Proc. of the 61st Annual Meeting of the Association for Computational Linguistics (Vol. 1: Long Papers). Toronto: ACL, 2023. 769–787. [doi: 10.18653/v1/2023.acl-long.45] find
  77. Wang XY, Chen YY, Yuan LF, Zhang YZ, Li YZ, Peng H, Ji H. Executable code actions elicit better LLM agents. In: Proc. of the 41st Int’l Conf. on Machine Learning (ICML). Vienna: OpenReview.net, 2024. 1–25. find our card
  78. Huang T, Sun ZH, Jin Z, Li G, Lyu C. Knowledge-aware code generation with large language models. In: Proc. of the 32nd IEEE/ACM Int’l Conf. on Program Comprehension (ICPC). Lisbon: ACM, 2024. 52–63. [doi: 10.1145/3643916.3644418] — [doi:10.1145/3643916.3644418](https://doi.org/10.1145/3643916.3644418) find
  79. Gur I, Furuta H, Huang AV, Safdari M, Matsuo Y, Eck D, Faust A. A real-world WebAgent with planning, long context understanding, and program synthesis. In: Proc. of the 12th Int’l Conf. on Learning Representations (ICLR). Vienna: OpenReview.net, 2024. 1–28. find
  80. Bairi R, Sonwane A, Kanade A, Vageesh DC, Iyer A, Parthasarathy S, Rajamani S, Ashok B, Shet S. CodePlan: Repository-level coding using LLMs and planning. Proc. of the ACM on Software Engineering, 2024, 1(FSE): 31. [doi: 10.1145/3643757] — [doi:10.1145/3643757](https://doi.org/10.1145/3643757) find
  81. Dainese N, Merler M, Alakuijala M, Marttinen P. Generating code world models with large language models guided by Monte Carlo tree search. In: Proc. of the 38th Int’l Conf. on Neural Information Processing Systems (NeurIPS). Vancouver: Curran Associates Inc., 2024. 1933. find
  82. Li JR, Le H, Zhou YB, Xiong CM, Savarese S, Sahoo D. CodeTree: Agent-guided tree search for code generation with large language models. In: Proc. of the 2025 Conf. of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Vol. 1: Long Paper… find
  83. Ni ZY, Li YF, Yang N, Shen D, Lv P, Dong DX. Tree-of-Code: A tree-structured exploring framework for end-to-end code generation and execution in complex task handling. arXiv:2412.15305, 2024. — [arXiv:2412.15305](https://arxiv.org/abs/2412.15305) find
  84. Han YW, Lyu C. Multi-stage guided code generation for large language models. Engineering Applications of Artificial Intelligence, 2025, 139: 109491. [doi: 10.1016/J.ENGAPPAI.2024.109491] — [doi:10.1016/j.engappai.2024.109491](https://doi.org/10.1016/j.engappai.2024.109491) find
  85. Aggarwal V, Kamal O, Japesh A, Jin ZJ, Schölkopf B. DARS: Dynamic action re-sampling to enhance coding agent performance by adaptive tree traversal. In: Proc. of the 63rd Annual Meeting of the Association for Computational Linguistics (Vol. 1: Long Papers). Vienna: ACL, 2025. 19808–19855. [doi: 10.1… find
  86. Ho CT, Ren HX, Khailany B. VerilogCoder: Autonomous verilog coding agents with graph-based planning and abstract syntax tree (AST)-based waveform tracing tool. In: Proc. of the 39th AAAI Conf. on Artificial Intelligence. Philadelphia: AAAI, 2025. 300–307. [doi: 10.1609/aaai.v39i1.32007] — [doi:10.1609/aaai.v39i1.32007](https://doi.org/10.1609/aaai.v39i1.32007) find
  87. Zainullina K, Golubev A, Trofimova M, Polezhaev S, Badertdinov I, Litvintseva D, Karasik S, Fisin F, Skvortsov S, Nekrashevich M, Shevtsov A, Yangel B. Guided search strategies in non-serializable environments with applications to software engineering agents. In: Proc. of the 42nd Int’l Conf. on Mac… find
  88. Jiang X, Dong YH, Tao YD, Liu HY, Jin Z, Li G. ROCODE: Integrating backtracking mechanism and program analysis in large language models for code generation. In: Proc. of the 47th Int’l Conf. on Software Engineering (ICSE). Ottawa: IEEE, 2025. 334–346. [doi: 10.1109/ICSE55347.2025.00133] — [doi:10.1109/icse55347.2025.00133](https://doi.org/10.1109/icse55347.2025.00133) find
  89. Lu YF, Ye FH, Li J, Gao Q, Liu C, Luo HB, Du N, Li XL, Ren FL. CodeTool: Enhancing programmatic tool invocation of LLMs via process supervision. In: Proc. of the 63rd Annual Meeting of the Association for Computational Linguistics (Vol. 1: Long Papers). Vienna: ACL, 2025. 18287–18304. [doi: 10.18653… find
  90. Li J, Shi XJ, Zhang KC, Li G, Jin Z, Li L, Zhang HZ, Li J, Liu F, Zhang YW, Tao ZW, Dong YH, Zhu YQ, Tao CY. GraphCodeAgent: Dual graph-guided LLM agent for retrieval-augmented repo-level code generation. arXiv:2504.10046, 2025. — [arXiv:2504.10046](https://arxiv.org/abs/2504.10046) find
  91. Gupta T, Weihs L, Kembhavi A. CodeNav: Beyond tool-use to using real-world codebases with LLM agents. arXiv:2406.12276, 2024. — [arXiv:2406.12276](https://arxiv.org/abs/2406.12276) find
  92. Acharya M, Zhang YF, Leach K, Huang Y. Optimizing code runtime performance through context-aware retrieval-augmented generation. In: Proc. of the 33rd Int’l Conf. on Program Comprehension (ICPC). Ottawa: IEEE, 2025. 1–5. [doi: 10.1109/ICPC66645.2025.00028] — [doi:10.1109/icpc66645.2025.00028](https://doi.org/10.1109/icpc66645.2025.00028) find
  93. Athale M, Vaddina V. Knowledge graph based repository-level code generation. In: Proc. of the 2025 Int’l Workshop on Large Language Models for Code (LLM4Code). Ottawa: IEEE, 2025. 169–176. [doi: 10.1109/LLM4Code66737.2025.00026] — [doi:10.1109/llm4code66737.2025.00026](https://doi.org/10.1109/llm4code66737.2025.00026) find
  94. Zhang YL, Zhao XR, Wang ZZ, Yang CY, Wei JY, Wu TS. cAST: Enhancing code retrieval-augmented generation with structural chunking via abstract syntax tree. arXiv:2506.15655, 2025. find
  95. Lai Y, Lee S, Chen GJ, Poddar S, Hu MK, Pan DZ, Luo P. AnalogCoder: Analog circuit design via training-free code generation. In: Proc. of the 39th AAAI Conf. on Artificial Intelligence. Philadelphia: AAAI, 2025. 379–387. [doi: 10.1609/aaai.v39i1.32016] — [doi:10.1609/aaai.v39i1.32016](https://doi.org/10.1609/aaai.v39i1.32016) find
  96. Chang TY, Chen SZ, Fan GD, Feng ZY. A self-iteration code generation method based on large language models. In: Proc. of the 29th Int’l Conf. on Parallel and Distributed Systems (ICPADS). Ocean Flower Island: IEEE, 2023. 275–281. [doi: 10.1109/ICPADS60453.2023.00049] — [doi:10.1109/icpads60453.2023.00049](https://doi.org/10.1109/icpads60453.2023.00049) find
  97. Chen XY, Lin M, Schärli N, Zhou D. Teaching large language models to self-debug. In: Proc. of the 12th Int’l Conf. on Learning Representations (ICLR). Vienna: OpenReview.net, 2024. 1–80. find
  98. Olausson TX, Inala JP, Wang CL, Gao JF, Solar-Lezama A. Is Self-repair a silver bullet for code generation? In: Proc. of the 12th Int’l Conf. on Learning Representations (ICLR). Vienna: OpenReview.net, 2024. 1–49. find
  99. Jiang N, Li XP, Wang SQ, Zhou Q, Hossain SB, Ray B, Kumar V, Ma XF, Deoras A. LEDEX: Training LLMs to better self-debug and explain code. In: Proc. of the 38th Int’l Conf. on Neural Information Processing Systems (NeurIPS). Vancouver: Curran Associates Inc., 2024. 1120. find
  100. Tao W, Zhou YC, Wang YL, Zhang WQ, Zhang HY, Cheng Y. MAGIS: LLM-based multi-agent framework for GitHub issue ReSolution. In: Proc. of the 38th Int’l Conf. on Neural Information Processing Systems (NeurIPS). Vancouver: Curran Associates Inc., 2024. 1647. find
  101. Zhang H, Cheng W, Wu YH, Hu W. A pair programming framework for code generation via multi-plan exploration and feedback-driven refinement. In: Proc. of the 39th IEEE/ACM Int’l Conf. on Automated Software Engineering (ASE). Sacramento: ACM, 2024. 1319–1331. [doi: 10.1145/3691620.3695506] — [doi:10.1145/3691620.3695506](https://doi.org/10.1145/3691620.3695506) find
  102. Lin F, Kim DJ, Chen TH. SOEN-101: Code generation by emulating software process models using large language model agents. In: Proc. of the 47th Int’l Conf. on Software Engineering (ICSE). Ottawa: IEEE, 2025. 1527–1539. [doi: 10.1109/ICSE55347.2025.00140] — [doi:10.1109/icse55347.2025.00140](https://doi.org/10.1109/icse55347.2025.00140) find
  103. Zhao YJ, Zhang HJ, Huang HX, Yu ZM, Zhao JS. MAGE: A multi-agent engine for automated RTL code generation. In: Proc. of the 62nd ACM/IEEE Design Automation Conf. (DAC). San Francisco: IEEE, 2025. 1–7. [doi: 10.1109/DAC63849.2025.11133191] — [doi:10.1109/dac63849.2025.11133191](https://doi.org/10.1109/dac63849.2025.11133191) find
  104. Islam MA, Ali ME, Parvez MR. MapCoder: Multi-agent code generation for competitive problem solving. In: Proc. of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol. 1: Long Papers). Bangkok: ACL, 2024. 4912–4944. [doi: 10.18653/v1/2024.acl-long.269] find
  105. Nunez A, Islam NT, Jha SK, Najafirad P. AutoSafeCoder: A multi-agent framework for securing LLM code generation through static analysis and fuzz testing. arXiv:2409.10737, 2024. — [arXiv:2409.10737](https://arxiv.org/abs/2409.10737) find
  106. Rahman A, Cvetkovic V, Reece K, Walters A, Hassan Y, Tummeti A, Torres B, Cooney D, Ellis M, Nikolopoulos DS. MARCO: A multi-agent system for optimizing HPC code generation using large language models. arXiv:2505.03906, 2025. — [arXiv:2505.03906](https://arxiv.org/abs/2505.03906) find
  107. Liu SW, Fang JY, Zhou H, Wang YX, Meng ZQ. SEW: Self-evolving agentic workflows for automated code generation. arXiv:2505.18646, 2025. — [arXiv:2505.18646](https://arxiv.org/abs/2505.18646) find
  108. Hu Y, Cai YZ, Du YX, Zhu XY, Liu XR, Yu ZJ, Hou YC, Tang S, Chen SH. Self-evolving multi-agent collaboration networks for software development. In: Proc. of the 13th Int’l Conf. on Learning Representations (ICLR). Singapore: OpenReview.net, 2025. 1–33. find
  109. Holt S, Luyten MR, van der Schaar M. L2MAC: Large language model automatic computer for extensive code generation. In: Proc. of the 12th Int’l Conf. on Learning Representations (ICLR). Vienna: OpenReview.net, 2024. 1–61. find
  110. Li YL, Li JD, Wang Q, Yang ML, Kong H, Wang SS. Cogito, ergo sum: A neurobiologically-inspired cognition-memory-growth system for code generation. arXiv:2501.18653, 2025. — [arXiv:2501.18653](https://arxiv.org/abs/2501.18653) find
  111. Qi DR, Miao ZJ, Wang JN. CleanAgent: Automating data standardization with LLM-based agents. arXiv:2403.08291, 2024. — [arXiv:2403.08291](https://arxiv.org/abs/2403.08291) find
  112. Ma YW, Cao RY, Cao YC, Zhang Y, Chen J, Liu YB, Liu YC, Li BH, Huang F, Li YB. Lingma SWE-GPT: An open development-process-centric language model for automated software improvement. arXiv:2411.00622, 2024. — [arXiv:2411.00622](https://arxiv.org/abs/2411.00622) find
  113. Guo XH, Wang XY, Chen YY, Li S, Han C, Li ML, Ji H. SyncMind: Measuring agent out-of-sync recovery in collaborative software engineering. In: Proc. of the 42nd Int’l Conf. on Machine Learning (ICML). Vancouver: OpenReview.net, 2025. 1–74. find
  114. Zhou A, Yan K, Shlapentokh-Rothman M, Wang HH, Wang YX. Language agent tree search unifies reasoning, acting, and planning in language models. In: Proc. of the 41st Int’l Conf. on Machine Learning (ICML). Vienna: OpenReview.net, 2024. 1–23. find
  115. Jin DM, Jin Z, Chen XH, Wang CH. MARE: Multi-agents collaboration framework for requirements engineering. arXiv:2405.03256, 2024. — [arXiv:2405.03256](https://arxiv.org/abs/2405.03256) find
  116. Jiang X, Dong YH, Fan ZY, Jin Z, Jiao WP, Li G. Exploring data-efficient adaptation of large language models for code generation. ACM Trans. Software Engineering Methodology, 2025. [doi: 10.1145/3772721] — [doi:10.1145/3772721](https://doi.org/10.1145/3772721) find
  117. Xu YH, Su HJ, Xing C, Mi BY, Liu Q, Shi WJ, Hui BY, Zhou F, Liu YT, Xie TB, Cheng ZJ, Zhao SH, Kong LP, Wang BL, Xiong CM, Yu T. Lemur: Harmonizing natural language and code for language agents. In: Proc. of the 12th Int’l Conf. on Learning Representations (ICLR). Vienna: OpenReview.net, 2024. 1–21. find
  118. Islam MA, Ali ME, Parvez MR. CodeSim: Multi-agent code generation and problem solving through simulation-driven planning and debugging. In: Findings of the Association for Computational Linguistics: NAACL 2025. Albuquerque: ACL, 2025. 5113–5139. [doi: 10.18653/v1/2025.findings-naacl.285] find
  119. Nguyen MH, Chau TP, Nguyen PX, Bui NDQ. AgileCoder: Dynamic collaborative agents for software development based on agile methodology. In: Proc. of the 2nd Int’l Conf. on AI Foundation Models and Software Engineering (FORGE). Ottawa: IEEE, 2025. 156–167. [doi: 10.1109/Forge66646.2025.00026] — [doi:10.1109/forge66646.2025.00026](https://doi.org/10.1109/forge66646.2025.00026) find
  120. Chen DK, Wang HB, Huo YH, Li YZ, Zhang HY. GameGPT: Multi-agent collaborative framework for game development. arXiv:2310.08067, 2023. find
  121. Zan DG, Yu AL, Liu W, Chen D, Shen B, Li W, Yao YF, Gong YS, Chen XL, Guan B, Yang ZG, Wang YJ, Wang QX, Cui LZ. CodeS: Natural language to code repository via multi-layer sketch. arXiv:2403.16443, 2024. find
  122. Zhang YT, Ruan HF, Fan ZY, Roychoudhury A. AutoCodeRover: Autonomous program improvement. In: Proc. of the 33rd ACM SIGSOFT Int’l Symp. on Software Testing and Analysis (ISSTA). Vienna: ACM, 2024. 1592–1604. [doi: 10.1145/3650212.3680384] — [doi:10.1145/3650212.3680384](https://doi.org/10.1145/3650212.3680384) find
  123. Yang J, Jimenez CE, Wettig A, Lieret K, Yao SY, Narasimhan K, Press O. SWE-agent: Agent-computer interfaces enable automated software engineering. In: Proc. of the 38th Int’l Conf. on Neural Information Processing Systems (NeurIPS). Vancouver: Curran Associates Inc., 2024. 1601. find our card
  124. Antoniades A, Örwall A, Zhang KX, Xie YX, Goyal A, Wang WY. SWE-search: Enhancing software agents with Monte Carlo tree search and iterative refinement. In: Proc. of the 13th Int’l Conf. on Learning Representations (ICLR). Singapore: OpenReview.net, 2025. 1–31. find
  125. Bouzenia I, Devanbu P, Pradel M. RepairAgent: An autonomous, LLM-based agent for program repair. In: Proc. of the 47th Int’l Conf. on Software Engineering (ICSE). Ottawa: IEEE, 2025. 2188–2200. [doi: 10.1109/ICSE55347.2025.00157] — [doi:10.1109/icse55347.2025.00157](https://doi.org/10.1109/icse55347.2025.00157) find
  126. Cen JP, Liu JX, Li ZX, Wang JJ. SQLFixAgent: Towards semantic-accurate text-to-SQL parsing via consistency-enhanced multi-agent collaboration. In: Proc. of the 39th AAAI Conf. on Artificial Intelligence. Philadelphia: AAAI, 2025. 49–57. [doi: 10.1609/aaai.v39i1.31979] — [doi:10.1609/aaai.v39i1.31979](https://doi.org/10.1609/aaai.v39i1.31979) find
  127. Yu ZM, Zhang HJ, Zhao YJ, Huang HX, Yao M, Ding K, Zhao JS. OrcaLoca: An LLM agent framework for software issue localization. In: Proc. of the 42nd Int’l Conf. on Machine Learning (ICML). Vancouver: OpenReview.net, 2025. 1–21. find
  128. Li HW, Tang YH, Wang SQ, Guo WB. PatchPilot: A stable and cost-efficient agentic patching framework. arXiv:2502.02747, 2025. — [arXiv:2502.02747](https://arxiv.org/abs/2502.02747) find
  129. Ma YW, Li YB, Dong YH, Jiang X, Cao RY, Chen J, Huang F, Li BH. Thinking longer, not larger: Enhancing software engineering agents via scaling test-time compute. arXiv:2503.23803, 2025. — [arXiv:2503.23803](https://arxiv.org/abs/2503.23803) find
  130. Ye H, Yang AZH, Hu C, Wang YL, Zhang T, Le Goues C. AdverIntent-agent: Adversarial reasoning for repair based on inferred program intent. Proc. of the ACM on Software Engineering, 2025, 2(ISSTA): ISSTA062. [doi: 10.1145/3728939] — [doi:10.1145/3728939](https://doi.org/10.1145/3728939) find
  131. Sohrabizadeh A, Song JL, Liu MJ, Roy R, Lee C, Raiman J, Catanzaro B. Nemotron-CORTEXA: Enhancing LLM agents for software engineering tasks via improved localization and solution diversity. In: Proc. of the 42nd Int’l Conf. on Machine Learning (ICML). Vancouver: OpenReview.net, 2025. 1–16. find
  132. Hu YX, Wang X, Wang YC, Zhang Y, Guo SY, Chen CY, Wang X, Zhou YF. AUITestAgent: Automatic requirements oriented GUI function testing. arXiv:2407.09018, 2024. — [arXiv:2407.09018](https://arxiv.org/abs/2407.09018) find
  133. Liu KB, Chen ZP, Liu YY, Zhang JM, Harman M, Han YD, Ma Y, Dong YH, Li G, Huang G. LLM-powered test case generation for detecting bugs in plausible programs. In: Proc. of the 63rd Annual Meeting of Association for Computational Linguistics (Vol. 1: Long Papers). Vienna: ACL, 2025. 430–440. [doi: 10.… find
  134. Fraser G, Arcuri A. EvoSuite: On the challenges of test case generation in the real world. In: Proc. of the 6th IEEE Int’l Conf. on Software Testing, Verification and Validation. Luxembourg: IEEE, 2013. 362–369. [doi: 10.1109/ICST.2013.51] — [doi:10.1109/icst.2013.51](https://doi.org/10.1109/icst.2013.51) find
  135. Bazalii M. Building AI agents to automate software test case creation. 2024. https://developer.nvidia.com/blog/building-ai-agents-to-automate-software-test-case-creation/ ↗
  136. Jiang ZY, Schmidt D, Srikanth D, Xu DX, Kaplan I, Jacenko D, Wu YX. AIDE: AI-driven exploration in the space of code. arXiv:2502.13138, 2025. find
  137. Pomian D, Bellur A, Dilhara M, Kurbatova Z, Bogomolov E, Sokolov A, Bryksin T, Dig D. EM-assist: Safe automated extractmethod refactoring with LLMs. In: Proc. of the 32nd Int’l Conf. on the Foundations of Software Engineering (FSE). Porto de Galinhas: ACM, 2024. 582–586. [doi: 10.1145/3663529.366380… — [doi:10.1145/3663529.3663803](https://doi.org/10.1145/3663529.3663803) find
  138. Wu D, Mu FW, Shi L, Guo ZQ, Liu K, Zhuang WG, Zhong YQ, Zhang L. iSMELL: Assembling LLMs with expert toolsets for code smell detection and refactoring. In: Proc. of the 39th Int’l Conf. on Automated Software Engineering (ICASE). Sacramento: ACM, 2024. 1345–1357. [doi: 10.1145/3691620.3695508] — [doi:10.1145/3691620.3695508](https://doi.org/10.1145/3691620.3695508) find
  139. Siddeeq S, Rasheed Z, Sami MA, Hasan M, Waseem M, Rasku J, Saari M, Kai-Kemell K, Abrahamsson P. Distributed approach to Haskell based applications refactoring with LLMs based multi-agent systems. arXiv:2502.07928, 2025. — [arXiv:2502.07928](https://arxiv.org/abs/2502.07928) find
  140. Fakhoury S, Naik A, Sakkas G, Chakraborty S, Lahiri SK. LLM-based test-driven interactive code generation: User study and empirical evaluation. IEEE Trans. on Software Engineering, 2024, 50(9): 2254–2268. [doi: 10.1109/TSE.2024.3428972] — [doi:10.1109/tse.2024.3428972](https://doi.org/10.1109/tse.2024.3428972) find
  141. Jia HX, Morris R, Ye H, Sarro F, Mechtaev S. Automated repair of ambiguous natural language requirements. arXiv:2505.07270, 2025. — [arXiv:2505.07270](https://arxiv.org/abs/2505.07270) find
  142. Vijayvargiya S, Zhou XH, Yerukola A, Sap M, Neubig G. Interactive agents to overcome ambiguity in software engineering. arXiv:2502.13069, 2025. — [arXiv:2502.13069](https://arxiv.org/abs/2502.13069) find
  143. González EA, Rothkopf R, Lerner S, Polikarpova N. HiLDe: Intentional code generation via human-in-the-loop decoding. arXiv:2505.22906, 2025. — [arXiv:2505.22906](https://arxiv.org/abs/2505.22906) find
  144. Austin J, Odena A, Nye M, Bosma M, Michalewski H, Dohan D, Jiang E, Cai C, Terry M, Le Q, Sutton C. Program synthesis with large language models. arXiv:2108.07732, 2021. find our card
  145. Dong YH, Ding JZ, Jiang X, Li G, Li Z, Jin Z. CodeScore: Evaluating code generation by learning code execution. ACM Trans. on Software Engineering and Methodology, 2025, 34(3): 77. [doi: 10.1145/3695991] — [doi:10.1145/3695991](https://doi.org/10.1145/3695991) find
  146. Liu JW, Xia CS, Wang YY, Zhang LM. Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation. In: Proc. of the 37th Int’l Conf. on Neural Information Processing Systems. New Orleans: Curran Associates Inc., 2023. 943. find our card
  147. Hendrycks D, Basart S, Kadavath S, Mazeika M, Arora A, Guo E, Burns C, Puranik S, He H, Song D, Steinhardt J. Measuring coding challenge competence with APPS. arXiv:2105.09938, 2021. find our card
  148. Jain N, Han K, Gu A, Li WD, Yan FJ, Zhang TJ, Wang SD, Solar-Lezama A, Sen K, Stoica I. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. In: Proc. of the 13th Int’l Conf. on Learning Representations (ICLR). Singapore: OpenReview.net, 2025. 1–41. find our card
  149. Dong YH, Jiang X, Liu HY, Jin Z, Gu B, Yang MF, Li G. Generalization or memorization: Data contamination and trustworthy evaluation for large language models. In: Findings of the Association for Computational Linguistics: ACL 2024. Bangkok: ACL, 2024. 12039–12050. [doi: 10.18653/v1/2024.findings-acl… find
  150. Jimenez CE, Yang J, Wettig A, Yao SY, Pei KX, Press O, Narasimhan KR. SWE-Bench: Can language models resolve real-world GitHub issues? In: Proc. of the 12th Int’l Conf. on Learning Representations (ICLR). Vienna: OpenReview.net, 2024. 1–51. find our card
  151. Xu K, Mao YW, Guan XY, Feng ZL. Web-Bench: A LLM code benchmark based on Web standards and frameworks. arXiv:2505.07473, 2025. find
  152. Aider. Aider LLM benchmarks: Quantitative evaluation of AI coding models. 2024. https://aider.chat/docs/benchmarks.html ↗
  153. Li J, Li G, Zhang XM, Dong YH, Jin Z. EvoCodeBench: An evolving code generation benchmark aligned with real-world code repositories. arXiv:2404.00599, 2024. — [arXiv:2404.00599](https://arxiv.org/abs/2404.00599) find
  154. Li J, Li G, Zhao YF, Li YM, Liu HY, Zhu H, Wang LC, Liu KB, Fang Z, Wang LS, Ding JZ, Zhang XM, Zhu YQ, Dong YH, Jin Z, Li BH, Huang F, Li YB, Gu B, Yang MF. DevEval: A manually-annotated code generation benchmark aligned with real-world code repositories. In: Findings of the Association for Computa… find
  155. Hu JW, Zheng WC, Liu YH, Liu Y. Optimizing token consumption in LLMs: A nano surge approach for code reasoning efficiency. arXiv:2504.15989, 2025. — [arXiv:2504.15989](https://arxiv.org/abs/2504.15989) find
  156. Zhao CG, Deng CQ, Ruan C, Dai DM, Gao HZ, Li JS, Zhang LY, Huang PP, Zhou SY, Ma SR, Liang WF, He Y, Wang YQ, Liu YX, Wei YX. Insights into DeepSeek-V3: Scaling challenges and reflections on hardware for AI architectures. In: Proc. of the 52nd Annual Int’l Symp. on Computer Architecture (ISCA). Toky… find
  157. Zhang KC, Zhang HZ, Li G, You JL, Li J, Zhao YF, Jin Z. SEAlign: Alignment training for software engineering agent. arXiv: 2503.18455, 2025. — [arXiv:2503.18455](https://arxiv.org/abs/2503.18455) find
  158. Jiang X, Dong YH, Liu MY, Deng HY, Wang T, Tao YD, Cao RY, Li BH, Jin Z, Jiao WP, Huang F, Li YB, Li G. CodeRL+: Improving code generation via reinforcement with execution semantics alignment. arXiv:2510.18471, 2025. find
  159. Lee H, Zhang ZQ, Lu HX, Zhang LM. SEC-bench: Automated benchmarking of LLM agents on real-world software security tasks. arXiv:2506.11791, 2025. — [arXiv:2506.11791](https://arxiv.org/abs/2506.11791) find
  160. Siddeeq S, Waseem M, Rasheed Z, Hasan MM, Rasku J, Saari M, Terho H, Makela K, Kemell KK, Abrahamsson P. LLM-based multi-agent system for intelligent refactoring of Haskell code. arXiv:2506.19481, 2025. — [arXiv:2506.19481](https://arxiv.org/abs/2506.19481) find
  161. Chen Z, Jiang LX. Evaluating software development agents: Patch patterns, code quality, and issue complexity in real-world GitHub scenarios. In: Proc. of the 2025 IEEE Int’l Conf. on Software Analysis, Evolution and Reengineering (SANER). Montreal: IEEE, 2025. 657–668. [doi: 10.1109/SANER64311.2025.… — [doi:10.1109/saner64311.2025.00068](https://doi.org/10.1109/saner64311.2025.00068) find
  162. GitHub. GitHub Copilot: Your AI pair programmer. 2021. https://github.com/copilot ↗
  163. Cognition AI. Devin: The first AI software engineer. 2024. https://devin.ai ↗
  164. Anysphere, Inc. Cursor: The AI-first code editor. 2023. https://www.cursor.com ↗
  165. Alibaba Cloud. Tongyi Lingma: Intelligent coding assistant tool based on Tongyi large model. 2023 (in Chinese). https://lingma.aliyun.com ↗
  166. Anthropic. Claude Code: A command-line tool for agentic coding. 2024. https://www.anthropic.com/claude-code ↗
  167. Hu EJ, Shen YL, Wallis P, Allen-Zhu Z, Li YZ, Wang SA, Wang L, Chen WZ. LoRA: Low-rank adaptation of large language models. In: Proc. of the 10th Int’l Conf. on Learning Representations (ICLR). OpenReview.net, 2022. 1–13. find
  168. Hua WY, Yang XJ, Jin MY, Li ZL, Cheng W, Tang RX, Zhang YF. TrustAgent: Towards safe and trustworthy LLM-based agents. In: Findings of the Association for Computational Linguistics: EMNLP 2024. Miami: ACL, 2024. 10000–10016. [doi: 10.18653/v1/2024. findings-emnlp.585] find
  169. Zhang RS, Javidnia N, Sheybani N, Koushanfar F. Robust and secure code watermarking for large language models via ML/Crypto codesign. arXiv:2502.02068, 2025. 附中文参考文献 [165] 阿里云. 通义灵码 (Tongyi Lingma): 基于通义大模型的智能编码辅助工具. 2023. https://lingma.aliyun.com 作者简介 董益宏, 博士生, CCF 学生会员, 主要研究领域为智能化软件工程, 多智能体, 大语言模… — [arXiv:2502.02068](https://arxiv.org/abs/2502.02068) ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dong-2025-code-generation-llm/