Elamur Research⌕
☰
home / papers / 2024 / CodeS: Natural Language to Code Repository via Multi-Layer Sketch

CodeS: Natural Language to Code Repository via Multi-Layer Sketch

Paper · 2024 · Daoguang Zan, Ailun Yu, Wei Liu et al.

Summary

The paper introduces NL2Repo, a task in which a model must build a whole software repository from a natural-language requirements document, here a project's README.md. These inputs are long and high-level. The output must stay structurally consistent across folders, files and interfaces, which autoregressive code LLMs struggle with when generating everything in one pass. The authors propose CodeS, which splits the job into three layers. RepoSketcher drafts the directory tree with import hints. FileSketcher writes a skeleton for each file, containing imports and function definitions with empty bodies. SketchFiller then completes each function body. Every stage receives the outputs of the earlier ones. CodeS is implemented in two ways: by prompting existing models (StarCoder2, CodeLlama, DeepSeekCoder, GPT-3.5) and by supervised fine-tuning. For evaluation, the authors build SketchEval, a benchmark of 19 recent GitHub repositories, and SketchBLEU, a repository-level adaptation of CodeBLEU. They also release a VSCode plugin, which 30 participants tested. Benchmark results and user studies indicate that CodeS is effective and practical, and the work makes repository-scale generation measurable.

Key points

  • SketchEval and SketchBLEU. SketchEval splits its 19 repositories into Easy (5), Middle (8) and Hard (6). SketchBLEU changes CodeBLEU in four ways:
    • BLEU is computed over all files concatenated.
    • The brevity penalty is softer.
    • A Match_struc term compares n-hop subtrees of a combined directory-plus-syntax tree.
    • Dataflow matching works at repository level via maximum weighted bipartite matching of functions.
  • Fine-tuned CodeS results. With CodeLlama, overall SketchBLEU grows with model size: 54.39 (7B), 57.23 (13B) and 63.31 (34B). Generating files in dependency order gives small extra gains, such as 63.87 for 34B, mostly on Hard repositories.
  • Transfer to other languages. A RepoSketcher fine-tuned only on Python (CodeLlama 13B) beats prompt engineering on other languages. Repository-sketch BLEU rises from 46.25 to 57.36 for Java, 36.68 to 50.14 for C++, and 62.88 to 71.57 for Go.
  • User studies. 30 non-author participants built two real projects, one of them a Blog app, using the plugin. In a survey of 6 users of CodeS Extended, most rated it effective. One rated it ineffective, finding its product design weaker than GitHub Copilot.
  • Availability. Code, the evaluation scripts and the multilingual evaluation sets are public at github.com/NL2Code/CodeS.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China244 works · 39,359 citations🏛️Chinese Academy of SciencesBeijing · China194 works · 23,906 citations🏢Huawei TechnologiesShenzhen · China75 works · 12,975 citations🎓Shandong UniversityJinan · China17 works · 789 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

Cite

Cite as Daoguang Zan, Ailun Yu, Wei Liu, Dong Chen, Bo Shen, Wei Li, Yafen Yao, Yongshun Gong, Xiaolin Chen, Bei Guan, Zhiguang Yang, Yongji Wang, Qianxiang Wang and Lizhen Cui. 2024. CodeS: Natural Language to Code Repository via Multi-Layer Sketch. arXiv:2403.16443.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2403.16443 · arXiv 2403.16443

@misc{zan2024codes,
  title = {{CodeS: Natural Language to Code Repository via Multi-Layer Sketch}},
  author = {Daoguang Zan, Ailun Yu, Wei Liu, Dong Chen, Bo Shen, Wei Li, Yafen Yao, Yongshun Gong, Xiaolin Chen, Bei Guan, Zhiguang Yang, Yongji Wang, Qianxiang Wang and Lizhen Cui},
  year = {2024},
  doi = {10.48550/arxiv.2403.16443},
  eprint = {2403.16443},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.16443}
}

References

69 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Zan D, Chen B, Zhang F, et al. Large language models meet nl2code: A survey[C]//Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023: 7443-7464. find
  2. Chen M, Tworek J, Jun H, et al. Evaluating large language models trained on code[J]. arXiv preprint arXiv:2107.03374, 2021.
  3. Li Y, Choi D, Chung J, et al. Competition-level code generation with alphacode[J]. Science, 2022, 378(6624): 1092-1097.
  4. Nijkamp E, Pang B, Hayashi H, et al. CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis[C]//The Eleventh International Conference on Learning Representations. 2022. find
  5. Li R, Allal L B, Zi Y, et al. Starcoder: may the source be with you![J]. arXiv preprint arXiv:2305.06161, 2023. find
  6. Lozhkov A, Li R, Allal L B, et al. StarCoder 2 and The Stack v2: The Next Generation[J]. arXiv preprint arXiv:2402.19173, 2024. find
  7. Roziere B, Gehring J, Gloeckle F, et al. Code llama: Open foundation models for code[J]. arXiv preprint arXiv:2308.12950, 2023.
  8. Guo D, Zhu Q, Yang D, et al. DeepSeek-Coder: When the Large Language Model Meets Programming–The Rise of Code Intelligence[J]. arXiv preprint arXiv:2401.14196, 2024. find
  9. Achiam J, Adler S, Agarwal S, et al. Gpt-4 technical report[J]. arXiv preprint arXiv:2303.08774, 2023.
  10. Anthropic. The Claude 3 Model Family: Opus, Sonnet, Haiku. find
  11. Ridnik T, Kredo D, Friedman I. Code Generation with Alpha-Codium: From Prompt Engineering to Flow Engineering. arXiv preprint arXiv:2401.08500. 2024 Jan 16. find
  12. Liu T, Jiang Y E, Monath N, et al. Autoregressive Structured Prediction with Language Models[C]//Findings of the Association for Computational Linguistics: EMNLP 2022. 2022: 993-1005. find
  13. Wells C. A generalization of the concept of sketch[J]. Theoretical Computer Science, 1990, 70(1): 159-178. find
  14. Tovey M, Porter S, Newman R. Sketching, concept development and automotive design[J]. Design studies, 2003, 24(2): 135-153. find
  15. Zan D, Chen B, Yang D, et al. CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generation[J]. Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, 2022: 2369-2375. find
  16. Solar-Lezama A. Program synthesis by sketching[M]. University of California, Berkeley, 2008. find
  17. Ellis K, Wong C, Nye M, Sablé-Meyer M, Morales L, Hewitt L, Cary L, Solar-Lezama A, Tenenbaum JB. Dreamcoder: Bootstrapping inductive program synthesis with wake-sleep library learning. InProceedings of the 42nd acm sigplan international conference on programming language design and implementation… find
  18. Levinson R. A general programming language for unified planning and control[J]. Artificial Intelligence, 1995, 76(1-2): 319-375. find
  19. Mou L, Li G, Zhang L, et al. Convolutional neural networks over tree structures for programming language processing[C]//Proceedings of the AAAI conference on artificial intelligence. 2016, 30(1). find
  20. Neelakantan A, Le Q V, Abadi M, et al. Learning a Natural Language Interface with Neural Programmer[C]//International Conference on Learning Representations. 2016. find
  21. Ouyang L, Wu J, Jiang X, et al. Training language models to follow instructions with human feedback[J]. Advances in neural information processing systems, 2022, 35: 27730-27744.
  22. Lu S, Guo D, Ren S, et al. CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation[C]//Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1). 2021. find
  23. Du X, Liu M, Wang K, et al. Evaluating Large Language Models in Class-Level Code Generation[C]//2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE). IEEE Computer Society, 2024: 865-865. find
  24. Qian C, Cong X, Yang C, et al. Communicative agents for software development[J]. arXiv preprint arXiv:2307.07924, 2023.
  25. Yang H, Yue S, He Y. Auto-gpt for online decision making: Benchmarks and additional opinions[J]. arXiv preprint arXiv:2306.02224, 2023. find
  26. https://github.com/reworkd/AgentGPT ↗
  27. Davis A M, Bersoff E H, Comer E R. A strategy for comparing alternative software development life cycle models[J]. IEEE Transactions on software Engineering, 1988, 14(10): 1453-1461. find
  28. Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen, Zihan Wang, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. 2023. CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X. In Proceedings of the 29th ACM SIGKDD Conference on…
  29. Fried D, Aghajanyan A, Lin J, et al. InCoder: A Generative Model for Code Infilling and Synthesis[C]//The Eleventh International Conference on Learning Representations. 2022. find
  30. Wang Y, Wang W, Joty S, et al. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021: 8696-8708. find
  31. Christopoulou F, Lampouras G, Gritta M, et al. Pangu-coder: Program synthesis with function-level language modeling[J]. arXiv preprint arXiv:2207.11280, 2022.
  32. Dou S, Liu Y, Jia H, et al. StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback[J]. arXiv preprint arXiv:2402.01391, 2024. — [arXiv:2402.01391](https://arxiv.org/abs/2402.01391) find
  33. Luo Z, Xu C, Zhao P, et al. WizardCoder: Empowering Code Large Language Models with Evol-Instruct[C]//The Twelfth International Conference on Learning Representations. 2023. find
  34. Shen B, Zhang J, Chen T, et al. Pangu-coder2: Boosting large language models for code with ranking feedback[J]. arXiv preprint arXiv:2307.14936, 2023. — [arXiv:2307.14936](https://arxiv.org/abs/2307.14936) find
  35. Zheng W, Sharan SP, Jaiswal AK, Wang K, Xi Y, Xu D, Wang Z. Outline, then details: Syntactically guided coarse-to-fine code generation. InInternational Conference on Machine Learning 2023 Jul 3 (pp. 42403-42419). PMLR. find
  36. Zelikman E, Huang Q, Poesia G, Goodman N, Haber N. Parsel: Algorithmic Reasoning with Language Models by Composing Decompositions. Advances in Neural Information Processing Systems. 2024 Feb 13;36. find
  37. Nguyen N, Nadi S. An empirical evaluation of GitHub copilot’s code suggestions[C]//Proceedings of the 19th International Conference on Mining Software Repositories. 2022: 1-5. find
  38. Dakhel A M, Majdinasab V, Nikanjam A, et al. Github copilot ai pair programmer: Asset or liability?[J]. Journal of Systems and Software, 2023, 203: 111734. find
  39. Yu H, Shen B, Ran D, Zhang J, Zhang Q, Ma Y, Liang G, Li Y, Wang Q, Xie T. Codereval: A benchmark of pragmatic code generation with generative pre-trained models. InProceedings of the 46th IEEE/ACM International Conference on Software Engineering 2024 Feb 6 (pp. 1-12). find
  40. Jimenez CE, Yang J, Wettig A, Yao S, Pei K, Press O, Narasimhan KR. SWE-bench: Can Language Models Resolve Real-world Github Issues?. InThe Twelfth International Conference on Learning Representations 2023 Oct 13.
  41. Iyer S, Konstas I, Cheung A, et al. Mapping Language to Code in Programmatic Context[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018: 1643-1652. find
  42. Yin P, Deng B, Chen E, et al. Learning to mine aligned code and natural language pairs from stack overflow[C]//Proceedings of the 15th international conference on mining software repositories. 2018: 476-486.
  43. Du X, Liu M, Wang K, et al. Evaluating Large Language Models in Class-Level Code Generation[C]//2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE). IEEE Computer Society, 2024: 865-865. find
  44. Ding Y, Wang Z, Ahmad W, et al. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion[J]. Advances in Neural Information Processing Systems, 2024, 36. find
  45. Liu T, Xu C, McAuley J. RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems[C]//The Twelfth International Conference on Learning Representations. 2023. find
  46. Zhang F, Chen B, Zhang Y, et al. RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation[C]//Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023: 2471-2484. find
  47. Oda Y, Fudaba H, Neubig G, et al. Learning to generate pseudo-code from source code using statistical machine translation[C]//2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2015: 574-584. find
  48. Austin J, Odena A, Nye M, et al. Program synthesis with large language models[J]. arXiv preprint arXiv:2108.07732, 2021.
  49. Hendrycks D, Basart S, Kadavath S, et al. Measuring Coding Challenge Competence With APPS[C]//Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). 2021.
  50. Cassano F, Gouwar J, Nguyen D, et al. MultiPL-E: a scalable and polyglot approach to benchmarking neural code generation[J]. IEEE Transactions on Software Engineering, 2023. find
  51. Lai Y, Li C, Wang Y, et al. DS-1000: A natural and reliable benchmark for data science code generation[C]//International Conference on Machine Learning. PMLR, 2023: 18319-18345. find
  52. Ren S, Guo D, Lu S, et al. Codebleu: a method for automatic evaluation of code synthesis[J]. arXiv preprint arXiv:2009.10297, 2020. find
  53. Papineni K, Roukos S, Ward T, et al. Bleu: a method for automatic evaluation of machine translation[C]//Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 2002: 311-318.
  54. https://github.com/hiyouga/LLaMA-Factory ↗
  55. Wolf T, Debut L, Sanh V, et al. Huggingface’s transformers: State-ofthe-art natural language processing[J]. arXiv preprint arXiv:1910.03771, 2019.
  56. Loshchilov I, Hutter F. Decoupled Weight Decay Regularization[C]//International Conference on Learning Representations. 2018. find
  57. Kwon W, Li Z, Zhuang S, et al. Efficient memory management for large language model serving with pagedattention[C]//Proceedings of the 29th Symposium on Operating Systems Principles. 2023: 611-626.
  58. Su J, Ahmed M, Lu Y, et al. Roformer: Enhanced transformer with rotary position embedding[J]. Neurocomputing[J]. Advances in Neural Information Processing Systems, 2024, 568: 127063.
  59. Yen H, Gao T, Chen D. Long-Context Language Modeling with Parallel Context Encoding. arXiv preprint arXiv:2402.16617. 2024 Feb 26.
  60. Chen Y, Qian S, Tang H, et al. LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models[C]//The Twelfth International Conference on Learning Representations. 2023.
  61. Zhang P, Liu Z, Xiao S, Shao N, Ye Q, Dou Z. Soaring from 4K to 400K: Extending LLM’s Context with Activation Beacon. arXiv preprint arXiv:2401.03462. 2024 Jan 7. — Long Context Compression with Activation Beacon
  62. Guo D, Ren S, Lu S, et al. GraphCodeBERT: Pre-training Code Representations with Data Flow[C]//International Conference on Learning Representations. 2020.
  63. Huang D, Nan Z, Hu X, Jin P, Peng S, Wen Y, Zhang R, Du Z, Guo Q, Pu Y, Chen Y. ANPL: Towards Natural Programming with Interactive Decomposition. Advances in Neural Information Processing Systems. 2024 Feb 13;36. find
  64. Li J, Li Y, Li G, et al. Skcoder: A sketch-based approach for automatic code generation[C]//2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023: 2124-2135. find
  65. Ge S, Goswami V, Zitnick C L, et al. Creative sketch generation[J]. arXiv preprint arXiv:2011.10039, 2020. find
  66. Lozhkov A, Li R, Allal LB, Cassano F, Lamy-Poirier J, Tazi N, Tang A, Pykhtar D, Liu J, Wei Y, Liu T. StarCoder 2 and The Stack v2: The Next Generation. arXiv preprint arXiv:2402.19173. 2024 Feb 29. find
  67. Zhang B, Liu Z, Cherry C, et al. When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method[C]//The Twelfth International Conference on Learning Representations. 2023. find
  68. Hall M W, Kennedy K. Efficient call graph analysis[J]. ACM Letters on Programming Languages and Systems (LOPLAS), 1992, 1(3): 227-242. find
  69. Scott Wu. Introducing Devin, the first AI software engineer. https://www.cognition-labs.com/introducing-devin ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zan-2024-codes-natural-language-code-repository/