Elamur Research⌕
☰
home / papers / 2023 / Self-Edit: Fault-Aware Code Editor for Code Generation

Self-Edit: Fault-Aware Code Editor for Code Generation

Paper · 2023 · Kechi Zhang, Zhuo Li, Jia Li et al.

Summary

Large language models still solve few competitive programming problems when only one submission per task is allowed. For example, GPT3 reaches roughly 7% pass@1 on APPS-test. The authors propose Self-Edit, a generate-and-edit pipeline modeled on how human programmers debug.

First, a frozen LLM, used as a black box, writes a program from the problem statement. The program is then run on the example test case that comes with the task. The outcome (passed, wrong answer, or an error such as a syntax fault, runtime exception or timeout) is converted into a templated comment. This comment carries details like the failing line, the error message, or expected versus actual output. A comparatively small fault-aware neural editor then takes the problem, the draft code and this comment, and outputs a revised program.

The method was evaluated on APPS and HumanEval with nine LLMs ranging from 110M to 175B parameters. It markedly raises average pass@1 and outperforms reranking-based post-processing while keeping a small, fixed sample budget. This matters because it improves generated code without retraining the LLM or drawing large numbers of candidate samples.

Key points

  • Relative gains in average pass@1 over direct generation are 89% on APPS-dev, 31% on APPS-test and 48% on HumanEval. The HumanEval result is out-of-distribution, which suggests the editor transfers to a different benchmark style.
  • Even for the largest model, GPT3-175B (text-davinci-002), the editor lifts APPS-dev pass@1 from 26.6% to 32.4%.
  • Execution results fall into three categories, each with its own comment template. On APPS-train, small models such as PyCodeGPT-110M produce SyntaxErrors more often, while GPT3 yields more programs that run normally.
  • Prior reranking approaches need heavy sampling (AlphaCode uses about 1 million samples per problem). Examples include the fault-aware neural ranker of Inala et al. (2022), CodeT and AlphaCode-style filtering. The authors present Self-Edit as the first editing-based post-processing method for competitive programming.
  • The authors list several limitations:
    • The editor uses small pretrained models.
    • The editor training set is built from only 10 LLM samples per problem.
    • No strict compute comparison with other methods is provided.
    • Multi-round editing suffers from a train/test mismatch.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China315 works · 43,267 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksInformation Systems · 1,730 worksSoftware · 119 worksComputer Science Applications · 131 works

🧭 Topics 1Research subjects and languages the work is about.Code & programming languages · 168 works

Cite

Cite as Kechi Zhang, Zhuo Li, Jia Li, Ge Li and Zhi Jin. 2023. Self-Edit: Fault-Aware Code Editor for Code Generation. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.acl-long.45 · arXiv 2305.04087

@inproceedings{zhang2023self,
  title = {{Self-Edit: Fault-Aware Code Editor for Code Generation}},
  author = {Kechi Zhang, Zhuo Li, Jia Li, Ge Li and Zhi Jin},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.45},
  eprint = {2305.04087},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.acl-long.45}
}

References

22 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018. Mapping language to code in programmatic context. arXiv preprint arXiv:1808.09588.
  2. Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. 2019. Spoc: Search-based pseudocode to code. Advances in Neural Information Processing Systems, 32.
  3. Shuvendu K. Lahiri, Aaditya Naik, Georgios Sakkas, Piali Choudhury, Curtis von Veh, Madanlal Musuvathi, Jeevana Priya Inala, Chenglong Wang, and Jianfeng Gao. 2022. Interactive code generation via test-driven user-intent formalization. CoRR, abs/2208.05950. — [arXiv:2208.05950](https://arxiv.org/abs/2208.05950) find
  4. Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu-Hong Hoi. 2022. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. In NeurIPS.
  5. Jia Li, Ge Li, Yongmin Li, and Zhi Jin. 2023a. Enabling programming thinking in large language models toward code generation. arXiv preprint arXiv:2305.06599. — [arXiv:2305.06599](https://arxiv.org/abs/2305.06599) find
  6. Jia Li, Ge Li, Zhuo Li, Zhi Jin, Xing Hu, Kechi Zhang, and Zhiyi Fu. 2022a. Codeeditor: Learning to edit source code with pre-trained models. arXiv preprint arXiv:2210.17040. — [arXiv:2210.17040](https://arxiv.org/abs/2210.17040) find
  7. Jia Li, Yongmin Li, Ge Li, Zhi Jin, Yiyang Hao, and Xing Hu. 2023b. Skcoder: A sketch-based approach for automatic code generation. arXiv preprint arXiv:2302.06144. — [arXiv:2302.06144](https://arxiv.org/abs/2302.06144) find
  8. Jia Li, Chongyang Tao, Huang Hu, Can Xu, Yining Chen, and Daxin Jiang. 2022b. Unsupervised cross-domain adaptation for response selection using self-supervised and adversarial training. In WSDM ’22: The Fifteenth ACM International Conference on Web Search and Data Mining, Virtual Event / Tempe, AZ,… find
  9. Jia Li, Yunfei Zhao, Yongmin Li, Ge Li, and Zhi Jin. 2023c. Towards enhancing in-context learning for code generation. arXiv preprint arXiv:2303.17780. — [arXiv:2303.17780](https://arxiv.org/abs/2303.17780) find
  10. Yujia Li, David H. Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, Jame… — Competition-Level Code Generation with AlphaCode
  11. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. A conversational paradigm for program synthesis. CoRR, abs/2203.13474. find
  12. Yusuke Oda, Hiroyuki Fudaba, Graham Neubig, Hideaki Hata, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. 2015. Learning to generate pseudo-code from source code using statistical machine translation. In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 57… find
  13. Richard Yuanzhe Pang and He He. 2021. Text generation by learning from demonstrations. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. find
  14. Jianhao Shen, Yichun Yin, Lin Li, Lifeng Shang, Xin Jiang, Ming Zhang, and Qun Liu. 2021. Generate & rank: A multi-task framework for math word problems. In Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021, pa… find
  15. Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida I. Wang. 2022. Natural language to code translation with execution. CoRR, abs/2204.11454.
  16. Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/ mesh-transformer-jax. ↗
  17. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… — Transformers: State-of-the-Art Natural Language Processing
  18. Frank F. Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. 2022. A systematic evaluation of large language models of code. In MAPS@PLDI 2022: 6th ACM SIGPLAN International Symposium on Machine Programming, San Diego, CA, USA, 13 June 2022, pages 1–10. ACM.
  19. Pengcheng Yin and Graham Neubig. 2018. Tranx: A transition-based neural abstract syntax parser for semantic parsing and code generation. arXiv preprint arXiv:1810.02720. find
  20. Daoguang Zan, Bei Chen, Dejian Yang, Zeqi Lin, Minsu Kim, Bei Guan, Yongji Wang, Weizhu Chen, and Jian-Guang Lou. 2022. CERT: Continual pre-training on sketches for library-oriented code generation. In The 2022 International Joint Conference on Artificial Intelligence. find
  21. Kechi Zhang, Ge Li, Jia Li, Zhuo Li, and Zhi Jin. 2023. Toolcoder: Teach code generation models to use api search tools. ArXiv, abs/2305.04032. — [arXiv:2305.04032](https://arxiv.org/abs/2305.04032) find
  22. Tianyi Zhang, Tao Yu, Tatsunori B. Hashimoto, Mike Lewis, Wen-tau Yih, Daniel Fried, and Sida I. Wang. 2022. Coder reviewer reranking for code generation. CoRR, abs/2211.16490. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2023-self-edit-fault-aware-code/