Elamur Research
☰
home / papers / 2024 / CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution

CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2401.03065

Summary

Code language models are usually judged on HumanEval or MBPP. Those benchmarks test whether a model can write short programs from a natural-language description, but they do not test whether it understands what a piece of code does when it runs. CRUXEval fills that gap. It has 800 short Python functions, each paired with one input and the output it produces. Two tasks come from these pairs: CRUXEval-O asks the model to predict the output, and CRUXEval-I asks it to find an input that produces a given output. An answer counts as correct only if it passes an executed assertion. The authors had Code Llama 34B generate many candidate functions and inputs, kept only simple, cheap-to-run programs, and then picked a sample size that lets models be told apart reliably. They tested twenty models. GPT-4 scored best but is still far from solving the benchmark, and open models trail it by a wide margin. Models fine-tuned on GPT-4-style distilled data score high on HumanEval but gain little here. This shows that writing code and reasoning about how it executes are different skills.

Key points

  • The benchmark has 800 functions of 3–13 lines with no complex arithmetic. They were made by a reusable generate-then-filter process that can produce harder versions later.
  • Without chain of thought (CoT), GPT-4 reaches pass@1 of about 67% on CRUXEval-I and 63% on CRUXEval-O. With CoT it rises to 75% and 81%. The best open-source models reach only about 47% and 44%.
  • Among base models, higher HumanEval scores go with higher CRUXEval scores. WizardCoder, Phind and Phi break this pattern: they score well on HumanEval but no better than Code Llama here.
  • CoT prompting and fine-tuning on input–output assertions both help, but neither comes close to solving the benchmark. Per-model standard deviations of about 1.5% mean small score gaps between models are not meaningful.
  • GPT-4 repeatedly fails on some very simple programs, such as chained rstrip calls and list-append loops. Output prediction lets you pinpoint which CoT step went wrong.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Computing & GPUs

Cited by

107 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 102 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2401.03065 · arXiv 2401.03065

@misc{gu2024cruxeval,
  title = {{CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution}},
  author = {Gu, Alex and Rozière, Baptiste and Leather, Hugh and Solar-Lezama, Armando and Synnaeve, Gabriel and Wang, Sida I.},
  year = {2024},
  doi = {10.48550/arxiv.2401.03065},
  eprint = {2401.03065},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.03065}
}

References

109 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agashe, R., Iyer, S., and Zettlemoyer, L. Juice: A large scale distantly supervised dataset for open domain context-based code generation. arXiv preprint arXiv:1910.02216, 2019. (Cited on pg. 3) find
  2. Ahmad, W. U., Tushar, M. G. R., Chakraborty, S., and Chang, K.-W. Avatar: A parallel corpus for java-python program translation. arXiv preprint arXiv:2108.11590, 2021. (Cited on pg. 4) find
  3. AI, D. Deepseek coder: Let the code write itself. DeepSeek-Coder, 2023. (Cited on pg. 3) find
  4. Allal, L. B., Li, R., Kocetkov, D., Mou, C., Akiki, C., Ferrandis, C. M., Muennighoff, N., Mishra, M., Gu, A., Dey, M., et al. Santacoder: don’t reach for the stars! arXiv preprint arXiv:2301.03988, 2023. (Cited on pg. 3) find our card
  5. Alon, U., Brody, S., Levy, O., and Yahav, E. code2seq: Generating sequences from structured representations of code. arXiv preprint arXiv:1808.01400, 2018. (Cited on pg. 4) find
  6. Arkoudas, K. Gpt-4 can’t reason. arXiv preprint arXiv:2308.03762, 2023. (Cited on pg. 4) find
  7. Athiwaratkun, B., Gouda, S. K., Wang, Z., Li, X., Tian, Y., Tan, M., Ahmad, W. U., Wang, S., Sun, Q., Shang, M., et al. Multi-lingual evaluation of code generation models. arXiv preprint arXiv:2210.14868, 2022. (Cited on pg. 3) find
  8. Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. (Cited on pg. 2, 3, 4) find our card
  9. Barone, A. V. M. and Sennrich, R. A parallel corpus of python functions and documentation strings for automated code documentation and code generation. arXiv preprint arXiv:1707.02275, 2017. (Cited on pg. 4) find
  10. Berabi, B., He, J., Raychev, V., and Vechev, M. Tfix: Learning to fix coding errors with a text-to-text transformer. In International Conference on Machine Learning, pp. 780–791. PMLR, 2021. (Cited on pg. 4) https://github.com/deepseek-ai/ ↗
  11. Berglund, L., Tong, M., Kaufmann, M., Balesni, M., Stickland, A. C., Korbak, T., and Evans, O. The reversal curse: Llms trained on” a is b” fail to learn” b is a”. arXiv preprint arXiv:2309.12288, 2023. (Cited on pg. 42) find
  12. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. (Cited on pg. 2, 7) find
  13. Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023. (Cited on pg. 4) find our card
  14. Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Anderson, C. J., Feldman, M. Q., et al. Multipl-e: A scalable and extensible approach to benchmarking neural code generation. arXiv preprint arXiv:2208.08227, 2022. (Cited on pg. 3) find
  15. Chen, B., Zhang, F., Nguyen, A., Zan, D., Lin, Z., Lou, J.-G., and Chen, W. Codet: Code generation with generated tests. arXiv preprint arXiv:2207.10397, 2022. (Cited on pg. 4) find
  16. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. (Cited on pg. 2, 3) find our card
  17. Chen, X., Lin, M., Schärli, N., and Zhou, D. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128, 2023. (Cited on pg. 4, 16) find
  18. Ding, Y., Wang, Z., Ahmad, W. U., Ramanathan, M. K., Nallapati, R., Bhatia, P., Roth, D., and Xiang, B. Cocomic: Code completion by jointly modeling in-file and cross-file context. arXiv preprint arXiv:2212.10007, 2022. (Cited on pg. 3) find
  19. Dziri, N., Lu, X., Sclar, M., Li, X. L., Jian, L., Lin, B. Y., West, P., Bhagavatula, C., Bras, R. L., Hwang, J. D., et al. Faith and fate: Limits of transformers on compositionality. arXiv preprint arXiv:2305.18654, 2023. (Cited on pg. 4) find our card
  20. Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., and Zhang, J. M. Large language models for software engineering: Survey and open problems. arXiv preprint arXiv:2310.03533, 2023. (Cited on pg. 2) — [arXiv:2310.03533](https://arxiv.org/abs/2310.03533) find
  21. Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., tau Yih, W., Zettlemoyer, L., and Lewis, M. Incoder: A generative model for code infilling and synthesis. preprint arXiv:2204.05999, 2022. (Cited on pg. 3) find
  22. Garg, S., Moghaddam, R. Z., Clement, C. B., Sundaresan, N., and Wu, C. Deepperf: A deep learning-based approach for improving software performance. arXiv preprint arXiv:2206.13619, 2022. (Cited on pg. 4) — [arXiv:2206.13619](https://arxiv.org/abs/2206.13619) find
  23. Giannou, A., Rajput, S., Sohn, J.-y., Lee, K., Lee, J. D., and Papailiopoulos, D. Looped transformers as programmable computers. arXiv preprint arXiv:2301.13196, 2023. (Cited on pg. 4) find
  24. Gudibande, A., Wallace, E., Snell, C., Geng, X., Liu, H., Abbeel, P., Levine, S., and Song, D. The false promise of imitating proprietary llms. arXiv preprint arXiv:2305.15717, 2023. (Cited on pg. 7) find our card
  25. Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C. C. T., Del Giorno, A., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O., et al. Textbooks are all you need. arXiv preprint arXiv:2306.11644, 2023. (Cited on pg. 3, 6, 7) find
  26. Gupta, R., Pal, S., Kanade, A., and Shevade, S. Deepfix: Fixing common c language errors by deep learning. In Proceedings of the aaai conference on artificial intelligence, volume 31, 2017. (Cited on pg. 4) find
  27. Haluptzok, P., Bowers, M., and Kalai, A. T. Language models can teach themselves to program better. arXiv preprint arXiv:2207.14502, 2022. (Cited on pg. 4) find
  28. Haque, M. M. A., Ahmad, W. U., Lourentzou, I., and Brown, C. Fixeval: Execution-based evaluation of program fixes for competitive programming problems. 2022. (Cited on pg. 4) find
  29. Hasan, M., Muttaqueen, T., Ishtiaq, A. A., Mehrab, K. S., Haque, M. M. A., Hasan, T., Ahmad, W. U., Iqbal, A., and Shahriyar, R. Codesc: A large code-description parallel dataset. arXiv preprint arXiv:2105.14220, 2021. (Cited on pg. 4) find
  30. Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., et al. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938, 2021. (Cited on pg. 3) find our card
  31. Hosseini, A., Reddy, S., Bahdanau, D., Hjelm, R. D., Sordoni, A., and Courville, A. Understanding by understanding not: Modeling negation in language models. arXiv preprint arXiv:2105.03519, 2021. (Cited on pg. 4) find
  32. Husain, H., Wu, H.-H., Gazit, T., Allamanis, M., and Brockschmidt, M. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436, 2019. (Cited on pg. 4) find our card
  33. Iyer, S., Konstas, I., Cheung, A., and Zettlemoyer, L. Summarizing source code using a neural attention model. In 54th Annual Meeting of the Association for Computational Linguistics 2016, pp. 2073–2083. Association for Computational Linguistics, 2016. (Cited on pg. 4) find
  34. Jain, N., Vaidyanath, S., Iyer, A., Natarajan, N., Parthasarathy, S., Rajamani, S., and Sharma, R. Jigsaw: Large language models meet program synthesis. In Proceedings of the 44th International Conference on Software Engineering, pp. 1219–1231, 2022. (Cited on pg. 3) find
  35. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023a. (Cited on pg. 6) find
  36. Jiang, N., Liu, K., Lutellier, T., and Tan, L. Impact of code language models on automated program repair. arXiv preprint arXiv:2302.05020, 2023b. (Cited on pg. 4) — [arXiv:2302.05020](https://arxiv.org/abs/2302.05020) find
  37. Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023. (Cited on pg. 3) find our card
  38. Jin, M., Shahriar, S., Tufano, M., Shi, X., Lu, S., Sundaresan, N., and Svyatkovskiy, A. Inferfix: End-to-end program repair with llms. arXiv preprint arXiv:2303.07263, 2023. (Cited on pg. 4) — [arXiv:2303.07263](https://arxiv.org/abs/2303.07263) find
  39. Key, D., Li, W.-D., and Ellis, K. I speak, you verify: Toward trustworthy neural program synthesis. arXiv preprint arXiv:2210.00848, 2022. (Cited on pg. 4) — [arXiv:2210.00848](https://arxiv.org/abs/2210.00848) find
  40. Lai, Y., Li, C., Wang, Y., Zhang, T., Zhong, R., Zettlemoyer, L., Yih, W.-t., Fried, D., Wang, S., and Yu, T. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pp. 18319–18345. PMLR, 2023. (Cited on pg. 3) find
  41. Le, H., Wang, Y., Gotmare, A. D., Savarese, S., and Hoi, S. C. H. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35:21314–21328, 2022. (Cited on pg. 4) find our card
  42. LeClair, A., Jiang, S., and McMillan, C. A neural model for generating natural language summaries of program subroutines. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pp. 795–806. find
  43. Lee, N., Sreenivasan, K., Lee, J. D., Lee, K., and Papailiopoulos, D. Teaching arithmetic to small transformers. arXiv preprint arXiv:2307.03381, 2023. (Cited on pg. 4) find
  44. Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161, 2023a. (Cited on pg. 3, 6) find
  45. Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. Competition-level code generation with alphacode. Science, 378(6624): 1092–1097, 2022. (Cited on pg. 3) find our card
  46. Li, Y., Bubeck, S., Eldan, R., Del Giorno, A., Gunasekar, S., and Lee, Y. T. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463, 2023b. (Cited on pg. 3, 6) find our card
  47. Liguori, P., Al-Hossami, E., Cotroneo, D., Natella, R., Cukic, B., and Shaikh, S. Can we generate shellcodes via natural language? an empirical study. Automated Software Engineering, 29(1):30, 2022. (Cited on pg. 4) find
  48. Liu, C., Lu, S., Chen, W., Jiang, D., Svyatkovskiy, A., Fu, S., Sundaresan, N., and Duan, N. Code execution with pre-trained language models. arXiv preprint arXiv:2305.05383, 2023a. (Cited on pg. 4) find
  49. Liu, H., Ning, R., Teng, Z., Liu, J., Zhou, Q., and Zhang, Y. Evaluating the logical reasoning ability of chatgpt and gpt-4. arXiv preprint arXiv:2304.03439, 2023b. (Cited on pg. 4) find our card
  50. Liu, J., Xia, C. S., Wang, Y., and Zhang, L. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. arXiv preprint arXiv:2305.01210, 2023c. (Cited on pg. 3) find our card
  51. Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1–35, 2023d. (Cited on pg. 15) find our card
  52. Liu, S., Gao, C., Chen, S., Nie, L. Y., and Liu, Y. Atom: Commit message generation based on abstract syntax tree and hybrid ranking. IEEE Transactions on Software Engineering, 48(5): 1800–1817, 2020. (Cited on pg. 4) find
  53. Liu, T., Xu, C., and McAuley, J. Repobench: Benchmarking repository-level code auto-completion systems. arXiv preprint arXiv:2306.03091, 2023e. (Cited on pg. 3) find
  54. Luo, Z., Xu, C., Zhao, P., Sun, Q., Geng, X., Hu, W., Tao, C., Ma, J., Lin, Q., and Jiang, D. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568, 2023. (Cited on pg. 3, 6, 7) find
  55. Madaan, A., Shypula, A., Alon, U., Hashemi, M., Ranganathan, P., Yang, Y., Neubig, G., and Yazdanbakhsh, A. Learning performance-improving code edits. arXiv preprint arXiv:2302.07867, 2023a. (Cited on pg. 4) find
  56. Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023b. (Cited on pg. 4, 16) find
  57. Malik, R. S., Patra, J., and Pradel, M. Nl2type: inferring javascript function types from natural language information. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pp. 304–315. find
  58. Merrill, W. and Sabharwal, A. The expresssive power of transformers with chain of thought. arXiv preprint arXiv:2310.07923, 2023. (Cited on pg. 4) find
  59. Merrill, W. C., Goldberg, Y., Schwartz, R., and Smith, N. A. Provable limitations of acquiring meaning from ungrounded form: What will future language models understand? Transactions of the Association for Computational Linguistics, 9:1047–1060, 2021. (Cited on pg. 4) find
  60. Miceli-Barone, A. V., Barez, F., Konstas, I., and Cohen, S. B. The larger they are, the harder they fail: Language models do not recognize identifier swaps in python. arXiv preprint arXiv:2305.15507, 2023. (Cited on pg. 4) find
  61. Mir, A. M., Latoškinas, E., Proksch, S., and Gousios, G. Type4py: Practical deep similarity learning-based type inference for python. In Proceedings of the 44th International Conference on Software Engineering, pp. 2241–2252, 2022. (Cited on pg. 4) find
  62. Mizrahi, M., Kaplan, G., Malkin, D., Dror, R., Shahaf, D., and Stanovsky, G. State of what art? a call for multi-prompt llm evaluation. arXiv preprint arXiv:2401.00595, 2023. (Cited on pg. 15) find
  63. Ni, A., Iyer, S., Radev, D., Stoyanov, V., Yih, W.-t., Wang, S., and Lin, X. V. Lever: Learning to verify language-to-code generation with execution. In International Conference on Machine Learning, pp. 26106–26128. PMLR, 2023. (Cited on pg. 4) find our card
  64. Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., and Xiong, C. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations, 2022. (Cited on pg. 3) find
  65. Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114, 2021. (Cited on pg. 4, 14, 54) find our card
  66. Olausson, T. X., Gu, A., Lipkin, B., Zhang, C. E., Solar-Lezama, A., Tenenbaum, J. B., and Levy, R. Linc: A neurosymbolic approach for logical reasoning by combining language models with first-order logic provers. arXiv preprint arXiv:2310.15164, 2023a. (Cited on pg. 4) find our card
  67. Olausson, T. X., Inala, J. P., Wang, C., Gao, J., and Solar-Lezama, A. Demystifying gpt self-repair for code generation. arXiv preprint arXiv:2306.09896, 2023b. (Cited on pg. 4, 16) find
  68. OpenAI, R. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2023. (Cited on pg. 2, 7) find our card
  69. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022. (Cited on pg. 2, 7) find
  70. Patil, S. G., Zhang, T., Wang, X., and Gonzalez, J. E. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334, 2023. (Cited on pg. 3) find our card
  71. Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., and Karri, R. Asleep at the keyboard? assessing the security of github copilot’s code contributions. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 754–768. find
  72. Peng, B., Galley, M., He, P., Cheng, H., Xie, Y., Hu, Y., Huang, Q., Liden, L., Yu, Z., Chen, W., and Gao, J. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint arXiv:2302.12813, 2023. (Cited on pg. 4, 16) find our card
  73. Royzen, M., Wei, J., and Coleman, R. Phind, 2023. URL https://www.phind.com. (Cited on pg. 3, 6, 7) ↗
  74. Roziere, B., Lachaux, M.-A., Chanussot, L., and Lample, G. Unsupervised translation of programming languages. Advances in Neural Information Processing Systems, 33:20601–20611, 2020. (Cited on pg. 4) find our card
  75. Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Remez, T., Rapin, J., et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023. (Cited on pg. 2, 3, 6) find our card
  76. Shen, B., Zhang, J., Chen, T., Zan, D., Geng, B., Fu, A., Zeng, M., Yu, A., Ji, J., Zhao, J., et al. Pangu-coder2: Boosting large language models for code with ranking feedback. arXiv preprint arXiv:2307.14936, 2023. (Cited on pg. 4) — [arXiv:2307.14936](https://arxiv.org/abs/2307.14936) find
  77. Shi, F., Fried, D., Ghazvininejad, M., Zettlemoyer, L., and Wang, S. I. Natural language to code translation with execution. arXiv preprint arXiv:2204.11454, 2022. (Cited on pg. 4) find
  78. Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E. H., Schärli, N., and Zhou, D. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pp. 31210–31227. PMLR, 2023. (Cited on pg. 4) find our card
  79. Shinn, N., Labash, B., and Gopinath, A. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366, 2023. (Cited on pg. 3, 4) find our card
  80. Shrivastava, D., Larochelle, H., and Tarlow, D. Repository-level prompt generation for large language models of code. In International Conference on Machine Learning, pp. 31693–31715. PMLR, 2023. (Cited on pg. 3) find
  81. Tian, Z. and Chen, J. Test-case-driven programming understanding in large language models for better code generation. arXiv preprint arXiv:2309.16120, 2023. (Cited on pg. 4) — [arXiv:2309.16120](https://arxiv.org/abs/2309.16120) find
  82. Tony, C., Mutas, M., Ferreyra, N. E. D., and Scandariato, R. Llmseceval: A dataset of natural language prompts for security evaluations. arXiv preprint arXiv:2303.09384, 2023. (Cited on pg. 4) — [arXiv:2303.09384](https://arxiv.org/abs/2303.09384) find
  83. Tufano, M., Watson, C., Bavota, G., Penta, M. D., White, M., and Poshyvanyk, D. An empirical study on learning bug-fixing patches in the wild via neural machine translation. ACM Transactions on Software Engineering and Methodology (TOSEM), 28(4):1–29, 2019. (Cited on pg. 4) find
  84. Tufano, M., Deng, S. K., Sundaresan, N., and Svyatkovskiy, A. Methods2test: A dataset of focal methods mapped to test cases. In Proceedings of the 19th International Conference on Mining Software Repositories, pp. 299–303, 2022. (Cited on pg. 4) find
  85. Tyen, G., Mansoor, H., Chen, P., Mak, T., and Cărbune, V. Llms cannot find reasoning errors, but can correct them! arXiv preprint arXiv:2311.08516, 2023. (Cited on pg. 16) — [arXiv:2311.08516](https://arxiv.org/abs/2311.08516) find
  86. Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, L., Creswell, A., Irving, G., and Higgins, I. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275, 2022. (Cited on pg. 16) find our card
  87. Wang, S., Li, Z., Qian, H., Yang, C., Wang, Z., Shang, M., Kumar, V., Tan, S., Ray, B., Bhatia, P., et al. Recode: Robustness evaluation of code generation models. arXiv preprint arXiv:2212.10264, 2022a. (Cited on pg. 3) find
  88. Wang, Z., Zhou, S., Fried, D., and Neubig, G. Execution-based evaluation for open-domain code generation. arXiv preprint arXiv:2212.10481, 2022b. (Cited on pg. 3) find
  89. Watson, C., Tufano, M., Moran, K., Bavota, G., and Poshyvanyk, D. On learning meaningful assert statements for unit test cases. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, pp. 1398–1409, 2020. (Cited on pg. 4) find
  90. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022. (Cited on pg. 10) find our card
  91. Wei, J., Durrett, G., and Dillig, I. Typet5: Seq2seq type inference using static analysis. arXiv preprint arXiv:2303.09564, 2023. (Cited on pg. 4) — [arXiv:2303.09564](https://arxiv.org/abs/2303.09564) find
  92. Wu, Z., Qiu, L., Ross, A., Akyürek, E., Chen, B., Wang, B., Kim, N., Andreas, J., and Kim, Y. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. arXiv preprint arXiv:2307.02477, 2023. (Cited on pg. 4) find
  93. Xia, C. S., Wei, Y., and Zhang, L. Practical program repair in the era of large pre-trained language models. arXiv preprint arXiv:2210.14179, 2022. (Cited on pg. 4) — [arXiv:2210.14179](https://arxiv.org/abs/2210.14179) find
  94. Xu, F. F., Alon, U., Neubig, G., and Hellendoorn, V. J. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, pp. 1–10, 2022. (Cited on pg. 3) find
  95. Yin, P., Li, W.-D., Xiao, K., Rao, A., Wen, Y., Shi, K., Howland, J., Bailey, P., Catasta, M., Michalewski, H., et al. Natural language to code generation in interactive data science notebooks. arXiv preprint arXiv:2212.09248, 2022. (Cited on pg. 3) find
  96. Zan, D., Chen, B., Zhang, F., Lu, D., Wu, B., Guan, B., Yongji, W., and Lou, J.-G. Large language models meet nl2code: A survey. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7443–7464, 2023. (Cited on pg. 2) find
  97. Zhang, F., Chen, B., Zhang, Y., Liu, J., Zan, D., Mao, Y., Lou, J.-G., and Chen, W. Repocoder: Repository-level code completion through iterative retrieval and generation. arXiv preprint arXiv:2303.12570, 2023a. (Cited on pg. 3) find
  98. Zhang, H., Li, L. H., Meng, T., Chang, K.-W., and Broeck, G. V. d. On the paradox of learning to reason from data. arXiv preprint arXiv:2205.11502, 2022. (Cited on pg. 4) — [arXiv:2205.11502](https://arxiv.org/abs/2205.11502) find
  99. Zhang, K., Li, G., Li, J., Li, Z., and Jin, Z. Toolcoder: Teach code generation models to use apis with search tools. arXiv preprint arXiv:2305.04032, 2023b. (Cited on pg. 3) — [arXiv:2305.04032](https://arxiv.org/abs/2305.04032) find
  100. Zhang, K., Li, Z., Li, J., Li, G., and Jin, Z. Self-edit: Fault-aware code editor for code generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 769–787, Toronto, Canada, July 2023c. Association for Computational Linguisti… find
  101. Zhang, K., Wang, D., Xia, J., Wang, W. Y., and Li, L. Algo: Synthesizing algorithmic programs with generated oracle verifiers. arXiv preprint arXiv:2305.14591, 2023d. (Cited on pg. 4) find
  102. Zhang, S., Chen, Z., Shen, Y., Ding, M., Tenenbaum, J. B., and Gan, C. Planning with large language models for code generation. arXiv preprint arXiv:2303.05510, 2023e. (Cited on pg. 4) find
  103. Zhang, S. D., Tigges, C., Biderman, S., Raginsky, M., and Ringer, T. Can transformers learn to solve problems recursively? arXiv preprint arXiv:2305.14699, 2023f. (Cited on pg. 4) find
  104. Zhang, T., Yu, T., Hashimoto, T., Lewis, M., Yih, W.-t., Fried, D., and Wang, S. Coder reviewer reranking for code generation. In International Conference on Machine Learning, pp. 41832–41846. PMLR, 2023g. (Cited on pg. 4) find
  105. Zhang, Z., Chen, C., Liu, B., Liao, C., Gong, Z., Yu, H., Li, J., and Wang, R. A survey on language models for code. 2023h. (Cited on pg. 3) find
  106. Zheng, Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Wang, Z., Shen, L., Wang, A., Li, Y., et al. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. arXiv preprint arXiv:2303.17568, 2023. (Cited on pg. 3) find our card
  107. Zhong, M., Liu, G., Li, H., Kuang, J., Zeng, J., and Wang, M. Codegen-test: An automatic code generation model integrating program test information. arXiv preprint arXiv:2202.07612, 2022. (Cited on pg. 4) — [arXiv:2202.07612](https://arxiv.org/abs/2202.07612) find
  108. Zhou, H., Bradley, A., Littwin, E., Razin, N., Saremi, O., Susskind, J., Bengio, S., and Nakkiran, P. What algorithms can transformers learn? a study in length generalization. arXiv preprint arXiv:2310.16028, 2023. (Cited on pg. 4) find
  109. Zhu, M., Jain, A., Suresh, K., Ravindran, R., Tipirneni, S., and Reddy, C. K. Xlcost: A benchmark dataset for cross-lingual code intelligence. arXiv preprint arXiv:2206.08474, 2022. (Cited on pg. 4) find