Elamur Research⌕
☰
home / papers / 2022 / MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation

MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation

Paper · 2022 · Federico Cassano, John Gouwar, Daniel Nguyen et al.

Summary

Code generation models are trained on many programming languages but are usually tested only on Python. This leaves open how well they work for developers using other languages. MultiPL-E addresses this by converting Python benchmarks based on unit tests into other languages. It does not translate whole programs. Each target language needs only a small compiler, about 200 lines, that converts four elements: the function signature, the docstring comment, simple test cases, and type annotations where the language requires them. The model writes the function body itself. A rule-based step adjusts Python-specific vocabulary in comments, and a containerized sandbox runs and classifies the generated code. The authors applied this to HumanEval and MBPP, producing parallel versions in 18 additional languages. They then evaluated Codex, CodeGen, and InCoder. Codex performs as well as or better than its Python results in several languages, including JavaScript, C++, Scala, and TypeScript. Because the framework is cheap to extend, it supports fairer multi-language evaluation of code models.

Key points

  • The resources MultiPL-HumanEval and MultiPL-MBPP cover 19 languages in total (Python plus 18 targets) and contain matching problems across languages. They are released at github.com/nuprl/MultiPL-E.
  • Adding a language is cheap: a translator of roughly 200 LOC plus an evaluation script of around 20 LOC.
  • Across models, JavaScript results are very strong and sometimes surpass Python.
  • Model perplexity correlates poorly with functional correctness, so it is an unreliable proxy for code quality.
  • Performance generally tracks language popularity, though some niche languages score comparably. Results depend on how prompts are translated, while static typing shows no clear positive or negative effect.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 worksComputer Science Applications · 131 works

🧭 Topics 1Research subjects and languages the work is about.Code & programming languages · 168 works

Cite

Cite as Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg and Abhinav Jangda. 2022. MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation. arXiv:2208.08227.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2208.08227 · arXiv 2208.08227

@misc{cassano2022multipl,
  title = {{MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation}},
  author = {Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg and Abhinav Jangda},
  year = {2022},
  doi = {10.48550/arxiv.2208.08227},
  eprint = {2208.08227},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2208.08227}
}

References

44 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021.
  2. J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton, “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732, 2021. [Online]. Available: https://arxiv.org/abs/2108.07732 ↗
  3. E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong, “Codegen: An open large language model for code with multi-turn program synthesis,” 2022. [Online]. Available: https://arxiv.org/abs/2203.13474 ↗
  4. D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, W.-t. Yih, L. Zettlemoyer, and M. Lewis, “Incoder: A generative model for code infilling and synthesis,” arXiv preprint arXiv:2204.05999, 2022. [Online]. Available: https://arxiv.org/abs/2204.05999 ↗
  5. S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy, K. McDonell, J. Phang, M. Pieler, U. S. Prashanth, S. Purohit, L. Reynolds, J. Tow, B. Wang, and S. Weinbach, “Gpt-neox-20b: An open-source autoregressive language model,” arXiv preprint arXiv:2204.06745, 2022. [Onl…
  6. E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong, “A conversational paradigm for program synthesis,” arXiv preprint arXiv:2203.13474, 2022. [Online]. Available: https://arxiv.org/abs/2203.13474 ↗
  7. F. F. Xu, U. Alon, G. Neubig, and V. J. Hellendoorn, “A systematic evaluation of large language models of code,” in Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, 2022, pp. 1–10.
  8. A. Ziegler, E. Kalliamvakou, X. A. Li, A. Rice, D. Rifkin, S. Simister, G. Sittampalam, and E. Aftandilian, “Productivity assessment of neural code completion,” in Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, 2022, pp. 21–29. find
  9. S. Kulal, P. Pasupat, K. Chandra, M. Lee, O. Padon, A. Aiken, and P. S. Liang, “Spoc: Search-based pseudocode to code,” in Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds., vol. 32. Curran Associates, Inc., 201…
  10. D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt, “Measuring coding challenge competence with apps,” arXiv preprint arXiv:2105.09938, 2021. [Online]. Available: https://arxiv.org/abs/2105.09938 ↗
  11. P. Yin, B. Deng, E. Chen, B. Vasilescu, and G. Neubig, “Learning to mine aligned code and natural language pairs from stack overflow,” in Proceedings of the 15th International Conference on Mining Software Repositories, ser. MSR ’18. New York, NY, USA: Association for Computing Machinery, 2018, p. 4…
  12. R. Alur, R. Bodik, G. Juniwal, M. Martin, M. Raghothaman, S. A. Seshia, R. Singh, A. Solar-Lezama, E. Torlak, and A. Udupa, “Syntax-guided synthesis,” in Formal Methods in Computer-Aided Design (FMCAD), 2013. find
  13. S. Chaudhuri, K. Ellis, O. Polozov, R. Singh, A. Solar-Lezama, and Y. Yue, “Neurosymbolic Programming,” Foundations and Trends in Programming Languages, vol. 7, no. 3, pp. 158–243, 2021. find
  14. S. Gulwani, O. Polozov, R. Singh et al., “Program synthesis,” Foundations and Trends® in Programming Languages, vol. 4, no. 1-2, pp. 1–119, 2017.
  15. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020.
  16. B. Wang and A. Komatsuzaki, “Gpt-j-6b: A 6 billion parameter autoregressive language model,” 2021. [Online]. Available: https://github.com/kingoflolz/mesh-transformer-jax ↗
  17. Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, “Codebert: A pre-trained model for programming and natural languages,” arXiv preprint arXiv:2002.08155, 2020. [Online]. Available: https://arxiv.org/abs/2002.08155 ↗
  18. C. Clement, D. Drain, J. Timcheck, A. Svyatkovskiy, and N. Sundaresan, “PyMT5: multi-mode translation of natural language and python code with transformers,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Online: Association for Computational Lingu…
  19. M. Tufano, D. Drain, A. Svyatkovskiy, S. K. Deng, and N. Sundaresan, “Unit test case generation with transformers and focal context,” arXiv preprint arXiv:2009.10297, 2020. [Online]. Available: https://arxiv.org/abs/2009.05617 ↗
  20. S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang, G. Li, L. Zhou, L. Shou, L. Zhou, M. Tufano, M. Gong, M. Zhou, N. Duan, N. Sundaresan, S. K. Deng, S. Fu, and S. Liu, “Codexglue: A machine learning benchmark dataset for code understanding and gene… find
  21. T. Ahmed and P. Devanbu, “Multilingual training for software engineering,” in Proceedings of the 44th International Conference on Software Engineering. ACM, 2022. find
  22. J. Wei, M. Goyal, G. Durrett, and I. Dillig, “LambdaNet: Probabilistic Type Inference using Graph Neural Networks,” in International Conference on Learning Representations (ICLR), 2020. find
  23. V. J. Hellendoorn, C. Bird, E. T. Barr, and M. Allamanis, “Deep Learning Type Inference,” in Fse, 2018. find
  24. M. Pradel, G. Gousios, J. Liu, and S. Chandra, “Type-Writer: Neural Type Prediction with Search-Based Validation,” in Esecfse, 2020. find
  25. I. Drori, S. Zhang, R. Shuttleworth, L. Tang, A. Lu, E. Ke, K. Liu, L. Chen, S. Tran, N. Cheng, R. Wang, N. Singh, T. L. Patti, J. Lynch, A. Shporer, N. Verma, E. Wu, and G. Strang, “A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning… find
  26. A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi, “The curious case of neural text degeneration,” in ICLR, 2020.
  27. S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, and S. Ma, “Codebleu: a method for automatic evaluation of code synthesis,” 2020. [Online]. Available: https://arxiv.org/abs/2009.10297 ↗
  28. T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, Z. Zhang, and D. Radev, “Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Langu…
  29. V. Zhong, C. Xiong, and R. Socher, “Seq2sql: Generating structured queries from natural language using reinforcement learning,” 2017. [Online]. Available: https://arxiv.org/abs/1709.00103 ↗
  30. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curra…
  31. A. Aghajanyan, B. Huang, C. Ross, V. Karpukhin, H. Xu, N. Goyal, D. Okhonko, M. Joshi, G. Ghosh, M. Lewis, and L. Zettlemoyer, “Cm3: A causal masked multimodal model of the internet,” arXiv preprint arXiv:2201.07520, 2022. [Online]. Available: https://arxiv.org/abs/2201.07520 ↗
  32. L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy, “The pile: An 800gb dataset of diverse text for language modeling,” 2021. [Online]. Available: https://arxiv.org/abs/2101.00027 ↗
  33. W. Zaremba, G. Brockman, and OpenAI, “Openai codex,” 2021. [Online]. Available: https://openai.com/blog/openai-codex/ ↗
  34. D. Bates, M. Mächler, B. Bolker, and S. Walker, “Fitting linear mixed-effects models using lme4,” Journal of Statistical Software, vol. 67, no. 1, pp. 1–48, 2015.
  35. B. Chen, F. Zhang, A. Nguyen, D. Zan, Z. Lin, J.-G. Lou, and W. Chen, “CodeT: Code generation with generated tests,” 2022. [Online]. Available: https://arxiv.org/abs/2207.10397 ↗
  36. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: A method for automatic evaluation of machine translation,” in Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ser. ACL ’02. USA: Association for Computational Linguistics, 2002, p. 311–318. [Online]. Availabl…
  37. Z. Wang, G. Cuenca, S. Zhou, F. F. Xu, and G. Neubig, “Mconala: A benchmark for code generation from multiple natural languages,” 2022. [Online]. Available: https://arxiv.org/abs/2203.08388 ↗
  38. S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer, “Mapping language to code in programmatic context,” arXiv preprint arXiv:1808.09588, 2018.
  39. T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, Z. Zhang, and D. Radev, “Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task,” 2018. [Online]. Available: https://arxiv.org/abs/1809.08887 ↗
  40. L. Tunstall, L. von Werra, and T. Wolf, Natural Language Processing with Transformers. O’Reilly Media, 2022. [Online]. Available: https://books.google.com/books?id=nTxbEAAAQBAJ ↗
  41. Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. D. Lago, T. Hubert, P. Choy, C. d. M. d’Autume, I. Babuschkin, X. Chen, P.-S. Huang, J. Welbl, S. Gowal, A. Cherepanov, J. Molloy, D. J. Mankowitz, E. S. Robson, P. Kohli, N. de Freitas, K. Kavuk… — Competition-Level Code Generation with AlphaCode
  42. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. Austin, M. Isard, G. Gur-… — PaLM: Scaling Language Modeling with Pathways
  43. B. Athiwaratkun, S. K. Gouda, Z. Wang, X. Li, Y. Tian, M. Tan, W. U. Ahmad, S. Wang, Q. Sun, M. Shang, S. K. Gonugondla, H. Ding, V. Kumar, N. Fulton, A. Farahani, S. Jain, R. Giaquinto, H. Qian, M. K. Ramanathan, R. Nallapati, B. Ray, P. Bhatia, S. Sengupta, D. Roth, and B. Xiang, “Multi-lingual ev… — Multi-lingual Evaluation of Code Generation Models
  44. T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. D. Iii, and K. Crawford, “Datasheets for datasets,” Communications of the ACM, vol. 64, no. 12, pp. 86–92, 2021. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cassano-2022-multipl-e-scalable-extensible-approach/