Elamur Research⌕
☰
home / papers / 2022 / A Systematic Evaluation of Large Language Models of Code

A Systematic Evaluation of Large Language Models of Code

Paper · 2022 · Frank F. Xu, Uri Alon, Graham Neubig et al.

Summary

The strongest code language models, such as Codex, are closed. Their weights and training data are unavailable, so researchers cannot fine-tune, inspect or extend them. It is also unclear how design choices affect performance, such as training on many programming languages versus one. The authors compare the largest accessible left-to-right models (Codex, GPT-J, GPT-Neo, GPT-NeoX-20B and CodeParrot) in two ways. The first is functional correctness on the HumanEval benchmark, which tests Python synthesis from natural-language prompts. The second is perplexity on a newly assembled held-out set of GitHub files in 12 programming languages, filtered to avoid overlap with the Pile. To fill a gap in open resources, they also train and release PolyCoder, a family of GPT-2-style models built only from multilingual source code. Codex leads in most languages, not just Python, and outperforms the Pile-trained models. However, PolyCoder 2.7B reaches the lowest perplexity of all models in C, Codex included. Publishing the models openly makes research on code LMs more reproducible and affordable outside large companies.

Key points

  • PolyCoder comes in three sizes (160M, 400M, 2.7B parameters). It was trained on 249GB of code covering 12 languages using a single machine, and is released at github.com/VHellendoorn/Code-LMs.
  • The HumanEval setup used 164 prompts with test cases and 100 samples per prompt. Sampling used nucleus sampling (top-p 0.95) at temperatures from 0.2 to 0.8.
  • Intrinsic evaluation used 100 random files per language. Perplexity was normalized by Pygments lexer tokens so that models with different tokenizers can be compared. Codex (code-davinci-001) may have seen the test data, so its scores could be inflated.
  • In C, perplexity was 2.33 for PolyCoder 2.7B, 2.37 for GPT-NeoX, 2.55 for Codex and 3.69 for GPT-Neo 2.7B. In Python, Codex scored 1.47 versus 3.18 for PolyCoder. CodeParrot was trained only on Python and does poorly in other languages.
  • Training longer (100K to 150K steps) helped the 2.7B model but not the 400M one. A high temperature such as 0.8 lowered Pass@100 for the 2.7B model, so temperature should be tuned for each model size.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States598 works · 191,225 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,894 worksEvaluation & benchmarks · 1,341 worksCode & programming languages · 168 works

Cite

Cite as Frank F. Xu, Uri Alon, Graham Neubig and Vincent J. Hellendoorn. 2022. A Systematic Evaluation of Large Language Models of Code. arXiv:2202.13169.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2202.13169 · arXiv 2202.13169

@misc{xu2022systematic,
  title = {{A Systematic Evaluation of Large Language Models of Code}},
  author = {Frank F. Xu, Uri Alon, Graham Neubig and Vincent J. Hellendoorn},
  year = {2022},
  doi = {10.48550/arxiv.2202.13169},
  eprint = {2202.13169},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2202.13169}
}

References

33 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. Unified pre-training for program understanding and generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2655–2668, Online, J…
  2. Miltiadis Allamanis. The adverse effects of code duplication in machine learning models of code. In Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software, pp. 143–153, 2019.
  3. Uri Alon, Roy Sadaka, Omer Levy, and Eran Yahav. Structural language models of code. In International Conference on Machine Learning, pp. 245–256. PMLR, 2020. find
  4. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021.
  5. Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. arXiv preprint arXiv:1809.10853, 2018.
  6. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of machine learning research, 3(Feb):1137–1155, 2003.
  7. Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021. URL https://doi.org/10.5281/zenodo.5297715. If you use this software, please cite it using these metadata. — [doi:10.5281/zenodo.5297715](https://doi.org/10.5281/zenodo.5297715) ↗
  8. Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT-NeoX-20B: An open-source autoregressive… — GPT-NeoX-20B: An Open-Source Autoregressive Language Model
  9. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.
  10. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
  11. Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. Advances in Neural Information Processing Systems, 32:7059–7069, 2019. find
  12. Aditya Desai, Sumit Gulwani, Vineet Hingorani, Nidhi Jain, Amey Karkare, Mark Marron, and Subhajit Roy. Program synthesis using natural language. In Proceedings of the 38th International Conference on Software Engineering, pp. 345–356, 2016. find
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  14. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155, 2020.
  15. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. find
  16. Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Liang Zhang, Wentao Han, Minlie Huang, et al. Pre-trained models: Past, present and future. AI Open, 2021. find
  17. Vincent J Hellendoorn and Premkumar Devanbu. Are deep neural networks the best choice for modeling source code? In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, pp. 763–773, 2017. find
  18. Vincent J. Hellendoorn and Anand Ashok Sawant. The growing cost of deep learning for source code. Commun. ACM, 65(1):31–33, dec 2021. ISSN 0001-0782. doi: 10.1145/3501261. URL https://doi.org/10.1145/3501261. — [doi:10.1145/3501261](https://doi.org/10.1145/3501261) ↗
  19. Abram Hindle, Earl T Barr, Mark Gabel, Zhendong Su, and Premkumar Devanbu. On the naturalness of software. Communications of the ACM, 59(5):122–131, 2016. find
  20. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751, 2019.
  21. Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436, 2019.
  22. Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. Learning and evaluating contextual embedding of source code. In International Conference on Machine Learning, pp. 5110–5121. PMLR, 2020. find
  23. Rafael-Michael Karampatsis, Hlib Babii, Romain Robbes, Charles Sutton, and Andrea Janes. Big code!= big vocabulary: Open-vocabulary models for source code. In 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE), pp. 1073–1085. find
  24. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019.
  25. Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, MING GONG, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie LIU. CodeXGLUE: A ma… find
  26. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. find
  27. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.
  28. Veselin Raychev, Martin Vechev, and Eran Yahav. Code completion with statistical language models. In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation, pp. 419–428, 2014. find
  29. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015.
  30. Lewis Tunstall, Leandro von Werra, and Thomas Wolf. Natural Language Processing with Transformers. ” O’Reilly Media, Inc.”, 2022. find
  31. Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax, May 2021. ↗
  32. Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:2109.00859, 2021. find
  33. Daniel Zügner, Tobias Kirschstein, Michele Catasta, Jure Leskovec, and Stephan Günnemann. Language-agnostic representation learning of source code from structure and context. In International Conference on Learning Representations, 2021. URL https://openreview.net/for um?id=Xh5eMZVONGF. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2022-systematic-evaluation-large-language-models/