Evaluating Large Language Models Trained on Code

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Computing & GPUs

Topics: Machine translation

Cited by

2529 papers of our library cite it (of the 140673 whose bibliographies we read); 203 of them have a page here, the first 100 below, 2326 not yet.

• Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
• Phi-4 Technical Report
• Large Language Models for Mathematical Reasoning: Progresses and Challenges
• Yi: Open Foundation Models by 01.AI
• Show your work: Scratchpads for intermediate computation with language models
• SantaCoder: don't reach for the stars!
• SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
• The Falcon Series of Open Language Models
• Exploring Length Generalization in Large Language Models
• PaLM 2 Technical Report
• A General Language Assistant as a Laboratory for Alignment
• Program Synthesis with Large Language Models
• ProofNet: Autoformalizing and Formally Proving Undergraduate-Level Mathematics
• LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
• Qwen Technical Report
• Qwen2.5-VL Technical Report
• Efficient Training of Language Models to Fill in the Middle
• GPT-NeoX-20B: An Open-Source Autoregressive Language Model
• On the Opportunities and Risks of Foundation Models
• AudioLM: a Language Modeling Approach to Audio Generation
• Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
• Sparks of Artificial General Intelligence: Early experiments with GPT-4
• InternLM2 Technical Report
• MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
• A Survey on Evaluation of Large Language Models
• Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks
• AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors
• AlpaGasus: Training A Better Alpaca with Fewer Data
• MEDITRON-70B: Scaling Medical Pretraining for Large Language Models
• TheoremQA: A Theorem-driven Question Answering Dataset
• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
• Binding Language Models in Symbolic Languages
• FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios
• Safe RLHF: Safe Reinforcement Learning from Human Feedback
• DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
• Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
• LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding
• A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models
• LawBench: Benchmarking Legal Knowledge of Large Language Models
• From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models
• Mathematical Capabilities of ChatGPT
• Complexity-Based Prompting for Multi-Step Reasoning
• Specializing Smaller Language Models towards Multi-Step Reasoning
• Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
• Precise Zero-Shot Dense Retrieval without Relevance Labels
• Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
• Gemini: A Family of Highly Capable Multimodal Models
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
• ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools
• GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models
• CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
• CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution
• The False Promise of Imitating Proprietary LLMs
• LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
• How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection
• Evaluating Large Language Models: A Comprehensive Survey
• AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators
• WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models
• Measuring Mathematical Problem Solving With the MATH Dataset
• MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework
• V-STaR: Training Verifiers for Self-Taught Reasoners
• MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
• Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents
• C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models
• Towards Reasoning in Large Language Models: A Survey
• Qwen2.5-Coder Technical Report
• Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2
• LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
• Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofs
• A Survey on Large Language Models for Code Generation
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• Challenges and Applications of Large Language Models
• Kimi K2: Open Agentic Intelligence
• OpenAssistant Conversations -- Democratizing Large Language Model Alignment
• ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic
• Training Language Models to Self-Correct via Reinforcement Learning
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training
• CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning
• Solving Quantitative Reasoning Problems with Language Models
• Competition-Level Code Generation with AlphaCode
• API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs
• CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society
• CMMLU: Measuring massive multitask language understanding in Chinese
• Multi-step Jailbreaking Privacy Attacks on ChatGPT
• Textbooks Are All You Need II: phi-1.5 technical report
• Towards General Text Embeddings with Multi-stage Contrastive Learning
• From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline
• DataComp-LM: In search of the next generation of training sets for language models
• EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
• LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
• TruthfulQA: Measuring How Models Mimic Human Falsehoods
• Are LLMs Good Text Diacritizers? An Arabic and Yoruba Case Study
• Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
• AgentBench: Evaluating LLMs as Agents
• LLM+P: Empowering Large Language Models with Optimal Planning Proficiency
• Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment
• Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
• Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2107.03374 · arXiv 2107.03374

@misc{chen2021evaluating,
  title = {{Evaluating Large Language Models Trained on Code}},
  author = {Chen, Mark and Tworek, Jerry and Jun, Heewoo and Yuan, Qiming and Pinto, Henrique Ponde de Oliveira and Kaplan, Jared and Edwards, Harri and Burda, Yuri and Joseph, Nicholas and Brockman, Greg and Ray, Alex and Puri, Raul and Krueger, Gretchen and Petrov, Michael and Khlaaf, Heidy and Sastry, Girish and Mishkin, Pamela and Chan, Brooke and Gray, Scott and Ryder, Nick and Pavlov, Mikhail and Power, Alethea and Kaiser, Lukasz and Bavarian, Mohammad and Winter, Clemens and Tillet, Philippe and Such, Felipe Petroski and Cummings, Dave and Plappert, Matthias and Chantzis, Fotios and Barnes, Elizabeth and Herbert-Voss, Ariel and Guss, William Hebgen and Nichol, Alex and Paino, Alex and Tezak, Nikolas and Tang, Jie and Babuschkin, Igor and Balaji, Suchir and Jain, Shantanu and Saunders, William and Hesse, Christopher and Carr, Andrew N. and Leike, Jan and Achiam, Josh and Misra, Vedant and Morikawa, Evan and Radford, Alec and Knight, Matthew and Brundage, Miles and Murati, Mira and Mayer, Katie and Welinder, Peter and McGrew, Bob and Amodei, Dario and McCandlish, Sam and Sutskever, Ilya and Zaremba, Wojciech},
  year = {2021},
  doi = {10.48550/arxiv.2107.03374},
  eprint = {2107.03374},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2107.03374}
}

References

123 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Cwe-327: Use of a broken or risky cryptographic algorithm, 2006. URL https://cwe.mitre.org/data/definitions/327.html. ↗
2. Cwe-780: Use of rsa algorithm without oaep, 2009. URL https: //cwe.mitre.org/data/definitions/780.html. find
3. A6:2017-security misconfiguration, 2017. URL https: //owasp.org/www-project-top-ten/2017/ A6 2017-Security Misconfiguration.html. find
4. Abid, A., Farooqi, M., and Zou, J. Persistent anti-muslim bias in large language models. arXiv preprint arXiv:2101.05783, 2021. find our card
5. Acemoglu, D. and Restrepo, P. Robots and jobs: Evidence from us labor markets. Journal of Political Economy, 128(6):2188–2244, 2020a. find
6. Acemoglu, D. and Restrepo, P. The wrong kind of ai? artificial intelligence and the future of labour demand. Cambridge Journal of Regions, Economy and Society, 13(1):25–35, 2020b. find
7. Agrawal, H., Horgan, J. R., London, S., and Wong, W. E. Fault localization using execution slices and dataflow tests. Proceedings of Sixth International Symposium on Software Reliability Engineering. ISSRE’95, pp. 143–151, 1995. find
8. Allamanis, M., Tarlow, D., Gordon, A., and Wei, Y. Bimodal modelling of source code and natural language. In Bach, F. and Blei, D. (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 2123–2132, Lille, France, 07–09… find
9. Alley, E. C., Khimulya, G., Biswas, S., AlQuraishi, M., and Church, G. M. Unified rational protein engineering with sequence-based deep representation learning. Nature methods, 16(12):1315–1322, 2019. find
10. Alon, U., Brody, S., Levy, O., and Yahav, E. code2seq: Generating sequences from structured representations of code. In International Conference on Learning Representations, 2018. find
11. Aye, G. A., Kim, S., and Li, H. Learning autocompletion from real-world datasets. 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 131–139, 2021. find
12. Baevski, A., Zhou, H., Mohamed, A., and Auli, M. wav2vec 2.0: A framework for self-supervised learning of speech representations. arXiv preprint arXiv:2006.11477, 2020. find our card
13. Balog, M., Gaunt, A., Brockschmidt, M., Nowozin, S., and Tarlow, D. Deepcoder: Learning to write programs. In 5th International Conference on Learning Representations (ICLR), 2017. find
14. Bao, H., Dong, L., and Wei, F. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021. — [arXiv:2106.08254](https://arxiv.org/abs/2106.08254) find
15. Barone, A. V. M. and Sennrich, R. A parallel corpus of python functions and documentation strings for automated code documentation and code generation. ArXiv, abs/1707.02275, 2017. find
16. Barrington, I. M. and Maciel, A. Lecture 3: Nondeterministic computation. https://people.clarkson.edu/˜alexis/PCMI/Notes/lectureB03.pdf, 2000. [Online; accessed 29-June-2000]. ↗
17. Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 610–623, 2021. find
18. Black, S., Gao, L., Wang, P., Leahy, C., and Biderman, S. GPT-Neo: Large scale autoregressive language modeling with mesh-tensorflow, 2021. URL http://github.com/eleutherai/gpt-neo. ↗
19. Blodgett, S. L., Barocas, S., Daumé III, H., and Wallach, H. Language (technology) is power: A critical survey of “bias” in nlp. arXiv preprint arXiv:2005.14050, 2020. find our card
20. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Ch… find
21. Bureau of Labor Statistics, U. D. o. L. Computer programmers. Occupational Outlook Handbook, 2021a. URL https: //www.bls.gov/ooh/computer-and-information-technology/computer-programmers.htm. find
22. Bureau of Labor Statistics, U. D. o. L. Bls - software developers. Occupational Outlook Handbook, 2021b. URL https: //www.bls.gov/ooh/computer-and-information-technology/software-developers.htm. find
23. Carlini, N., Tramèr, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., Oprea, A., and Raffel, C. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21). USENIX Association, August 2021. U… find our card
24. Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D., and Sutskever, I. Generative pretraining from pixels. In International Conference on Machine Learning, pp. 1691–1703. PMLR, 2020. find
25. Child, R., Gray, S., Radford, A., and Sutskever, I. Generating long sequences with sparse transformers. ArXiv, abs/1904.10509, 2019. find
26. Christiano, P. Clarifying ”ai alignment”. AI Alignment Forum, 2018. URL https://www.alignmentforum.org/posts/ZeE7EKHTFMBs8eMxn/clarifying-ai-alignment. ↗
27. Clarkson, M. R., Finkbeiner, B., Koleini, M., Micinski, K. K., Rabe, M. N., and Sánchez, C. Temporal logics for hyperproperties. In International Conference on Principles of Security and Trust, pp. 265–284. Springer, 2014. find
28. Clement, C., Drain, D., Timcheck, J., Svyatkovskiy, A., and Sundaresan, N. Pymt5: Multi-mode translation of natural language and python code with transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9052–9065, 2020. find
29. Crawford, K. The trouble with bias. NIPS 2017 Keynote, 2017. URL https://www.youtube.com/watch?v=fMym BKWQzk. ↗
30. Crawford, K. Atlas of AI: Power, Politics, and the Planetary Costs of Artificial Intelligence. Yale University Press, 2021. find
31. Dai, A. M. and Le, Q. V. Semi-supervised sequence learning. Advances in neural information processing systems, 28:3079– 3087, 2015. find
32. Das, A., Kottur, S., Gupta, K., Singh, A., Yadav, D., Moura, J. M., Parikh, D., and Batra, D. Visual dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 326–335, 2017. find
33. Davis, B. Protecting applications with automated software diversity, Sep 2018. URL https://galois.com/blog/2018/09/protecting-applications-with-automated-software-diversity. ↗
34. Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., and Łukasz Kaiser. Universal transformers, 2019. find
35. Devlin, J., Uesato, J., Bhupatiraju, S., Singh, R., rahman Mohamed, A., and Kohli, P. Robustfill: Neural program learning under noisy i/o. In ICML, 2017. find
36. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. find
37. Dhariwal, P., Jun, H., Payne, C., Kim, J. W., Radford, A., and Sutskever, I. Jukebox: A generative model for music. arXiv preprint arXiv:2005.00341, 2020. — [arXiv:2005.00341](https://arxiv.org/abs/2005.00341) find
38. Drain, D., Wu, C., Svyatkovskiy, A., and Sundaresan, N. Generating bug-fixes using pretrained transformers. Proceedings of the 5th ACM SIGPLAN International Symposium on Machine Programming, 2021. find
39. Eghbal, N. Working in public: the making and maintenance of open source software. Stripe Press, 2020. find
40. Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. Codebert: A pre-trained model for programming and natural languages. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1536–1547, 2020. find our card
41. Frey, C. B. The technology trap. Princeton University Press, 2019. find
42. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. The pile: An 800gb dataset of diverse text for language modeling. 2020. find
43. Goldblum, M., Tsipras, D., Xie, C., Chen, X., Schwarzschild, A., Song, D., Madry, A., Li, B., and Goldstein, T. Dataset security for machine learning: Data poisoning, backdoor attacks, and defenses, 2021. find
44. Goues, C. L., Dewey-Vogt, M., Forrest, S., and Weimer, W. A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each. 2012 34th International Conference on Software Engineering (ICSE), pp. 3–13, 2012. find
45. Graves, A. Generating sequences with recurrent neural networks, 2014. find
46. Graves, A., Wayne, G., and Danihelka, I. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014. find
47. Graves, A., Wayne, G., Reynolds, M., Harley, T., Danihelka, I., Grabska-Barwińska, A., Colmenarejo, S. G., Grefenstette, E., Ramalho, T., Agapiou, J., et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538 (7626):471–476, 2016. find
48. Gulwani, S. Automating string processing in spreadsheets using input-output examples. In PoPL’11, January 26-28, 2011, Austin, Texas, USA, January 2011. find
49. Gulwani, S., Harris, W. R., and Singh, R. Spreadsheet data manipulation using examples. Commun. ACM, 55:97–105, 2012. find
50. He, P., Liu, X., Gao, J., and Chen, W. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654, 2020. find our card
51. Helmuth, T. and Spector, L. General program synthesis benchmark suite. In Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation, pp. 1039–1046, 2015. find
52. Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., et al. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938, 2021. find our card
53. Hindle, A., Barr, E. T., Su, Z., Gabel, M., and Devanbu, P. On the naturalness of software. In 2012 34th International Conference on Software Engineering (ICSE), pp. 837–847. find
54. Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The curious case of neural text degeneration, 2020. find
55. Husain, H., Wu, H.-H., Gazit, T., Allamanis, M., and Brockschmidt, M. Codesearchnet challenge: Evaluating the state of semantic code search. ArXiv, abs/1909.09436, 2019. find our card
56. Jain, P., Jain, A., Zhang, T., Abbeel, P., Gonzalez, J., and Stoica, I. Contrastive code representation learning. ArXiv, abs/2007.04973, 2020. find
57. Jeffrey, D., Feng, M., Gupta, N., and Gupta, R. Bugfix: A learning-based tool to assist developers in fixing bugs. 2009 IEEE 17th International Conference on Program Comprehension, pp. 70– 79, 2009. find
58. Jones, C. and Bonsignour, O. The economics of software quality. Addison-Wesley Professional, 2011. find
59. Kaiser, Ł. and Sutskever, I. Neural gpus learn algorithms. arXiv preprint arXiv:1511.08228, 2015. find
60. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models, 2020. find
61. Kenton, Z., Everitt, T., Weidinger, L., Gabriel, I., Mikulik, V., and Irving, G. Alignment of language agents. arXiv preprint arXiv:2103.14659, 2021. find
62. Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., and Socher, R. Ctrl: A conditional transformer language model for controllable generation, 2019. find our card
63. Korel, B. and Rilling, J. Application of dynamic slicing in program debugging. In AADEBUG, 1997. find
64. Koza, J. R., Andre, D., Keane, M. A., and Bennett III, F. H. Genetic programming III: Darwinian invention and problem solving, volume 3. Morgan Kaufmann, 1999. find
65. Kulal, S., Pasupat, P., Chandra, K., Lee, M., Padon, O., Aiken, A., and Liang, P. S. Spoc: Search-based pseudocode to code. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associa… find
66. Lacasse, N. Open-sourcing gvisor, a sandboxed container runtime, 2018. find
67. Lachaux, M.-A., Rozière, B., Chanussot, L., and Lample, G. Unsupervised translation of programming languages. ArXiv, abs/2006.03511, 2020. find our card
68. Leveson, N. Improving the standard risk matrix: Part 1. 2019. URL http://sunnyday.mit.edu/Risk-Matrix.pdf. ↗
69. Li, P. L., Ko, A. J., and Begel, A. What distinguishes great software engineers? Empirical Software Engineering, 25(1):322–352, 2020. find
70. Ling, W., Blunsom, P., Grefenstette, E., Hermann, K. M., Kočiskỳ, T., Wang, F., and Senior, A. Latent predictor networks for code generation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 599–609, 2016. find
71. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692, 2019. find
72. Lu, J., Batra, D., Parikh, D., and Lee, S. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. arXiv preprint arXiv:1908.02265, 2019. find our card
73. Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., Li, G., Zhou, L., Shou, L., Zhou, L., Tufano, M., Gong, M., Zhou, M., Duan, N., Sundaresan, N., Deng, S. K., Fu, S., and Liu, S. Codexglue: A machine learning benchmark dataset for code un… find
74. Maddison, C. J. and Tarlow, D. Structured generative models of natural source code. In Proceedings of the 31st International Conference on International Conference on Machine Learning (ICML), pp. II–649, 2014. find
75. Manna, Z. and Waldinger, R. J. Toward automatic program synthesis. 14(3):151–165, March 1971. ISSN 0001-0782. doi: 10.1145/362566.362568. URL https://doi.org/10.1145/362566.362568. — [doi:10.1145/362566.362568](https://doi.org/10.1145/362566.362568) ↗
76. Masanet, E., Shehabi, A., Lei, N., Smith, S., and Koomey, J. Recalibrating global data center energy-use estimates. Science, 367(6481):984–986, 2020. find
77. Menezes, A., van Oorschot, P., and Vanstone, S. Handbook of Applied Cryptography. Discrete Mathematics and Its Applications. CRC Press, 2018. ISBN 9780429881329. URL https: //books.google.com/books?id=YyCyDwAAQBAJ. find
78. Menick, J. and Kalchbrenner, N. Generating high fidelity images with subscale pixel networks and multidimensional upscaling, 2018. find
79. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pp. 3111–3119, 2013. find
80. Ohm, M., Plate, H., Sykosch, A., and Meier, M. Backstabber’s knife collection: A review of open source software supply chain attacks, 2020. find
81. O’Keefe, C., Lansky, D., Clark, J., and Payne, C. Comment regarding request for comments on intellectual property protection for artificial intelligence innovation. Before the United States Patent and Trademark Office Department of Commerce, 2019. URL https://perma.cc/ZS7G-2QWF. ↗
82. O*NET. 15-1252.00 - software developers, 2021. URL https://www.onetonline.org/link/summary/15-1252.00. ↗
83. Oord, A. v. d., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499, 2016. find our card
84. Oord, A. v. d., Li, Y., and Vinyals, O. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. find our card
85. O’Neill, M. and Spector, L. Automatic programming: The open issue? Genetic Programming and Evolvable Machines, pp. 1–12, 2019. find
86. Pantridge, E., Helmuth, T., McPhee, N. F., and Spector, L. On the difficulty of benchmarking inductive program synthesis methods. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 1589–1596, 2017. find
87. Patterson, D., Gonzalez, J., Le, Q., Liang, C., Munguia, L.-M., Rothchild, D., So, D., Texier, M., and Dean, J. Carbon emissions and large neural network training. arXiv preprint arXiv:2104.10350, 2021. find our card
88. Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. Deep contextualized word representations. arXiv preprint arXiv:1802.05365, 2018. find
89. Pierrot, T., Ligner, G., Reed, S., Sigaud, O., Perrin, N., Laterre, A., Kas, D., Beguir, K., and de Freitas, N. Learning compositional neural programs with recursive tree search and planning, 2021. find
90. Planning, S. The economic impacts of inadequate infrastructure for software testing. National Institute of Standards and Technology, 2002. find
91. Python Software Foundation and JetBrains. Python developers survey 2020 results, 2020. URL https: //www.jetbrains.com/lp/python-developers-survey-2020/. find
92. Qi, Z., Long, F., Achour, S., and Rinard, M. An analysis of patch plausibility and correctness for generate-and-validate patch generation systems. Proceedings of the 2015 International Symposium on Software Testing and Analysis, 2015. find
93. Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. Improving language understanding by generative pre-training. 2018. find
94. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. 2019. find
95. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020, 2021. find our card
96. Raffel, C., Shazeer, N. M., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. ArXiv, abs/1910.10683, 2020. find
97. Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. Zero-shot text-to-image generation. ArXiv, abs/2102.12092, 2021. find our card
98. Reed, S. and de Freitas, N. Neural programmer-interpreters, 2016. find
99. Ren, S., Guo, D., Lu, S., Zhou, L., Liu, S., Tang, D., Sundaresan, N., Zhou, M., Blanco, A., and Ma, S. Codebleu: a method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009.10297, 2020. — [arXiv:2009.10297](https://arxiv.org/abs/2009.10297) find
100. Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15), 2021. find
101. Rokon, M. O. F., Islam, R., Darki, A., Papalexakis, E. E., and Faloutsos, M. Sourcefinder: Finding malware source-code from publicly available repositories in github. In 23rd International Symposium on Research in Attacks, Intrusions and Defenses (RAID 2020), pp. 149–163, San Sebastian, October 2020… find
102. Schuster, R., Song, C., Tromer, E., and Shmatikov, V. You autocomplete me: Poisoning vulnerabilities in neural code completion. The Advanced Computing Systems Association, 2020. URL https://www.usenix.org/system/files/sec21summer schuster.pdf. ↗
103. Schwartz, R., Dodge, J., Smith, N. A., and Etzioni, O. Green ai, 2019. find
104. Shin, E. C., Polosukhin, I., and Song, D. Improving neural program synthesis with inferred execution traces. Advances in Neural Information Processing Systems, 31:8917–8926, 2018. find
105. Simon, H. A. Experiments with a heuristic compiler. J. ACM, 10(4):493–506, October 1963. ISSN 0004-5411. doi: 10.1145/321186.321192. URL https://doi.org/10.1145/321186.321192. — [doi:10.1145/321186.321192](https://doi.org/10.1145/321186.321192) ↗
106. Stack Overflow. 2020 developer survey, 2020. URL https://insights.stackoverflow.com/survey/2020\#overview. ↗
107. Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. Learning to summarize from human feedback, 2020. find
108. Sukhbaatar, S., Szlam, A., Weston, J., and Fergus, R. End-to-end memory networks, 2015. find
109. Sutskever, I., Vinyals, O., and Le, Q. V. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104–3112, 2014. find
110. Trinkenreich, B., Wiese, I., Sarma, A., Gerosa, M., and Steinmacher, I. Women’s participation in open source software: A survey of the literature. arXiv preprint arXiv:2105.08777, 2021. — [arXiv:2105.08777](https://arxiv.org/abs/2105.08777) find
111. Tufano, M., Watson, C., Bavota, G., Penta, M. D., White, M., and Poshyvanyk, D. An empirical study on learning bug-fixing patches in the wild via neural machine translation. ACM Transactions on Software Engineering and Methodology (TOSEM), 28:1 – 29, 2019. find
112. Tufano, M., Drain, D., Svyatkovskiy, A., Deng, S. K., and Sundaresan, N. Unit test case generation with transformers and focal context. 2020. find
113. Van Oord, A., Kalchbrenner, N., and Kavukcuoglu, K. Pixel recurrent neural networks. In International Conference on Machine Learning, pp. 1747–1756. PMLR, 2016. find
114. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, v… find
115. Wang, B. and Komatsuzaki, A. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax, May 2021. ↗
116. Weston, J., Chopra, S., and Bordes, A. Memory networks, 2015. find
117. Woolf, M. Fun and dystopia with ai-based code generation using gpt-j-6b, June 2021. URL https://minimaxir.com/2021/06/gpt-j-6b/. ↗
118. Xu, F. F., Vasilescu, B., and Neubig, G. In-ide code generation from natural language: Promise and challenges. arXiv preprint arXiv:2101.11149, 2021. — [arXiv:2101.11149](https://arxiv.org/abs/2101.11149) find
119. Yin, P. and Neubig, G. A syntactic neural model for general-purpose code generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 440–450, 2017. find our card
120. Zaremba, W. and Sutskever, I. Learning to execute. arXiv preprint arXiv:1410.4615, 2014. find
121. Zellers, R., Lu, X., Hessel, J., Yu, Y., Park, J. S., Cao, J., Farhadi, A., and Choi, Y. Merlot: Multimodal neural script knowledge models. arXiv preprint arXiv:2106.02636, 2021. — [arXiv:2106.02636](https://arxiv.org/abs/2106.02636) find
122. Zhao, T. Z., Wallace, E., Feng, S., Klein, D., and Singh, S. Calibrate before use: Improving few-shot performance of language models. arXiv preprint arXiv:2102.09690, 2021. find our card
123. Ziegler, A. A first look at rote learning in github copilot suggestions., Jun 2021. URL https://docs.github.com/en/github/copilot/research-recitation. ↗