Elamur Research
☰
home / papers / 2021 / Program Synthesis with Large Language Models

Program Synthesis with Large Language Models

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2108.07732

Summary

The paper asks how far general-purpose large language models can go at writing short programs in a mainstream language like Python. Earlier synthesis work mostly targeted narrow DSLs built for that purpose. The authors release two benchmarks. MBPP (Mostly Basic Programming Problems) holds 974 crowd-sourced, entry-level tasks, each with a plain-language description, a reference solution and three assert-based tests. MathQA-Python has 23,914 math word problems whose MathQA solutions were rewritten as Python. They test Transformer LMs from 244M to 137B parameters, using both few-shot prompting and fine-tuning. Accuracy rises roughly log-linearly with model size. The largest model solves most MBPP tasks, but it often needs many samples that are then checked against the tests. Natural-language hints from a person repair many errors. Still, the models mostly cannot predict what a given program outputs. The work gives reusable benchmarks for code generation and shows that LMs write code fluently while understanding little of what it does.

Key points

  • On MBPP, the 137B model solves 59.6% of problems few-shot, with no code-specific fine-tuning. Fine-tuning on just 374 tasks adds about 10 percentage points for most model sizes.
  • A hand-checked, edited subset of MBPP is easier: on 100 shared problems, few-shot accuracy goes from 63% to 79%.
  • On MathQA-Python, the largest model reaches 33.4% few-shot and 83.8% after fine-tuning.
  • Four rounds of dialog with human feedback raise success from 30% to 65%, which halves the error rate.
  • Models do poorly at predicting program outputs, both few-shot and fine-tuned. Overlap between MBPP and the pre-training data is small, so memorization is unlikely. The work is related to APPS and to Codex/HumanEval.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Computing & GPUs

Cited by

1108 papers of our library cite it (of the 140673 whose bibliographies we read); 80 of them have a page here, 1028 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2108.07732 · arXiv 2108.07732

@misc{austin2021program,
  title = {{Program Synthesis with Large Language Models}},
  author = {Austin, Jacob and Odena, Augustus and Nye, Maxwell and Bosma, Maarten and Michalewski, Henryk and Dohan, David and Jiang, Ellen and Cai, Carrie and Terry, Michael and Le, Quoc and Sutton, Charles},
  year = {2021},
  doi = {10.48550/arxiv.2108.07732},
  eprint = {2108.07732},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2108.07732}
}

References

105 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ferran Alet, Javier Lopez-Contreras, James Koppel, Maxwell Nye, Armando Solar-Lezama, Tomas Lozano-Perez, Leslie Kaelbling, and Joshua Tenenbaum. A large-scale benchmark for few-shot program induction and synthesis. In Marina Meila and Tong Zhang, editors, International Conference on Machine Learnin… find
  2. Miltiadis Allamanis. A survey of machine learning on source code. https://ml4code.github.io/, 2021. Accessed 2021-07-11. ↗
  3. Miltiadis Allamanis, Earl T Barr, Christian Bird, and Charles Sutton. Learning natural coding conventions. In Symposium on the Foundations of Software Engineering (FSE), 2014. find
  4. Miltiadis Allamanis, Hao Peng, and Charles Sutton. A Convolutional Attention Network for Extreme Summarization of Source Code. In International Conference in Machine Learning (ICML), 2016. find
  5. Miltiadis Allamanis, Earl T Barr, Premkumar Devanbu, and Charles Sutton. A survey of machine learning for big code and naturalness. ACM Computing Surveys, 51(4), September 2018a. find
  6. Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. Learning to represent programs with graphs. In International Conference on Learning Representations (ICLR), February 2018b. find
  7. Miltos Allamanis and Charles Sutton. Mining source code repositories at massive scale using language modeling. In Working Conference on Mining Software Repositories (MSR), 2013a. find
  8. Miltos Allamanis and Charles Sutton. Mining source code repositories at massive scale using language modeling. In Working Conference on Mining Software Repositories (MSR), 2013b. find
  9. Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. code2vec: Learning distributed representations of code. Proceedings of the ACM on Programming Languages, 3(POPL):40, 2019. find
  10. Rajeev Alur, Rastislav Bodík, Garvit Juniwal, Milo M. K. Martin, Mukund Raghothaman, Sanjit A. Seshia, Rishabh Singh, Armando Solar-Lezama, Emina Torlak, and Abhishek Udupa. Syntax-guided synthesis. In Formal Methods in Computer-Aided Design, FMCAD 2013, Portland, OR, USA, October 20-23, 2013, pages… find
  11. IEEE, 2013. URL http://ieeexplore.ieee.org/document/6679385/. ↗
  12. Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. CoRR, abs/1905.13319, 2019. URL http://arxiv.org/abs/1905.13319. ↗ our card
  13. J. W. Backus, R. J. Beeber, S. Best, R. Goldberg, L. M. Haibt, H. L. Herrick, R. A. Nelson, D. Sayre, P. B. Sheridan, H. Stern, I. Ziller, R. A. Hughes, and R. Nutt. The FORTRAN automatic coding system. In Papers Presented at the February 26-28, 1957, Western Joint Computer Conference: Techniques fo… — [doi:10.1145/1455567.1455599](https://doi.org/10.1145/1455567.1455599) find
  14. Matej Balog, Alexander L Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. Deepcoder: Learning to write programs. In International Conference on Learning Representations (ICLR), 2017. arXiv:1611.01989. find
  15. Emily M. Bender and Alexander Koller. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185– 5198, Online, July 2020. Association for Computational Linguistics. doi: 10.1865… find our card
  16. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, pages 610–623, New York, NY, USA, March 2021. As… find
  17. David Bieber, Charles Sutton, Hugo Larochelle, and Daniel Tarlow. Learning to execute programs with instruction pointer attention graph neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 86… find
  18. Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large scale autoregressive language modeling with mesh-tensorflow, 2021. URL http://github.com/eleutherai/gpt-neo. ↗
  19. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… find
  20. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. arXiv preprint arXiv:2012.07805, 2020. find our card
  21. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea… find our card
  22. Xinyun Chen, Chang Liu, and Dawn Song. Execution-guided neural program synthesis. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. find
  23. OpenReview.net, 2019a. URL https://openreview.net/forum?id=H1gfOiAqYm. ↗
  24. Zimin Chen, S J Kommrusch, M Tufano, Louis-Noël Pouchet, Denys Poshyvanyk, and Martin Monperrus. Sequencer: Sequence-to-sequence learning for end-to-end program repair. IEEE Transactions on Software Engineering, 2019b. find
  25. Colin B. Clement, Dawn Drain, Jonathan Timcheck, Alexey Svyatkovskiy, and Neel Sundaresan. PyMT5: Multi-mode translation of natural language and python code with transformers. 2020. URL http://arxiv.org/abs/2010.03150. ↗
  26. B.J. Copeland. Alan Turing’s Electronic Brain: The Struggle to Build the ACE, the World’s Fastest Computer. OUP Oxford, 2012. ISBN 9780199609154. URL https://books.google.com/books?id=YhQZnczOS7kC. ↗
  27. Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. In Neural Information Processing Systems, 2015. find
  28. Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. Robustfill: Neural program learning under noisy I/O. CoRR, abs/1703.07469, 2017. URL http://arxiv.org/abs/1703.07469. ↗
  29. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019. find
  30. Elizabeth Dinella, Hanjun Dai, Ziyang Li, Mayur Naik, Le Song, and Ke Wang. Hoppity: Learning graph transformations to detect and fix bugs in programs. In International Conference on Learning Representations, September 2019. find
  31. Kevin Ellis, Lucas Morales, Mathias Sablé-Meyer, Armando Solar-Lezama, and Josh Tenenbaum. Learning libraries of subroutines for neurally–guided bayesian program induction. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information P… find
  32. Kevin Ellis, Maxwell Nye, Yewen Pu, Felix Sosa, Josh Tenenbaum, and Armando Solar-Lezama. Write, execute, assess: Program synthesis with a REPL. In NeurIPS, 2019. find
  33. Kevin Ellis, Catherine Wong, Maxwell I. Nye, Mathias Sablé-Meyer, Luc Cary, Lucas Morales, Luke B. Hewitt, Armando Solar-Lezama, and Joshua B. Tenenbaum. Dreamcoder: Growing generalizable, interpretable knowledge with wake-sleep bayesian program learning. CoRR, abs/2006.08381, 2020. URL https://arxi… find
  34. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. CodeBERT: A pre-trained model for programming and natural languages. Findings of the Association for Computational Linguistics: EMNLP 2020, February 2020. find our card
  35. Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. CoRR, abs/1410.5401, 2014. find
  36. Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwinska, Sergio Gomez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, Adrià Puigdomènech Badia, Karl Moritz Hermann, Yori Zwols, Georg Ostrovski, Adam Cain, Helen King, Christopher Summerfield, P… find
  37. Sumit Gulwani. Automating string processing in spreadsheets using input-output examples. In Proceedings of the 38th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL ’11, pages 317–330, New York, NY, USA, 2011. ACM. ISBN 978-1-4503-0490-0. doi: 10.1145/1926385.1926423.… find
  38. Sumit Gulwani, Oleksandr Polozov, Rishabh Singh, et al. Program synthesis. Foundations and Trends® in Programming Languages, 4(1-2):1–119, 2017. find
  39. Vincent J Hellendoorn, Christian Bird, Earl T Barr, and Miltiadis Allamanis. Deep learning type inference. In ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), pages 152–162, New York, New York, USA, 2018. ACM Press. find
  40. Vincent J Hellendoorn, Charles Sutton, Rishabh Singh, Petros Maniatis, and David Bieber. Global relational models of source code. In International Conference on Learning Representations (ICLR), September 2019. find
  41. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS. May 2021. URL http://arxiv.org/abs/2105.09938. ↗ our card
  42. Abram Hindle, Earl Barr, Zhendong Su, Prem Devanbu, and Mark Gable. On the “naturalness” of software. In International Conference on Software Engineering (ICSE). 2012. find
  43. Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Association of Computational Linguistics (ACL), 2018. find
  44. Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. CodeSearchNet challenge: Evaluating the state of semantic code search. September 2019. URL http://arxiv.org/abs/1909.09436. ↗ our card
  45. Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. Mapping language to code in programmatic context. In Empirical Methods in Natural Language Processing (EMNLP), August 2018. find
  46. Ellen Jiang, Edwin Toh, Alejandra Molina, Aaron Donsbach, Carrie Cai, and Michael Terry. Genline and genform: Two tools for interacting with generative language models in a code editor. Adjunct Publication of the 34th Annual ACM Symposium on User Interface Software and Technology, 2021. find
  47. Rafal Józefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016. find
  48. Lukasz Kaiser and Ilya Sutskever. Neural gpus learn algorithms. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. find
  49. Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. Learning and evaluating contextual embedding of source code. In International Conference on Machine Learning (ICML), 2020. find
  50. Svetoslav Karaivanov, Veselin Raychev, and Martin Vechev. Phrase-Based statistical translation of programming languages. In Onward, 2014. find
  51. Rafael-Michael Karampatsis, Hlib Babii, Romain Robbes, Charles Sutton, and Andrea Janes. Big code != big vocabulary: Open-Vocabulary models for source code. In International Conference on Software Engineering (ICSE), March 2020. find
  52. Seohyun Kim, Jinman Zhao, Yuchi Tian, and Satish Chandra. Code prediction by feeding trees to transformers. March 2020. URL http://arxiv.org/abs/2003.13848. ↗
  53. Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Eduardo Blanco and Wei Lu, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018: System Demonstratio… find our card
  54. Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy Liang. SPoC: Search-based pseudocode to code. In Advances in Neural Information Processing Systems, 2019. find
  55. Karol Kurach, Marcin Andrychowicz, and Ilya Sutskever. Neural random-access machines. In International Conference on Learning Representations, (ICLR), 2016. find
  56. Claire Le Goues, Thanhvu Nguyen, Stephanie Forrest, and Westley Weimer. GenProg: A generic method for automatic software repair. IEEE Trans. Software Eng., 38(1):54–72, January 2012. find
  57. Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. April 2021. URL http://arxiv.org/abs/2104.08691. ↗
  58. Belinda Z. Li, Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural language models. ArXiv, abs/2106.00737, 2021. find
  59. Xiang Lisa Li and Percy Liang. Prefix-Tuning: Optimizing continuous prompts for generation. January 2021. URL http://arxiv.org/abs/2101.00190. ↗
  60. Wang Ling, Phil Blunsom, Edward Grefenstette, Karl Moritz Hermann, Tomás Kociský, Fumin Wang, and Andrew Senior. Latent predictor networks for code generation. In Annual Meeting of the Association for Computational Linguistics (ACL), 2016. find
  61. Fan Long and Martin Rinard. Automatic patch generation by learning correct code. In ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL), pages 298–312, New York, NY, USA, January 2016. Association for Computing Machinery. find
  62. Annie Louis, Santanu Kumar Dash, Earl T Barr, Michael D Ernst, and Charles Sutton. Where should I comment my code? A dataset and model for predicting locations that need comments. In International Conference on Software Engineering (ICSE; NIER track), 2020. find
  63. Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. CodeXGLUE: A ma… find
  64. Chris J Maddison and Daniel Tarlow. Structured generative models of natural source code. In International Conference on Machine Learning (ICML), pages 649–657. 2014. find
  65. Zohar Manna and Richard Waldinger. Knowledge and reasoning in program synthesis. Artificial Intelligence, 6(2): 175–208, 1975. find
  66. Zohar Manna and Richard J Waldinger. Toward automatic program synthesis. Communications of the ACM, 14(3): 151–165, 1971. find
  67. Antonio Mastropaolo, Simone Scalabrino, Nathan Cooper, David Nader Palacio, Denys Poshyvanyk, Rocco Oliveto, and Gabriele Bavota. Studying the usage of Text-To-Text transfer transformer to support Code-Related tasks. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), page… find
  68. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. Recurrent neural network based language model. In INTERSPEECH, 2010. find
  69. Vijayaraghavan Murali, Letao Qi, Swarat Chaudhuri, and Chris Jermaine. Neural sketch learning for conditional program generation. In International Conference on Learning Representations (ICLR), 2018. find
  70. Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N Nguyen. Lexical statistical machine translation for language migration. In Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE), NIER, 2013. find
  71. Augustus Odena and Charles Sutton. Learning to represent programs with property signatures. CoRR, abs/2002.09030, 2020. URL https://arxiv.org/abs/2002.09030. — [arXiv:2002.09030](https://arxiv.org/abs/2002.09030) ↗
  72. Augustus Odena, Kensen Shi, David Bieber, Rishabh Singh, and Charles Sutton. BUSTLE: bottom-up program-synthesis through learning-guided exploration. CoRR, abs/2007.14381, 2020. URL https://arxiv.org/abs/2007.14381. ↗
  73. Irene Vlassi Pandi, Earl T Barr, Andrew D Gordon, and Charles Sutton. OptTyper: Probabilistic type inference by optimising logical and natural constraints. April 2020. URL http://arxiv.org/abs/2004.00348. — [arXiv:2004.00348](https://arxiv.org/abs/2004.00348) ↗
  74. Sheena Panthaplackel, Junyi Jessy Li, Milos Gligoric, and Raymond J Mooney. Deep Just-In-Time inconsistency detection between comments and source code. In AAAI Conference on Artificial Intelligence, 2021. find
  75. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In North American Chapter of the Association for Computational Linguistics (NAACL), 2018. find
  76. Amir Pnueli and Roni Rosner. On the synthesis of a reactive module. In SIGPLAN-SIGACT symposium on Principles of Programming Languages (POPL), pages 179–190. ACM, 1989. find
  77. Michael Pradel and Koushik Sen. DeepBugs: a learning approach to name-based bug detection. Proc. ACM Program. Lang., 2(OOPSLA):1–25, October 2018. find
  78. Michael Pradel, Georgios Gousios, Jason Liu, and Satish Chandra. TypeWriter: neural type prediction with search-based validation. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2020, pages 20… find
  79. Ruchir Puri, David S Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladmir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, and Ulrich Finkler. Project CodeNet: A Large-Scale AI for code dataset for learning a diversity of coding ta… find
  80. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. https://blog.openai.com/language-unsupervised, 2018. ↗
  81. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. find
  82. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.10683, 2019. URL http://arxiv.org/abs/1910.10683. ↗
  83. Veselin Raychev, Martin Vechev, and Eran Yahav. Code completion with statistical language models. In ACM PLDI, 2014. find
  84. Veselin Raychev, Martin Vechev, and Andreas Krause. Predicting program properties from “big code”. In ACM Symposium on Principles of Programming Languages (POPL), 2015. find
  85. Veselin Raychev, Pavol Bielik, and Martin Vechev. Probabilistic model for code with decision trees. In OOPSLA, 2016. find
  86. Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. Unsupervised translation of programming languages. Adv. Neural Inf. Process. Syst., 33:20601–20611, 2020. find our card
  87. Roei Schuster, Congzheng Song, Eran Tromer, and Vitaly Shmatikov. You autocomplete me: Poisoning vulnerabilities in neural code completion. In 30th USENIX Security Symposium (USENIX Security 21), July 2020. find
  88. Tal Schuster, Ashwin Kalyan, Oleksandr Polozov, and Adam Tauman Kalai. Programming puzzles. June 2021. URL http://arxiv.org/abs/2106.05784. — [arXiv:2106.05784](https://arxiv.org/abs/2106.05784) ↗
  89. David E. Shaw, William R. Swartout, and C. Cordell Green. Inferring LISP programs from examples. In International Joint Conference on Artificial Intelligence (IJCAI), pages 260–267, 1975. URL http://ijcai.org/Proceedings/75/Papers/037.pdf. ↗
  90. Armando Solar-Lezama. Introduction to program synthesis. https://people.csail.mit.edu/asolar/SynthesisCourse/TOC.htm, 2018. Accessed: 2018-09-17. ↗
  91. Armando Solar-Lezama, Liviu Tancau, Rastislav Bodík, Sanjit A. Seshia, and Vijay A. Saraswat. Combinatorial sketching for finite programs. In Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS 2006, San Jose, CA, USA, October 21-25, 2006, pages 404–415. find
  92. Phillip D Summers. A methodology for LISP program construction from examples. Journal of the ACM (JACM), 24(1): 161–175, 1977. find
  93. Ilya Sutskever, James Martens, and Geoffrey E Hinton. Generating text with recurrent neural networks. In International Conference on Machine Learning (ICML), 2011. find
  94. Jeffrey Svajlenko, Judith F Islam, Iman Keivanloo, Chanchal K Roy, and Mohammad Mamun Mia. Towards a big data curated benchmark of inter-project code clones. In 2014 IEEE International Conference on Software Maintenance and Evolution, pages 476–480, September 2014. find
  95. Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. IntelliCode compose: Code generation using transformer. May 2020. URL http://arxiv.org/abs/2005.08025. ↗
  96. Daniel Tarlow, Subhodeep Moitra, Andrew Rice, Zimin Chen, Pierre-Antoine Manzagol, Charles Sutton, and Edward Aftandilian. Learning to fix build errors with Graph2Diff neural networks. November 2019. URL http://arxiv. org/abs/1911.01205. find
  97. Emina Torlak and Rastislav Bodik. Growing solver-aided languages with rosette. In ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming & Software, Onward! 2013, pages 135–152, New York, NY, USA, October 2013. Association for Computing Machinery. find
  98. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017. find
  99. R.J. Waldinger, R.C.T. Lee, and SRI International. PROW: A Step Toward Automatic Program Writing. SRI International, 1969. URL https://books.google.com/books?id=3BITSQAACAAJ. ↗
  100. Jiayi Wei, Maruth Goyal, Greg Durrett, and Isil Dillig. LambdaNet: Probabilistic type inference using graph neural networks. In International Conference on Learning Representations, 2020. find
  101. Michihiro Yasunaga and Percy Liang. Graph-based, self-supervised program repair from diagnostic feedback. In International Conference on Machine Learning, May 2020. find
  102. Pengcheng Yin and Graham Neubig. A syntactic neural model for general-purpose code generation. In Association for Computational Linguistics (ACL), 2017. find our card
  103. Wojciech Zaremba and Ilya Sutskever. Learning to execute. ArXiv, abs/1410.4615, 2014. find
  104. Maksym Zavershynskyi, Alex Skidanov, and Illia Polosukhin. NAPS: Natural program synthesis dataset. In Workshop on Neural Abstract Machines & Program Induction (NAMPI), July 2018. find
  105. Amit Zohar and Lior Wolf. Automatic program synthesis of long programs with a learned garbage collector. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 2094–2103. Curran Associates, Inc., 2018… find