A Syntactic Neural Model for General-Purpose Code Generation
Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P17-1041.pdf
Summary
This paper looks at turning natural-language descriptions into code in a general-purpose language such as Python. Earlier neural work, notably Ling et al. (2016), treated this as plain sequence-to-sequence generation, so nothing ensured the output was a valid program. Yin and Neubig instead have the model build an Abstract Syntax Tree (AST), which the astor library then turns into source code. A grammar model breaks tree construction into a series of steps. APPLY RULE steps pick production rules from the Python abstract grammar, and GEN TOKEN steps fill in terminal values, either from a vocabulary or by copying them from the input. Because the grammar is built in, the network does not have to learn syntax from limited data. The RNN decoder also gets extra "parent feeding" connections that follow the tree's structure. On the HS (HearthStone) and DJANGO code generation benchmarks, the model clearly outperforms the previous state of the art. It is also competitive on the IFTTT semantic parsing benchmark. The work shows that treating program syntax as prior knowledge helps neural models produce longer and more complex code.
Key points
- Absolute accuracy gains over Ling et al. (2016) were 11.7% and 9.3% on the two Python tasks (HS and DJANGO).
- On IFTTT, full-tree accuracy reached 82.0%, the best among the neural models and close to a logistic-regression baseline with hand-built features. The smaller gap there suggests structure matters most for complex, general-purpose code.
- Other techniques include unary closures, which merge chains of single-child rules into one step, a pointer-style mechanism for copying rare variable names and literals, and depth-first, left-to-right tree expansion.
- Error analysis of failed DJANGO outputs: about 30% were wrong copies and about 25% were partial implementations. Only about 2% of failures across both datasets were actually equivalent code, which supports exact match as the metric.
- The implementation is public as NL2code (github.com/neulab/NL2code).
Cited by
- Unified Pre-training for Program Understanding and Generation
- Program Synthesis with Large Language Models
- TabFact: A Large-scale Dataset for Table-based Fact Verification
- Evaluating Large Language Models Trained on Code
- Binding Language Models in Symbolic Languages
- FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios
- DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
- GraphCodeBERT: Pre-training Code Representations with Data Flow
- CodeSearchNet Challenge: Evaluating the State of Semantic Code Search
- Unsupervised Translation of Programming Languages
- CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning
- Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow
- Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task
Cite
@inproceedings{yin2017syntactic,
title = {{A Syntactic Neural Model for General-Purpose Code Generation}},
author = {Yin, Pengcheng and Neubig, Graham},
booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
year = {2017},
doi = {10.18653/v1/p17-1041},
eprint = {1704.01696},
archivePrefix = {arXiv},
url = {https://doi.org/10.18653/v1/p17-1041}
}References
- Miltiadis Allamanis, Daniel Tarlow, Andrew D. Gordon, and Yi Wei. 2015. Bimodal modelling of source code and natural language. In Proceedings of ICML. volume 37. find
- David Alvarez-Melis and Tommi S. Jaakkola. 2017. Tree-structured decoding with doubly recurrent neural networks. In Proceedings of ICLR. find