Elamur Research⌕
☰
home / papers / 2018 / code2seq: Generating Sequences from Structured Representations of Code

code2seq: Generating Sequences from Structured Representations of Code

Paper · 2018 · Uri Alon, Shaked Brody, Omer Levy et al.

Summary

The paper addresses the automatic generation of a text sequence from a code snippet. Examples include generating a method name or a short natural-language description of the code. Earlier state-of-the-art approaches took machine translation models (seq2seq, NMT) and represented code as a flat sequence of tokens. The authors propose code2seq, which instead uses the syntactic structure of the program. A code snippet is parsed into an abstract syntax tree (AST) and represented as a set of paths between pairs of leaves in this tree. Each path is encoded with a bi-directional LSTM. The values of the terminal nodes are split into subtokens, and their vectors are summed. At each training iteration, the paths are resampled at random. At each generation step, the decoder focuses its attention on the most relevant paths. The model was evaluated on two tasks (Java method names and C# snippet descriptions), two languages and four datasets, the largest of which contains about 16M examples. code2seq outperformed both models designed specifically for code and strong NMT architectures. Thus, a structural representation of code gives better results than treating code as plain text, and the approach can be applied to any language that has a parser.

Key points

  • Encoding idea: a program is represented as an unordered set of k random paths in the AST. The decoder state is initialized with the average of these path vectors. The vocabulary of path nodes has up to 364 symbols.
  • Java code summarization (method name prediction): code2seq achieves an F1 of 43.02 on Java-small, 53.23 on Java-med and 59.19 on Java-large. This is higher than ConvAttention, Paths+CRFs, code2vec, BiLSTM, TreeLSTM and Transformer.
  • C# code captioning on StackOverflow data: BLEU 23.04 versus 20.53 for CodeNN. The other baselines (MOSES, IR, SUM-NN, Transformer, TreeLSTM) scored even lower.
  • Ablation: performance drops when AST nodes, splitting tokens into subtokens, attention, path resampling or the full decoder are removed. A model with tokens only and no syntactic paths is much weaker than the full model.
  • Open resources: the code, data and trained models are published at github.com/tech-srl/code2seq, and a demo is available at code2seq.org. The model also generates Javadoc more accurately than prior work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelf 1The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 works

🧭 Topics 1Research subjects and languages the work is about.Machine translation · 1,105 works

Cite

Cite as Uri Alon, Shaked Brody, Omer Levy and Eran Yahav. 2018. code2seq: Generating Sequences from Structured Representations of Code. arXiv:1808.01400.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1808.01400 · arXiv 1808.01400

@misc{alon2018code2seq,
  title = {{code2seq: Generating Sequences from Structured Representations of Code}},
  author = {Uri Alon, Shaked Brody, Omer Levy and Eran Yahav},
  year = {2018},
  doi = {10.48550/arxiv.1808.01400},
  eprint = {1808.01400},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1808.01400}
}

References

43 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Miltiadis Allamanis, Earl T. Barr, Christian Bird, and Charles Sutton. Suggesting accurate method and class names. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, pages 38–49, New York, NY, USA, 2015a. ACM. ISBN 978-1-4503-3675-8. doi:10.1145/2786… — [doi:10.1145/2786805.2786849](https://doi.org/10.1145/2786805.2786849) find
  2. Miltiadis Allamanis, Daniel Tarlow, Andrew D. Gordon, and Yi Wei. Bimodal Modelling of Source Code and Natural Language. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of JMLR Proceedings, pages 2123–2132. JMLR.org, 2015b. find
  3. Miltiadis Allamanis, Hao Peng, and Charles A. Sutton. A convolutional attention network for extreme summarization of source code. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, pages 2091–2100, 2016. URL http://jmlr.org/p… ↗
  4. Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. Learning to represent programs with graphs. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=BJOFETxR-. ↗
  5. Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. A general path-based representation for predicting program properties. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2018, pages 404–419, New York, NY, USA, 2018. ACM. ISBN 978-1-4503… find
  6. Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. Code2vec: Learning distributed representations of code. Proc. ACM Program. Lang., 3(POPL):40:1–40:29, January 2019. ISSN 2475-1421. doi:10.1145/3290353. URL http://doi.acm.org/10.1145/3290353. ↗
  7. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473, 2014. URL http://arxiv.org/abs/1409.0473. ↗
  8. Matej Balog, Alexander L Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. Deepcoder: Learning to write programs. In ICLR, 2017. find
  9. Joost Bastings, Ivan Titov, Wilker Aziz, Diego Marcheggiani, and Khalil Simaan. Graph convolutional encoders for syntax-aware neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1957–1967, Copenhagen, Denmark, September 2017. A…
  10. Pavol Bielik, Veselin Raychev, and Martin T. Vechev. PHOG: probabilistic model for code. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, pages 2933–2942, 2016. URL http://jmlr.org/proceedings/papers/v48/bielik16.html. ↗
  11. Pavol Bielik, Veselin Raychev, and Martin Vechev. Program synthesis for character level language modeling. In ICLR, 2017. find
  12. Marc Brockschmidt, Miltiadis Allamanis, Alexander L. Gaunt, and Oleksandr Polozov. Generative code modeling with graphs. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bke4KsA5FX. ↗
  13. Xinyun Chen, Chang Liu, and Dawn Song. Tree-to-tree neural networks for program translation. CoRR, abs/1802.03691, 2018. URL http://arxiv.org/abs/1802.03691. ↗
  14. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.
  15. Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. Robustfill: Neural program learning under noisy i/o. In International Conference on Machine Learning, pages 990–998, 2017. find
  16. Patrick Fernandes, Miltiadis Allamanis, and Marc Brockschmidt. Structured neural summarization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=H1ersoRqtm. ↗
  17. Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 249–256, 2010. find
  18. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 9(8):1735– 1780, November 1997. ISSN 0899-7667. doi:10.1162/neco.1997.9.8.1735. URL http://dx.doi.org/10.1162/neco.1997.9.8.1735. ↗
  19. Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. Deep code comment generation. In Proceedings of the 26th Conference on Program Comprehension, pages 200–210. ACM, 2018. find
  20. Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. Summarizing source code using a neural attention model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers, 2016. URL htt… find
  21. G. Klein, Y. Kim, Y. Deng, J. Senellart, and A. M. Rush. OpenNMT: Open-Source Toolkit for Neural Machine Translation. ArXiv e-prints, 2017.
  22. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. Moses: Open source toolkit for statistical machine translation. In Proceedin… find
  23. Pablo Loyola, Edison Marrese-Taylor, and Yutaka Matsuo. A neural architecture for generating natural language descriptions from source code changes. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 287– 292. Association for Co… find
  24. Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 1412–1421, 2015. URL http://ac…
  25. Vijayaraghavan Murali, Swarat Chaudhuri, and Chris Jermaine. Bayesian sketch learning for program synthesis. CoRR, abs/1703.05698, 2017. URL http://arxiv.org/abs/1703.05698. ↗
  26. Yurii E Nesterov. A method for solving the convex programming problem with convergence rate o (1/kˆ 2). In Dokl. Akad. Nauk SSSR, volume 269, pages 543–547, 1983. find
  27. Yusuke Oda, Hiroyuki Fudaba, Graham Neubig, Hideaki Hata, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. Learning to generate pseudo-code from source code using statistical machine translation (t). In Automated Software Engineering (ASE), 2015 30th IEEE/ACM International Conference on, pages 574… find
  28. Chris Piech, Jonathan Huang, Andy Nguyen, Mike Phulsuksombati, Mehran Sahami, and Leonidas Guibas. Learning program embeddings to propagate feedback on student code. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37, ICML’15, pages 1093–1… find
  29. Maxim Rabinovich, Mitchell Stern, and Dan Klein. Abstract syntax networks for code generation and semantic parsing. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1139–1149. Association for Computational Linguistics, 2017. do… find
  30. Veselin Raychev, Martin Vechev, and Eran Yahav. Code completion with statistical language models. SIGPLAN Not., 49(6):419–428, June 2014. ISSN 0362-1340. doi:10.1145/2666356.2594321. URL http://doi.acm.org/10.1145/2666356.2594321. — [doi:10.1145/2666356.2594321](https://doi.org/10.1145/2666356.2594321) ↗
  31. Veselin Raychev, Martin Vechev, and Andreas Krause. Predicting program properties from ”big code”. In Proceedings of the 42Nd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL ’15, pages 111–124, New York, NY, USA, 2015. ACM. ISBN 978-1-4503-3300-9. doi:10.1145/2676726… find
  32. Veselin Raychev, Pavol Bielik, and Martin Vechev. Probabilistic model for code with decision trees. In Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2016, pages 731–747, New York, NY, USA, 2016. ACM. ISBN 978… find
  33. Reuven Rubinstein. The cross-entropy method for combinatorial and continuous optimization. Methodology and Computing in Applied Probability, 1(2):127–190, 1999. find
  34. Reuven Y Rubinstein. Combinatorial optimization, cross-entropy, ants and rare events. Stochastic Optimization: Algorithms and Applications, 54:303–363, 2001. find
  35. Alexander M. Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 379–389, 2015. URL http://aclweb.o… ↗
  36. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany, August 2016. Association for Computatio…
  37. Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014.
  38. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pages 3104–3112, 2014.
  39. Kai Sheng Tai, Richard Socher, and Christopher D. Manning. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language… find
  40. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pages 6000–6010, 2017.
  41. Pengcheng Yin and Graham Neubig. A syntactic neural model for general-purpose code generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 440–450. Association for Computational Linguistics, 2017. doi:10.18653/v1/P17-1041.…
  42. Table 4: Our model significantly outperforms code2vec on the code2vec dataset. Our Java-large dataset (same as in Table 1) Model Prec Rec code2vec (Alon et al., 2019) 48.2 38.4 64.0 55.02 code2seq (this work) find
  43. Table 5: Statistics of our datasets. \#projects - training \#projects - validation \#projects - test \#examples - training \#examples - validation \#examples - test Avg. number of paths (training) Avg. code length - lines (training) Avg. code length - tokens (training) Avg. code length - subtokenized (tra… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alon-2018-code2seq-generating-sequences-structured-representations/