Elamur Research⌕
☰
home / papers / 2016 / A Convolutional Attention Network for Extreme Summarization of Source Code

A Convolutional Attention Network for Extreme Summarization of Source Code

Paper · 2016 · Miltiadis Allamanis, Hao Peng and Charles Sutton

Summary

The authors study "extreme" summarization of source code. The model receives only the tokens of a code snippet, such as the body of a Java method, and must generate a short, meaningful name that resembles a function name. Earlier approaches depended on predefined features, such as the signature or the class. This method uses no such features, so it works on arbitrary snippets and on dynamically typed languages. The authors propose a neural network with an attention mechanism built on convolutions. Convolutional layers without pooling detect local patterns that do not depend on their position in the code. The GRU state stores information about the parts of the name that have already been generated and selects the patterns that matter at the current step. The second component is a copy mechanism that transfers a token from the code directly into the name. This lets the model handle words that did not occur in the training data. A separate meta-attention decides at each step whether to generate a word or copy one. On 10 popular Java projects, the model outperformed standard attention mechanisms. The work is useful for code comprehension and code search tools. It also shows that code is a distinct modality for machine learning.

Key points

  • Data: 10 popular open-source Java projects; the examples in the paper are taken from libgdx. The input averages 72 subtokens, and the target name about 3.
  • Subtokens: identifiers are split into parts (getInputStream → get, Input, Stream). The name is generated one subtoken at a time between the markers <s> and </s>.
  • Two models: conv_attention uses two convolutions (the first with ReLU), gating by the state h_{t−1}, L2 normalization and SoftMax. copy_attention also mixes generation from the vocabulary with copying from the code. The copy weights κ concentrate on a few tokens, and the probability λ determines the choice between the two mechanisms.
  • Evaluation: four algorithms are compared. The proposed approach performs best, and standard NMT models with attention fall short of it.
  • Regularization: during training, with a probability equal to the dropout rate, the GRU receives the predicted embedding instead of the true one, following the idea of Bengio et al. (2015). Visualizations are published at groups.inf.ed.ac.uk/cup/codeattention.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom381 works · 80,747 citations🎓Peking UniversityBeijing · China315 works · 43,268 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 2Research subjects and languages the work is about.Summarization · 243 worksCode & programming languages · 168 works

Cite

Cite as Miltiadis Allamanis, Hao Peng and Charles Sutton. 2016. A Convolutional Attention Network for Extreme Summarization of Source Code. arXiv:1602.03001.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1602.03001 · arXiv 1602.03001

@misc{allamanis2016convolutional,
  title = {{A Convolutional Attention Network for Extreme Summarization of Source Code}},
  author = {Miltiadis Allamanis, Hao Peng and Charles Sutton},
  year = {2016},
  doi = {10.48550/arxiv.1602.03001},
  eprint = {1602.03001},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1602.03001}
}

References

48 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Allamanis, Miltiadis, Barr, Earl T, Bird, Christian, and Sutton, Charles. Learning natural coding conventions. In Symposium on the Foundations of Software Engineering (FSE), 2014. find
  2. Allamanis, Miltiadis, Barr, Earl T, Bird, Christian, and Sutton, Charles. Suggesting accurate method and class names. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. find
  3. Allamanis, Miltiadis, Tarlow, Daniel, Gordon, Andrew, and Wei, Yi. Bimodal modelling of source code and natural language. In ICML, 2015b. find
  4. Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio, Yoshua. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.
  5. Bengio, Samy, Vinyals, Oriol, Jaitly, Navdeep, and Shazeer, Noam. Scheduled sampling for sequence prediction with recurrent neural networks. arXiv preprint arXiv:1506.03099, 2015. find
  6. Binkley, Dave, Davis, Marcia, Lawrie, Dawn, Maletic, Jonathan I, Morrell, Christopher, and Sharif, Bonita. The impact of identifier style on effort and comprehension. Empirical Software Engineering, 2013. find
  7. Blunsom, Phil, Grefenstette, Edward, Kalchbrenner, Nal, et al. A convolutional neural network for modelling sentences. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, 2014.
  8. Cho, Kyunghyun, van Merriënboer, Bart, Bahdanau, Dzmitry, and Bengio, Yoshua. On the properties of neural machine translation: Encoder–decoder approaches. Syntax, Semantics and Structure in Statistical Translation, pp. 103, 2014. — On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
  9. Collobert, Ronan and Weston, Jason. A unified architecture for natural language processing: deep neural networks with multitask learning. In ICML, 2008. find
  10. Denil, Misha, Demiraj, Alban, Kalchbrenner, Nal, Blunsom, Phil, and de Freitas, Nando. Modelling, visualising and summarising documents with a single convolutional neural network. arXiv preprint arXiv:1406.3830, 2014. — [arXiv:1406.3830](https://arxiv.org/abs/1406.3830) find
  11. Dyer, Chris, Ballesteros, Miguel, Ling, Wang, Matthews, Austin, and Smith, Noah A. Transition-based dependency parsing with stack long short-term memory. In ACL, 2015.
  12. Gousios, Georgios and Spinellis, Diomidis. GHTorrent: Github’s data from a firehose. In Mining Software Repositories (MSR), 2012 9th IEEE Working Conference on, 2012. find
  13. Graves, Alex, Wayne, Greg, and Danihelka, Ivo. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014. find
  14. Grefenstette, Edward, Hermann, Karl Moritz, Suleyman, Mustafa, and Blunsom, Phil. Learning to transduce with unbounded memory. In NIPS, 2015. find
  15. He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian. Delving deep into rectifiers: surpassing human-level performance on ImageNet classification. arXiv preprint arXiv:1502.01852, 2015. find
  16. Hermann, Karl Moritz, Kocisky, Tomas, Grefenstette, Edward, Espeholt, Lasse, Kay, Will, Suleyman, Mustafa, and Blunsom, Phil. Teaching machines to read and comprehend. In NIPS, 2015.
  17. Hindle, Abram, Barr, Earl T, Su, Zhendong, Gabel, Mark, and Devanbu, Premkumar. On the naturalness of software. In International Conference on Software Engineering (ICSE), 2012. find
  18. Hinton, Geoffrey, Srivastava, Nitsh, and Swersky, Kevin. Neural networks for machine learning. Online Course at coursera. org, Lecture, 6, 2012. find
  19. Joulin, Armand and Mikolov, Tomas. Inferring algorithmic patterns with stack-augmented recurrent nets. arXiv preprint arXiv:1503.01007, 2015. find
  20. Karaivanov, Svetoslav, Raychev, Veselin, and Vechev, Martin. Phrase-based statistical translation of programming languages. In Proceedings of the 2014 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming & Software. find
  21. Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. ImageNet classification with deep convolutional neural networks. In NIPS, 2012.
  22. LeCun, Y., Boser, B., Denker, J. S., Howard, R. E., Hab-bard, W., Jackel, L. D., and Henderson, D. Nips. chapter Handwritten Digit Recognition with a Back-propagation Network. 1990. find
  23. LeCun, Yann, Bottou, Leon, Bengio, Yoshua, and Haffner, Patrick. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998. find
  24. Liblit, Ben, Begel, Andrew, and Sweetser, Eve. Cognitive perspectives on the role of naming in computer programs. In Proceedings of the 18th Annual Psychology of Programming Workshop, 2006. find
  25. Maas, Andrew L., Hannun, Awni Y., and Ng, Andrew Y. Rectified linear units improve restricted Boltzmann machines. In ICML Workshop on Deep Learning for Audio, Speech, and Language Processing (WDLASL), 2013. find
  26. Maddison, Chris and Tarlow, Daniel. Structured generative models of natural source code. In ICML, 2014. find
  27. Mnih, Volodymyr, Heess, Nicolas, Graves, Alex, et al. Recurrent models of visual attention. In NIPS, 2014. find
  28. Mou, Lili, Li, Ge, Zhang, Lu, Wang, Tao, and Jin, Zhi. Convolutional neural networks over tree structures for programming language processing. In Proceedings of the 30th AAAI Conference on Artificial Intelligence, 2016. find
  29. Movshovitz-Attias, Dana, Cohen, WW, and W. Cohen, William. Natural language models for predicting programming comments. In ACL, 2013. find
  30. Nair, Vinod and Hinton, Geoffrey E. Rectified linear units improve restricted Boltzmann machines. In ICML, 2010. find
  31. Neelakantan, Arvind, Le, Quoc V, and Sutskever, Ilya. Neural programmer: Inducing latent programs with gradient descent. arXiv preprint arXiv:1511.04834, 2015. find
  32. Nguyen, Anh Tuan, Nguyen, Tung Thanh, and Nguyen, Tien N. Migrating code with statistical machine translation. In Companion Proceedings of the 36th International Conference on Software Engineering. ACM, 2014. find
  33. Nguyen, Tung Thanh, Nguyen, Anh Tuan, Nguyen, Hoan Anh, and Nguyen, Tien N. A statistical semantic language model for source code. In Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE), 2013. find
  34. Oda, Yusuke, Fudaba, Hiroyuki, Neubig, Graham, Hata, Hideaki, Sakti, Sakriani, Toda, Tomoki, and Nakamura, Satoshi. Learning to generate pseudo-code from source code using statistical machine translation. In Automated Software Engineering (ASE), 2015 30th IEEE/ACM International Conference on. find
  35. Papineni, Kishore, Roukos, Salim, Ward, Todd, and Zhu, Wei-Jing. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, 2002.
  36. Piech, Chris, Huang, Jonathan, Nguyen, Andy, Phulsuk-sombati, Mike, Sahami, Mehran, and Guibas, Leonidas J. Learning program embeddings to propagate feedback on student code. In ICML, 2015. find
  37. Raychev, Veselin, Vechev, Martin, and Krause, Andreas. Predicting program properties from “big code”. In Proceedings of the 42nd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages. ACM, 2015. find
  38. Reed, Scott and de Freitas, Nando. Neural programmer-interpreters. arXiv preprint arXiv:1511.06279, 2015. find
  39. Rush, Alexander M, Chopra, Sumit, and Weston, Jason. A neural attention model for abstractive sentence summarization. In EMNLP, 2015.
  40. Snoek, Jasper, Larochelle, Hugo, and Adams, Ryan P. Practical bayesian optimization of machine learning algorithms. In NIPS, 2012. find
  41. Srivastava, Nitish, Hinton, Geoffrey E., Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 2014.
  42. Sutskever, Ilya, Martens, James, Dahl, George, and Hinton, Geoffrey. On the importance of initialization and momentum in deep learning. In ICML, 2013. find
  43. Szegedy, Christian, Liu, Wei, Jia, Yangqing, Sermanet, Pierre, Reed, Scott, Anguelov, Dragomir, Erhan, Dumitru, Vanhoucke, Vincent, and Rabinovich, Andrew. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015. find
  44. Takang, Armstrong A, Grubb, Penny A, and Macredie, Robert D. The effects of comments and identifier names on program comprehensibility: an experimental investigation. J. Prog. Lang., 1996. find
  45. Tu, Zhaopeng, Su, Zhendong, and Devanbu, Premkumar. On the localness of software. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, 2014. find
  46. Vinyals, Oriol, Fortunato, Meire, and Jaitly, Navdeep. Pointer networks. In NIPS, 2015. find
  47. Xu, Kelvin, Ba, Jimmy, Kiros, Ryan, Cho, Kyunghyun, Courville, Aaron C, Salakhutdinov, Ruslan, Zemel, Richard S, and Bengio, Yoshua. Show, attend and tell: Neural image caption generation with visual attention. In ICML, 2015.
  48. Zaremba, Wojciech and Sutskever, Ilya. Learning to execute. arXiv preprint arXiv:1410.4615, 2014. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/allamanis-2016-convolutional-attention-network-extreme-summarization/