Elamur Research⌕
☰
home / papers / 2015 / Bimodal Modelling of Source Code and Natural Language

Bimodal Modelling of Source Code and Natural Language

Paper · 2015 · Miltos Allamanis, Daniel Tarlow, Andrew Gordon et al.

Summary

The authors build a probabilistic model that jointly describes short natural-language phrases and fragments of source code. The approach combines two lines of work: statistical models of code and bimodal models of image–text pairs. The model takes the form P(C | L). Code is represented as a parse tree that is built step by step. Each choice of child nodes depends on the text query and on the part of the tree already built. The text is encoded as the average of word vectors (bag of words). The tree context is given by the preceding tokens and the types of ancestor nodes. These two representations are combined multiplicatively. Training uses noise contrastive estimation with the AdaGrad optimizer, and a PCFG serves as the noise distribution. The data consist of "short description + code" pairs, in particular StackOverflow question titles together with code from the answers. The model was tested on two retrieval tasks: finding code for a query and selecting a description for code (captioning). For unseen code, the correct description often lands among the top positions of the ranking. Taking code structure into account consistently improves the results. This is one of the early attempts to transfer multimodal methods to the code–language pair, which opens the way to semantic code search and automatic documentation.

Key points

  • Probabilities are obtained by normalizing scores over the empirical set of child tuples for each node type. If a simple frequency-based estimate is used instead of the learned score, the model reduces to an ordinary PCFG.
  • The partial-tree context consists of the 10 most recent tokens and the 10 node types on the path to the root. On synthetic data, the element-wise product of the text and tree vectors (multiplicative) proved better than their sum (additive).
  • The model was evaluated on the perls-captions, perls-all and SO (StackOverflow) datasets. Selecting a description for code turned out to be easier than finding code from a description. Noisier text and code degrade quality only moderately.
  • The difference between the maximum and the mean cross-entropy works as a confidence measure. If answers are given for only 16% of the fragments, the MRR in description retrieval rises from 0.517 to 0.698, i.e., by roughly 35%. The Spearman correlation with MRR is ρ = 0.315 for the multiplicative model versus 0.268 for nl-only.
  • Models that take code structure into account consistently and noticeably outperform the nl-only variant, which ignores it. According to the authors, without structure the correlations within the code have to be explained through the text, which makes the task harder and increases the risk of overfitting.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom396 works · 82,279 citations🏢Microsoft Research (United Kingdom)Cambridge · United Kingdom15 works · 3,537 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksInformation Systems · 1,888 works

🧭 Topics 1Research subjects and languages the work is about.Code & programming languages · 194 works

Cite

Cite as Miltos Allamanis, Daniel Tarlow, Andrew Gordon and Yi Wei. 2015. Bimodal Modelling of Source Code and Natural Language. ICML 2015, PMLR 37:2123–2132.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{allamanis2015bimodal,
  title = {{Bimodal Modelling of Source Code and Natural Language}},
  author = {Miltos Allamanis, Daniel Tarlow, Andrew Gordon and Yi Wei},
  booktitle = {ICML 2015, PMLR 37:2123–2132},
  year = {2015},
  url = {https://proceedings.mlr.press/v37/}
}

References

32 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Allamanis, Miltiadis and Sutton, Charles. Mining source code repositories at massive scale using language modeling. In Working Conference on Mining Software Repositories (MSR), 2013. find
  2. Allamanis, Miltiadis and Sutton, Charles. Mining Idioms from Source Code. In Symposium on the Foundations of Software Engineering (FSE), 2014.
  3. Allamanis, Miltiadis, Barr, Earl T, Bird, Christian, and Sutton, Charles. Learning natural coding conventions. In Symposium on the Foundations of Software Engineering (FSE), 2014. find
  4. Bajracharya, Sushil, Ossher, Joel, and Lopes, Cristina. Sourcerer: An infrastructure for large-scale collection and analysis of open-source code. Science of Computer Programming, 79:241–259, 2014. find
  5. Duchi, John, Hazan, Elad, and Singer, Yoram. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12:2121–2159, 2011. find
  6. Fang, Hao, Gupta, Saurabh, Iandola, Forrest, Srivastava, Rupesh, Deng, Li, Dollár, Piotr, Gao, Jianfeng, He, Xiaodong, Mitchell, Margaret, Platt, John, et al. From captions to visual concepts and back. arXiv preprint arXiv:1411.4952, 2014.
  7. Gulwani, Sumit and Marron, Mark. Nlyze: Interactive programming by natural language for spreadsheet data analysis and manipulation. SIGMOD, 2014. find
  8. Gutmann, Michael U and Hyvärinen, Aapo. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. The Journal of Machine Learning Research, 13(1):307–361, 2012. find
  9. Hindle, Abram, Barr, Earl T, Su, Zhendong, Gabel, Mark, and Devanbu, Premkumar. On the naturalness of software. In International Conference on Software Engineering (ICSE), 2012. find
  10. Hochreiter, Sepp and Schmidhuber, Jürgen. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. find
  11. Karaivanov, Svetoslav, Raychev, Veselin, and Vechev, Martin. Phrase-based statistical translation of programming languages. In Proceedings of the 2014 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming & Software, pp. 173–184. find
  12. Karpathy, Andrej and Fei-Fei, Li. Deep visual-semantic alignments for generating image descriptions. arXiv preprint arXiv:1412.2306, 2014.
  13. Keivanloo, Iman, Rilling, Juergen, and Zou, Ying. Spotting working code examples. In Proceedings of the 36th International Conference on Software Engineering, pp. 664–675. ACM, 2014. find
  14. Kiros, Ryan, Zemel, R, and Salakhutdinov, Ruslan. Multimodal neural language models. In Proc. NIPS Deep Learning Workshop, 2013. find
  15. Kushman, Nate and Barzilay, Regina. Using semantic unification to generate regular expressions from natural language. North American Chapter of the Association for Computational Linguistics (NAACL), 2013. find
  16. Liang, Percy, Jordan, Michael I, and Klein, Dan. Learning dependency-based compositional semantics. Computational Linguistics, 39(2):389–446, 2013. find
  17. Maddison, Chris and Tarlow, Daniel. Structured generative models of natural source code. In Proceedings of The 31st International Conference on Machine Learning, pp. 649–657, 2014. find
  18. Memisevic, Roland and Hinton, Geoffrey. Unsupervised learning of image transformations. In Computer Vision and Pattern Recognition, 2007. CVPR’07. IEEE Conference on, pp. 1–8. find
  19. Mitchell, Jeff and Lapata, Mirella. Composition in distributional models of semantics. Cognitive Science, 34(8): 1388–1439, 2010. find
  20. Mnih, Andriy and Kavukcuoglu, Koray. Learning word embeddings efficiently with noise-contrastive estimation. In Advances in Neural Information Processing Systems, pp. 2265–2273, 2013. find
  21. Mnih, Andriy and Teh, Yee Whye. A fast and simple algorithm for training neural probabilistic language models. In ICML, 2012.
  22. Movshovitz-Attias, Dana, Cohen, WW, and W. Cohen, William. Natural Language Models for Predicting Programming Comments. In Proc of the ACL, 2013. find
  23. MSDN. URL https://MSDN.microsoft.com/en-us/library/bb397906.aspx. .NET Compiler Platform. URL https://github. com/dotnet/roslyn. Nguyen, Tung Thanh, Nguyen, Anh Tuan, Nguyen, Hoan Anh, and Nguyen, Tien N. A statistical semantic language model for source code. In Joint Meeting of the European Softwar… ↗
  24. Raychev, Veselin, Vechev, Martin, and Krause, Andreas. Predicting program properties from big code. In Proceedings of the 42nd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, pp. 111–124. ACM, 2015. find
  25. Socher, Richard, Karpathy, Andrej, Le, Quoc V, Manning, Christopher D, and Ng, Andrew Y. Grounded compositional semantics for finding and describing images with sentences. Transactions of the Association for Computational Linguistics, 2:207–218, 2014.
  26. Srivastava, Nitish and Salakhutdinov, Ruslan R. Multimodal learning with deep boltzmann machines. In Advances in neural information processing systems, pp. 2222–2230, 2012. find
  27. Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc VV. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pp. 3104–3112, 2014.
  28. Taylor, Graham W and Hinton, Geoffrey E. Factored conditional restricted boltzmann machines for modeling motion style. In Proceedings of the 26th annual international conference on machine learning, pp. 1025–1032. ACM, 2009. find
  29. Tu, Zhaopeng, Su, Zhendong, and Devanbu, Premkumar. On the localness of software. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 269–280. ACM, 2014. find
  30. Vinyals, Oriol, Toshev, Alexander, Bengio, Samy, and Erhan, Dumitru. Show and tell: A neural image caption generator. arXiv preprint arXiv:1411.4555, 2014.
  31. Zelle, John M. and Mooney, Raymond J. Learning to parse database queries using inductive logic programming. In Proceedings of the National Conference on Artificial Intelligence (AAAI), 1996.
  32. Zettlemoyer, Luke S and Collins, Michael. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In Proceedings of Uncertainty in Artificial Intelligence (UAI), 2005.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/allamanis-2015-bimodal-modelling-source-code-natural/