Elamur Research
☰
home / papers / 2018 / Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow

Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1805.08949

Summary

Data-driven tools for code retrieval, summarization and synthesis from natural language need large parallel collections in which NL statements and code are closely aligned. Stack Overflow holds such data, but earlier extraction heuristics, such as taking every code block or only blocks from accepted answers, either add noise or discard much usable material. Yin et al. frame the task as classification. For each question title, which serves as the intent, every line-contiguous fragment of code in the answers is treated as a candidate. A classifier then judges each candidate using two kinds of features. Hand-crafted structural features, which are mostly language-independent, estimate whether the fragment is a sensible piece of code. Learned correspondence features come from a neural translation-style model that scores code given NL and NL given code. On small annotated Python and Java sets, the approach recovers many more pairs, and more accurate ones, than the baselines. It also performs reasonably when trained on one language and applied to the other. This matters because it makes NL-code corpus building practical for languages that lack annotations.

Key points

  • Each mined item is a triple: an intent (English description), optional context (setup code such as imports or variable definitions) and a snippet that actually implements the intent.
  • The labeled data covers only 152 Python and 102 Java question threads. These were drawn from the March 2017 SO dump by combining the top-100 most-viewed questions with a view-weighted sample, and only the top-3 answers per question were annotated.
  • Compared with prior heuristics, the method extracts up to roughly ten times more aligned pairs at equal accuracy, or more than halves the error rate when the number of pairs is held fixed.
  • About 36% of Python-tagged questions are "how to" requests, which is the category the work targets.
  • The annotations, annotation interface and mining code are released at conala-corpus.github.io (the CoNaLa project). The paper appeared at MSR '18.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Computing & GPUs

Cited by

100 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 97 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1805.08949 · arXiv 1805.08949

@misc{yin2018learning,
  title = {{Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow}},
  author = {Yin, Pengcheng and Deng, Bowen and Chen, Edgar and Vasilescu, Bogdan and Neubig, Graham},
  year = {2018},
  doi = {10.48550/arxiv.1805.08949},
  eprint = {1805.08949},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1805.08949}
}

References

48 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Miltiadis Allamanis, Earl T Barr, Christian Bird, and Charles Sutton. 2015. Suggesting Accurate Method and Class Names. In Joint Meeting on Foundations of Software Engineering (ESEC/FSE). ACM, 38–49. find
  2. Miltiadis Allamanis, Hao Peng, and Charles Sutton. 2016. A Convolutional Attention Network for Extreme Summarization of Source Code. arXiv preprint arXiv:1602.03001 (2016). find
  3. Miltiadis Allamanis, Daniel Tarlow, Andrew D Gordon, and Yi Wei. 2015. Bimodal Modelling of Source Code and Natural Language. In International Conference on Machine Learning (ICML). 2123–2132. find
  4. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In International Conference on Learning Representations (ICLR). find our card
  5. Antonio Valerio Miceli Barone and Rico Sennrich. 2017. A Parallel Corpus of Python Functions and Documentation Strings for Automated Code Documentation and Code Generation. arXiv preprint arXiv:1707.02275 (2017). find
  6. Shaunak Chatterjee, Sudeep Juvekar, and Koushik Sen. 2009. Sniff: A Search Engine for Java Using Free-form Queries. In International Conference on Fundamental Approaches to Software Engineering. Springer, 385–400. find
  7. Aditya Desai, Sumit Gulwani, Vineet Hingorani, Nidhi Jain, Amey Karkare, Mark Marron, Subhajit Roy, and others. 2016. Program Synthesis using Natural Language. In International Conference on Software Engineering (ICSE). ACM, 345– 356. find
  8. Premkumar Devanbu. 2015. New Initiative: the Naturalness of Software. In International Conference on Software Engineering (ICSE), Vol. 2. IEEE, 543–546. find
  9. Christine Franks, Zhaopeng Tu, Premkumar Devanbu, and Vincent Hellendoorn. 2015. CACHECA: A Cache Language Model based Code Suggestion Tool. In International Conference on Software Engineering (ICSE), Vol. 2. IEEE, 705–708. find
  10. Mark Gabel and Zhendong Su. 2010. A Study of the Uniqueness of Source Code. In International Symposium on Foundations of Software Engineering (FSE). ACM, 147–156. find
  11. Yarin Gal and Zoubin Ghahramani. 2016. A Theoretically Grounded Application of Dropout in Recurrent Neural Networks. In Annual Conference on Neural Information Processing Systems (NIPS). 1019–1027. find our card
  12. Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. Incorporating Copying Mechanism in Sequence-to-Sequence Learning. In Annual Meeting of the Association for Computational Linguistics (ACL). ACL, 1631–1640. find our card
  13. Abram Hindle, Earl T Barr, Mark Gabel, Zhendong Su, and Premkumar Devanbu. 2016. On the naturalness of software. Commun. ACM 59, 5 (2016), 122–131. find
  14. Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning Deep Structured Semantic Models for Web Search using Clickthrough Data. In International Conference on Information and Knowledge Management (CIKM). ACM, 2333–2338. find
  15. Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016. Summarizing Source Code using a Neural Attention Model. In Annual Meeting of the Association for Computational Linguistics (ACL). ACL, 2073–2083. find
  16. Alan Jaffe, Jeremy Lacomis, Edward J. Schwartz, Claire Le Goues, and Bogdan Vasilescu. 2018. Meaningful Variable Names for Decompiled Code: A Machine Translation Approach. In International Conference on Program Comprehension (ICPC). ACM. find
  17. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent Continuous Translation Models. In Conference on Empirical Methods in Natural Language Processing (EMNLP). ACL, 1700–1709. find
  18. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. CoRR abs/1412.6980 (2014). http://arxiv.org/abs/1412.6980 ↗
  19. Philipp Koehn. 2010. Statistical Machine Translation. Cambridge Press. find
  20. Xi Victoria Lin, Chenglong Wang, Luke Zettlemoyer, and Michael D Ernst. 2018. NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating System. International Conference on Language Resources and Evaluation (LREC) (2018). find
  21. Nicholas Locascio, Karthik Narasimhan, Eduardo De Leon, Nate Kushman, and Regina Barzilay. 2016. Neural Generation of Regular Expressions from Natural Language with Minimal Domain Knowledge. In Conference on Empirical Methods in Natural Language Processing (EMNLP). ACL, 1918–1923. find
  22. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective Approaches to Attention-based Neural Machine Translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP). ACL, 1412–1421. find
  23. Lance A Miller. 1981. Natural Language Programming: Styles, Strategies, and Contrasts. IBM Systems Journal 20, 2 (1981), 184–215. find
  24. Dana Movshovitz-Attias and William W Cohen. 2013. Natural Language Models for Predicting Programming Comments. In Annual Meeting of the Association for Computational Linguistics (ACL). ACL, 35–40. find
  25. Graham Neubig. 2017. Neural Machine Translation and Sequence-to-Sequence Models: A Tutorial. arXiv preprint arXiv:1703.01619 (2017). — [arXiv:1703.01619](https://arxiv.org/abs/1703.01619) find
  26. Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, Kevin Duh, Manaal Faruqui, Cynthia Gan, Dan Garrette, Yangfeng Ji, Lingpeng Kong, Adhiguna Kuncoro, Gaurav Kumar, Chaitanya Malaviya, Pau… find our card
  27. Anh Tuan Nguyen, Hoan Anh Nguyen, Tung Thanh Nguyen, and Tien N Nguyen. 2014. Statistical Learning Approach for Mining API Usage Mappings for Code Migration. In International Conference on Automated Software Engineering (ASE). ACM, 457–468. find
  28. Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N Nguyen. 2013. Lexical Statistical Machine Translation for Language Migration. In Joint Meeting on Foundations of Software Engineering (ESEC/FSE). ACM, 651–654. find
  29. Haoran Niu, Iman Keivanloo, and Ying Zou. 2016. Learning to Rank Code Examples for Code Search Engines. Empirical Software Engineering (2016), 1–33. find
  30. Yusuke Oda, Hiroyuki Fudaba, Graham Neubig, Hideaki Hata, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. 2015. Learning to Generate Pseudo-Code from Source Code Using Statistical Machine Translation. In International Conference on Automated Software Engineering (ASE). IEEE, 574–584. find
  31. Sebastiano Panichella, Jairo Aponte, Massimiliano Di Penta, Andrian Marcus, and Gerardo Canfora. 2012. Mining Source Code Descriptions from Developer Communications. In International Conference on Program Comprehension (ICPC). IEEE, 63–72. find
  32. Chris Quirk, Raymond Mooney, and Michel Galley. 2015. Language to Code: Learning Semantic Parsers for If-This-Then-That Recipes. In Annual Meeting of the Association for Computational Linguistics (ACL). 878–888. find
  33. Maxim Rabinovich, Mitchell Stern, and Dan Klein. 2017. Abstract Syntax Networks for Code Generation and Semantic Parsing. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vancouver, Canada,… find
  34. Veselin Raychev, Martin Vechev, and Andreas Krause. 2015. Predicting Program Properties from “Big Code”. In ACM Symposium on Principles of Programming Languages (POPL). ACM, 111–124. find
  35. Victor S Sheng, Foster Provost, and Panagiotis G Ipeirotis. 2008. Get Another Label? Improving Data Quality and Data Mining using Multiple, Noisy Labelers. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 614–622. find
  36. Richard Socher, Eric H Huang, Jeffrey Pennin, Christopher D Manning, and Andrew Y Ng. 2011. Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection. In Advances in Neural Information Processing Systems (NIPS). 801–809. find
  37. Giriprasad Sridhara, Lori Pollock, and K Vijay-Shanker. 2011. Generating Parameter Comments and Integrating with Method Summaries. In International Conference on Program Comprehension (ICPC). IEEE, 71–80. find
  38. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research 15, 1 (2014), 1929–1958. find
  39. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014. Sequence to Sequence Learning with Neural Networks. In Advances in Neural Information Processing Systems (NIPS). 3104–3112. find
  40. Bogdan Vasilescu, Casey Casalnuovo, and Premkumar Devanbu. 2017. Recovering Clear, Natural Identifiers from Obfuscated JavaScript Names. In Joint Meeting on the Foundations of Software Engineering (ESEC/FSE). ACM. to appear. find
  41. Yushi Wang, Jonathan Berant, and Percy Liang. 2015. Building a Semantic Parser Overnight. In Annual Meeting of the Association for Computational Linguistics (ACL). ACL, 1332–1342. find
  42. Yi Wei, Nirupama Chandrasekaran, Sumit Gulwani, and Youssef Hamadi. 2015. Building Bing Developer Assistant. Technical Report. MSR-TR-2015-36, Microsoft Research. find
  43. Edmund Wong, Taiyue Liu, and Lin Tan. 2015. CloCom: Mining Existing Source Code for Automatic Comment Generation. In International Conference on Software Analysis, Evolution, and Reengineering (SANER). IEEE, 380–389. find
  44. Edmund Wong, Jinqiu Yang, and Lin Tan. 2013. AutoComment: Mining question and answer sites for automatic comment generation. In International Conference on Automated Software Engineering (ASE). IEEE, 562–567. find
  45. Di Yang, Aftab Hussain, and Cristina Videira Lopes. 2016. From query to usable code: an analysis of Stack Overflow code snippets. In Working Conference on Mining Software Repositories (MSR). ACM, 391–402. find
  46. Ziyu Yao, Daniel S. Weld, Wei-Peng Chen, and Huan Sun. 2018. StaQC: A Systematically Mined Question-Code Dataset from Stack Overflow. In WWW 2018: The 2018 Web Conference. find
  47. Pengcheng Yin and Graham Neubig. 2017. A Syntactic Neural Model for General-Purpose Code Generation. In Meeting of the Association for Computational Linguistics (ACL). find our card
  48. Alexey Zagalsky, Ohad Barzilay, and Amiram Yehudai. 2012. Example overflow: Using Social Media for Code Recommendation. In International Workshop on Recommendation Systems for Software Engineering (RSSE). IEEE Press, 38–42. find