Elamur Research
☰
home / papers / 2020 / CodeSearchNet Challenge: Evaluating the State of Semantic Code Search

CodeSearchNet Challenge: Evaluating the State of Semantic Code Search

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1909.09436

Summary

Searching for code with natural-language queries is hard because queries and source code share very little vocabulary. The field also had no large, purpose-built benchmark for measuring progress. To address this, the authors from GitHub and Microsoft Research released the CodeSearchNet Corpus, built from permissively licensed open-source GitHub projects in six languages and parsed with TreeSitter. Each function's documentation was cut down to its first paragraph and filtered with heuristics so that it could stand in as a rough substitute for a search query. Alongside the corpus they released the CodeSearchNet Challenge, an evaluation set of real-world queries with relevance scores assigned by experts. They trained baseline neural encoders that map code and text into a shared embedding space (NBoW, 1D-CNN, biRNN, self-attention) and compared them with ElasticSearch. Self-attention did best on the documentation-based proxy task. On the expert-labelled queries, however, the simple NBoW model came out on top and keyword-based ElasticSearch was competitive. This suggests that matching rare terms matters a lot and that docstrings do not train models well for real search. The resource became a standard benchmark for code retrieval and for code–language pretraining.

Key points

  • The CodeSearchNet Corpus contains about 6.4M functions in Go, Java, JavaScript, PHP, Python and Ruby. About 2.3M of them come paired with documentation, and the data is split 80/10/10 into train, validation and test sets.
  • Filtering removed docstrings shorter than 3 tokens, functions shorter than 3 lines, test functions, constructors and standard methods such as __str__/toString, plus near-duplicate functions.
  • The Challenge has 99 queries, taken from Bing searches with high click-through to code and from StaQC rewrites. It includes 4,026 expert annotations on a 0–3 scale, and inter-annotator agreement was moderate (Cohen κ = 0.47).
  • On the proxy task, each query is ranked against 999 distractors. The SelfAtt encoder scored the highest average MRR (0.7011), and the biRNN scored the lowest (0.4262).
  • The paper lists open challenges: representing rare identifiers, using control and data flow, applying BERT-style pretraining, and treating code quality as a ranking signal.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Computing & GPUs

Cited by

221 papers of our library cite it (of the 140673 whose bibliographies we read); 15 of them have a page here, 206 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1909.09436 · arXiv 1909.09436

@misc{husain2020codesearchnet,
  title = {{CodeSearchNet Challenge: Evaluating the State of Semantic Code Search}},
  author = {Husain, Hamel and Wu, Ho-Hsiang and Gazit, Tiferet and Allamanis, Miltiadis and Brockschmidt, Marc},
  year = {2020},
  doi = {10.48550/arxiv.1909.09436},
  eprint = {1909.09436},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1909.09436}
}

References

25 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Miltiadis Allamanis. 2018. The Adverse Effects of Code Duplication in Machine Learning Models of Code. arXiv preprint arXiv:1812.06469 (2018). find
  2. Miltiadis Allamanis, Earl T Barr, Premkumar Devanbu, and Charles Sutton. 2018. A survey of machine learning for big code and naturalness. ACM Computing Surveys (CSUR) 51, 4 (2018), 81. find
  3. Miltiadis Allamanis, Hao Peng, and Charles Sutton. 2016. A Convolutional Attention Network for Extreme Summarization of Source Code. In Proceedings of the International Conference on Machine Learning (ICML). find
  4. Uri Alon, Omer Levy, and Eran Yahav. 2018. code2seq: Generating sequences from structured representations of code. arXiv preprint arXiv:1808.01400 (2018). find
  5. Antonio Valerio Miceli Barone and Rico Sennrich. 2017. A parallel corpus of Python functions and documentation strings for automated code documentation and code generation. arXiv preprint arXiv:1707.02275 (2017). find
  6. Jose Cambronero, Hongyu Li, Seohyun Kim, Koushik Sen, and Satish Chandra. 2019. When Deep Learning Met Code Search. arXiv preprint arXiv:1905.03813 (2019). — [arXiv:1905.03813](https://arxiv.org/abs/1905.03813) find
  7. Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the Properties of Neural Machine Translation: Encoder–Decoder Approaches. Syntax, Semantics and Structure in Statistical Translation (2014). find our card
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv Encoder CodeSearchNet Challenge– NDCG Within Text Code Go Java JS PHP Python ElasticSearch NBoW NBoW 1D-CNN 1D-CNN biRNN biRNN SelfAtt SelfA… find
  9. Patrick Fernandes, Miltiadis Allamanis, and Marc Brockschmidt. 2018. Structured Neural Summarization. arXiv preprint arXiv:1811.01824 (2018). find
  10. Philip Gage. 1994. A new algorithm for data compression. The C Users Journal 12, 2 (1994), 23–38. find
  11. Xiaodong Gu, Hongyu Zhang, and Sunghun Kim. 2018. Deep code search. In 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE). IEEE, 933–944. find
  12. Xiaodong Gu, Hongyu Zhang, Dongmei Zhang, and Sunghun Kim. 2016. Deep API Learning. In Proceedings of the International Symposium on Foundations of Software Engineering (FSE). find
  13. Tatsunori B Hashimoto, Kelvin Guu, Yonatan Oren, and Percy S Liang. 2018. A retrieve-and-edit framework for predicting structured outputs. In Advances in Neural Information Processing Systems. 10073–10083. find
  14. Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018. Mapping language to code in programmatic context. arXiv preprint arXiv:1808.09588 (2018). find
  15. Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882 (2014). find our card
  16. Xi Victoria Lin, Chenglong Wang, Luke Zettlemoyer, and Michael D. Ernst. 2018. NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating System. In International Conference on Language Resources and Evaluation. find
  17. Wang Ling, Edward Grefenstette, Karl Moritz Hermann, Tomas Kocisky, Andrew Senior, Fumin Wang, and Phil Blunsom. 2016. Latent Predictor Networks for CodeSearchNet Challenge– NDCG All Ruby Avg Go Java JS PHP Python Ruby Avg 0.395 0.680 0.420 0.185 0.515 0.650 0.337 0.574 0.404 0.145 0.493 0.566 0.186… find
  18. Cristina V Lopes, Petr Maj, Pedro Martins, Vaibhav Saini, Di Yang, Jakub Zitny, Hitesh Sajnani, and Jan Vitek. 2017. DéjàVu: a map of code duplicates on GitHub. Proceedings of the ACM on Programming Languages 1, OOPSLA (2017), 84. find
  19. Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to Information Retrieval. Cambridge University Press. find
  20. Bhaskar Mitra, Nick Craswell, et al. 2018. An introduction to neural information retrieval. Foundations and Trends® in Information Retrieval 13, 1 (2018), 1–126. find
  21. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). find
  22. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems. 5998–6008. find
  23. Ziyu Yao, Jayavardhan Reddy Peddamail, and Huan Sun. 2019. CoaCor: Code Annotation for Code Retrieval with Reinforcement Learning. (2019). find
  24. Ziyu Yao, Daniel S Weld, Wei-Peng Chen, and Huan Sun. 2018. StaQC: A Systematically Mined Question-Code Dataset from Stack Overflow. In Proceedings of the 2018 World Wide Web Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 1693–1703. find
  25. Pengcheng Yin and Graham Neubig. 2017. A Syntactic Neural Model for General-Purpose Code Generation. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). find our card