Elamur Research⌕
☰
home / papers / 2021 / CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks

CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks

Paper · 2021 · Ruchir Puri, David S. Kung, Geert Janssen et al.

Summary

AI for Code research has lacked a large, well-curated benchmark comparable to ImageNet. IBM Research addresses this with CodeNet, a dataset built from submissions to two online judge platforms, AIZU and AtCoder. It holds almost 14 million programs, about 500 million lines, in 55 programming languages, grouped under 4053 problems. Each submission has metadata: an acceptance or error status, CPU time, memory use, code size, and the language. HTML problem statements are included, and sample test inputs and outputs cover 98.5% of samples, so program correctness can be checked by running the code. The authors also supply preprocessing tools that convert source into tokens and other model-ready forms. Baseline experiments cover code classification, code similarity and masked token prediction. When the MISIM model is trained on the C++1000 subset, it transfers better to an unseen dataset than when trained on GCJ-297. The resource matters because it supports many tasks: clone detection, code search, bug repair, performance prediction and translation between languages.

Key points

  • Scale: 13,916,868 submissions across 4053 problems. 53.6% (7,460,588) are accepted, 29.5% are wrong answers, and the rest failed time or memory limits.
  • Languages: 95% of the code is C++, Python, Java, C, Ruby or C#. C++ is the largest, with 8,008,527 submissions (57%).
  • Layout: the dataset is organized as data/(problem_id)/(language)/(submission). It includes CSV metadata at the dataset and problem levels, plus a derived folder with token sequences and near-duplicate information.
  • Generalization test: MISIM was trained on C++1000 and on GCJ-297, then evaluated with MAP@R on POJ-104. The C++1000 model scored about 12% higher, which the authors attribute to deduplication and lower problem bias.
  • Masked language modelling: a BERT-like model on C++1000 tokens reached top-1 accuracy of 0.9104 and top-5 accuracy of 0.9935.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksInformation Systems · 1,730 worksComputer Science Applications · 131 works

Cite

Cite as Ruchir Puri, David S. Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, Ulrich Finkler, Susan Malaika and Frederick Reiss. 2021. CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks. arXiv:2105.12655.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2105.12655 · arXiv 2105.12655

@misc{puri2021codenet,
  title = {{CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks}},
  author = {Ruchir Puri, David S. Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, Ulrich Finkler, Susan Malaika and Frederick Reiss},
  year = {2021},
  doi = {10.48550/arxiv.2105.12655},
  eprint = {2105.12655},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2105.12655}
}

References

53 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Miltiadis Allamanis, Earl T Barr, Premkumar Devanbu, and Charles Sutton. A survey of machine learning for big code and naturalness. ACM Computing Surveys (CSUR), 51(4):1–37, 2018.
  2. Yanming Yang, Xin Xia, David Lo, and John Grundy. A survey on deep learning for software engineering. arXiv preprint arXiv:2011.14597, 2020. — [arXiv:2011.14597](https://arxiv.org/abs/2011.14597) find
  3. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — Evaluating Large Language Models Trained on Code
  4. Marie-Anne Lachaux, Baptiste Roziere, Lowik Chanussot, and Guillaume Lample. Unsupervised translation of programming languages. In NeurIPS, 2020.
  5. Zheng Wang and Michael O’Boyle. Machine learning in compiler optimization. Proceedings of the IEEE, 106(11):1879–1901, 2018. find
  6. http://ibm.biz/cfcsc-codenet. ↗
  7. Women in data science. https://widsconference.org/. ↗
  8. Yutaka Watanobe. Aizu online judge. https://onlinejudge.u-aizu.ac.jp. ↗
  9. Atcoder. https://atcoder.jp/. ↗
  10. Yunhui Zheng, Saurabh Pujar, Burn Lewis, Luca Buratti, Edward Epstein, Bo Yang, Jim Laredo, Alessandro Morari, and Zhong Su. D2a: A dataset built for ai-based vulnerability detection methods using differential analysis. In Proceedings of the ACM/IEEE 43rd International Conference on Software Enginee… find
  11. Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. Devign: Effective vulnerability identification by learning comprehensive programsemantics via graph neural networks. In Advances in Neural Information Processing Systems, pages 10197–10207. NeurIPS Foundation, 2019. find
  12. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gonga, Linjun Shou, Bing Qin, Ting Liu, and Daxin Jiang. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155v4, 2020.
  13. Miltiadis Allamanis and Charles Sutton. Mining source code repositories at massive scale using language modeling. In 10th Working Conference on Mining Software Repositories (MSR), page 207–216. IEEE, 2013. find
  14. Veselin Raychev, Pavol Bielik, and Martin Vechev. Probabilistic model for code with decision trees. ACM SIGPLAN Notices, 2016. find
  15. Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys Poshyvanyk. An empirical study on learning bug-fixing patches in the wild via neural machine translation. In ACM Transactions on Software Engineering and Methodology (TOSEM), pages 1–29, 2019. find
  16. Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436v3, 2019.
  17. Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. Mapping language to code in programmatic context. arXiv preprint arXiv:1808.09588, 2018.
  18. Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. Codexglue: A ma… find
  19. Tal Ben-Nun, Alice Shoshana Jakobovits, and Torsten Hoefler. Neural code comprehension: A learnable representation of code semantics. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 3588–3600.… find
  20. Farhan Ullah, Hamad Naeem, Sohail Jabbar, Shehzad Khalid, Muhammad Ahsan Latif, Fadi Al-turjman, and Leonardo Mostarda. Cyber security threats detection in internet of things using deep learning approach. IEEE Access, 7:124379–124389, 2019. find
  21. Fangke Ye, Shengtian Zhou, Anand Venkat, Ryan Marcus, Nesime Tatbul, Jesmin Jahan Tithi, Niranjan Hasabnis, Paul Petersen, Mattson. Timothy, Tim Kraska, Pradeep Dubey, Vivek Sarkar, and Justin Gottschlich. Misim: A novel code similarity system, 2021. find
  22. https://sites.google.com/site/treebasedcnn/home/problemdescription. ↗
  23. gcj-dataset. https://openreview.net/attachment?id=AZ4vmLoJft&name=supplementary_material. ↗
  24. Miltiadis Allamanis. The adverse effects of code duplication in machine learning models of code. In Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software, Onward! 2019, page 143–153, New York, NY, USA, 2019. Association f…
  25. Wikipedia. Jaccard index — Wikipedia, the free encyclopedia. https://en.wikipedia.org/wiki/Jaccard_index, 2020. ↗
  26. Terence Parr. The Definitive ANTLR 4 Reference. Pragmatic Bookshelf, 2nd edition, 2013. find
  27. Sifei Luan, Di Yang, Celeste Barnaby, Koushik Sen, and Satish Chandra. Aroma: code recommendation via structural code search. Proceedings of the ACM on Programming Languages, 3(OOPSLA):1–28, Oct 2019. find
  28. IBM T.J. Watson Research Center. Wala. https://github.com/wala/WALA, 2021. ↗
  29. Forbes on codenet. https://www.forbes.com/sites/moorinsights/2021/06/04/ibm-codenet-artificial-intelligence-that-can-program-computers-and-solve-a-100-billion-legacy-code-problem/?sh=343813636cdc. https://venturebeat.com/2021/05/10/ibms-codenet- ↗
  30. Zdnet on codenet. https://www.zdnet.com/article/ibm-launches-autosql-watson-orchestrate-codenet-enterprise-ai-tools-at-think/. ↗
  31. Project codenet repository. https://github.com/IBM/Project_CodeNet. ↗
  32. Luca Buratti, Saurabh Pujar, Mihaela Bornea, Scott McCarley, Yunhui Zheng, Gaetano Rossiello, Alessandro Morari, Jim Laredo, Veronika Thost, Yufan Zhuang, and Giacomo Domeniconi. Exploring software naturalness through neural language models, 2020. find
  33. Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
  34. Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In ICLR, 2019.
  35. Veronika Thost and Jie Chen. Directed acyclic graph neural networks. In ICLR, 2021. find
  36. Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. Codexglue: A… find
  37. Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning, 2018. find
  38. Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.00687, 2020. find
  39. Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
  40. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and S… — PyTorch: An Imperative Style, High-Performance Deep Learning Library
  41. Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks?, 2019.
  42. Hitesh Sajnani. Large-Scale Code Clone Detection. PhD thesis, University of California, Irvine, 2016. find
  43. Yujia Li, Chenjie Gu, Thomas Dullien, Oriol Vinyals, and Pushmeet Kohli. Graph matching network for learning the similarity of graph structured objects. In International Conference on Machine Learning (ICML), 2019. find
  44. Graph-matching-networks. https://github.com/Lin-Yijie/Graph-Matching-Networks. ↗
  45. Kevin Musgrave, Serge J. Belongie, and Ser-Nam Lim. A metric learning reality check. CoRR, abs/2003.08505, 2020. — [arXiv:2003.08505](https://arxiv.org/abs/2003.08505) find
  46. Codenet dataset. https://developer.ibm.com/exchanges/data/all/project-codenet. ↗
  47. Ankur Singh. "end-to-end masked language modeling with bert". https://keras.io/examples/nlp/masked_language_modeling. ↗
  48. Zimin Chen, Steve Kommrusch, Michele Tufano, Louis-Noël Pouchet, Denys Poshyvanyk, and Martin Monperrus. Sequencer: Sequence-to-sequence learning for end-to-end program repair. IEEE Transaction on Software Engineering, 2019. find
  49. Michihiro Yasunaga and Percy Liang. Break-it-fix-it: Unsupervised learning for program repair. In International Conference on Machine Learning (ICML), 2021. find
  50. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… find
  51. Xinyun Chen, Chang Liu, and Dawn Song. Tree-to-tree neural networks for program translation. In NeurIPS, 2018. find
  52. Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. In ICLR, 2018.
  53. Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. CodeXGLUE: A ma… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/puri-2021-codenet-large-scale-ai-code/