Elamur Research⌕
☰
home / papers / 2022 / XLCoST: A Benchmark Dataset for Cross-lingual Code Intelligence

XLCoST: A Benchmark Dataset for Cross-lingual Code Intelligence

Paper · 2022 · Ming Zhu, Aneesh Jain, Karthik Suresh et al.

Summary

Machine learning models for source code have plenty of unlabeled GitHub data to learn from. Tasks like translation, summarization, synthesis and search, however, need aligned examples, and existing parallel corpora are small or cover only one or two languages. XLCoST fills this gap with a benchmark built from GeeksForGeeks solutions. On that site, a single algorithmic problem can have solutions in as many as seven languages (C, C++, C#, Java, Javascript, PHP and Python), and these versions follow the same structure and carry the same comments. The authors split each program at its comments, which produces snippet-level alignment across languages. Each snippet is paired with its English comment, and each program with its problem description. From this data they define 10 tasks: translation, summarization, synthesis, NL code search and cross-lingual code search, each at snippet and program level. They report baselines for every task, including CodeBERT, PLBART, CodeT5 and RoBERTa. CodeT5 is generally the strongest generator. Pretraining on snippets markedly improves program translation for low-resource C. The resource matters because it is the largest and most finely aligned multilingual parallel code dataset described so far.

Key points

  • Scale: 11,265 problems, 57,661 unique programs and 509,091 snippets, averaging 8.81 snippets per program. This yields about 1,002,296 parallel snippet pairs and roughly 123K parallel program pairs.
  • Comparison with other datasets: XLCoST is far larger than CoST (132,046 pairs), AVATAR (57,414 pairs, Java–Python only) and the CodeXGLUE translation set (11,800 pairs, Java–C# only).
  • Task design: 10 tasks split into generation (code-to-code, code-to-text, text-to-code) and retrieval (NL and XL code search). Each task exists at both snippet and program granularity.
  • Snippet transfer for C: Snippet-level transfer substantially boosts program translation involving C. For example, PLBART on C→Java rises from 13.43 to 70.78, and CodeT5 on PHP→C rises from 65.8 to 77.58.
  • Release and limitations: The data and code are available at github.com/reddy-lab-code-research/XLCoST. Generated programs often fail to compile, typically because of unbalanced braces or truncated output.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Virginia TechBlacksburg · United States42 works · 43,568 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksInformation Systems · 1,888 works

🧭 Topics 3Research subjects and languages the work is about.Evaluation & benchmarks · 1,394 worksMultilingual & cross-lingual NLP · 888 worksCode & programming languages · 194 works

Cite

Cite as Ming Zhu, Aneesh Jain, Karthik Suresh, Roshan Ravindran, Sindhu Tipirneni and Chandan K. Reddy. 2022. XLCoST: A Benchmark Dataset for Cross-lingual Code Intelligence. arXiv:2206.08474.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2206.08474 · arXiv 2206.08474

@misc{zhu2022xlcost,
  title = {{XLCoST: A Benchmark Dataset for Cross-lingual Code Intelligence}},
  author = {Ming Zhu, Aneesh Jain, Karthik Suresh, Roshan Ravindran, Sindhu Tipirneni and Chandan K. Reddy},
  year = {2022},
  doi = {10.48550/arxiv.2206.08474},
  eprint = {2206.08474},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2206.08474}
}

References

20 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021a. Unified pre-training for program understanding and generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2655–2668.
  2. Wasi Uddin Ahmad, Md Golam Rahman Tushar, Saikat Chakraborty, and Kai-Wei Chang. 2021b. Avatar: A parallel corpus for java-python program translation. arXiv preprint arXiv:2108.11590. find
  3. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (1).
  5. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1536–1547,…
  6. Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436.
  7. C-JS 15.68 27.1 31.42 73.27 76.42 78.59 C-PHP 31.71 37.61 25.17 60.71 64.28 68.31 C++-C 21.87 31.99 72.61 79.72 85.67 88.96 Java-C 21.27 30.52 63.4 77.3 78.76 82.08 Py-C 18.32 24.07 29.01 47.48 56.44 60.97 C\#-C 21.57 34.16 57.6 74.09 77.38 80.93 JS-C 19.79 29.67 67.71 72.6 74.56 79.58 PHP-C 22.19 28… — BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
  8. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the…
  9. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742. find
  10. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
  11. Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. In Thirty-fifth Conference on Neural Information Processing Systems… find
  12. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
  13. Ruchir Puri, David S Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladmir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, et al. 2021. Project codenet: A large-scale ai for code dataset for learning a diversity of coding tasks. arXiv preprint arXiv:2105.12655.
  14. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. find
  15. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.
  16. Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: a method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009.10297. find
  17. Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. 2020. Unsupervised translation of programming languages. In NeurIPS.
  18. Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8696–8708. find
  19. Ming Zhu, Karthik Suresh, and Chandan K Reddy. 2022. Multilingual code snippets training for program translation. In 36th AAAI Conference on Artificial Intelligence (AAAI). find
  20. Daniel Z "ugner, Tobias Kirschstein, Michele Catasta, Jure Leskovec, and Stephan G "unnemann. 2021. Language-agnostic representation learning of source code from structure and context. In International Conference on Learning Representations (ICLR). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhu-2022-xlcost-benchmark-dataset-cross-lingual/