Elamur Research⌕
☰
home / papers / 2023 / Measuring The Impact Of Programming Language Distribution

Measuring The Impact Of Programming Language Distribution

Paper · 2023 · Gabriel Orlanski, Kefan Xiao, Xavier Garcia et al.

Summary

Code LLMs are usually trained on GitHub data, where each language's share reflects how much code exists rather than how many developers use it. As a result, popular but data-poor languages such as Go and Rust get weak support, and most benchmarks ignore them. The authors address both evaluation and training. For evaluation, they built BabelCode, an open-source framework that ports a benchmark to many languages. It describes test-case types in a small DSL, renders test scripts from templates, executes them, and reports per-test outcomes, memory use and runtime. With BabelCode they created TP3, a translation benchmark built from the hand-written verification functions of Python Programming Puzzles. For training, they used Unimax to cap how often any language's data is repeated. They then pretrained 1B, 2B and 4B decoder-only models, using UL2 and causal LM objectives, on natural versus balanced mixtures of 14 languages. Balancing clearly improves low-resource languages at a moderate cost to high-resource ones, and that cost shrinks as models grow. This gives a practical recipe for more equitable multilingual code models.

Key points

  • BabelCode supports four benchmarks in 14 languages (github.com/google-research/babelcode). Unlike MBXP, it is open-sourced and does not require Python source benchmarks. Unlike MultiPL-E, it covers translation as well as generation and includes an extensive test suite. It also handles type-aware equality, with tolerances of 1e-6 for floats and 1e-9 for doubles.
  • TP3 (Translating Python Programming Puzzles) is derived from P3 (Schuster et al., 2021). Its expert-written source functions are harder to translate than those in TransCoder-style benchmarks.
  • Effect of balanced (Unimax) training:
    • Average pass@k rises by 12.34% overall.
    • Low-resource languages gain 66.48%, while high-resource languages lose 12.94%.
    • On translation tasks, low-resource languages gain 30.77% and high-resource languages lose 19.58%.
  • Effect of model size: Low-resource gains do not grow with model size. However, the high-resource deficit relative to the baseline narrows to 2.47% at 4B.
  • Share of test cases passed per problem (low-resource languages): 5.13% for Natural, 9.53% for Unimax 1 and 10.48% for Unimax 2. This suggests that generation benefits mainly from unique, non-repeated data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksInformation Systems · 1,888 worksComputer Science Applications · 184 works

🧭 Topics 1Research subjects and languages the work is about.Code & programming languages · 194 works

Cite

Cite as Gabriel Orlanski, Kefan Xiao, Xavier Garcia, Jeffrey Hui, Joshua Howland, Jonathan Malmaud, Jacob Austin, Rishabh Singh and Michele Catasta. 2023. Measuring The Impact Of Programming Language Distribution. arXiv:2302.01973.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2302.01973 · arXiv 2302.01973

@misc{orlanski2023measuring,
  title = {{Measuring The Impact Of Programming Language Distribution}},
  author = {Gabriel Orlanski, Kefan Xiao, Xavier Garcia, Jeffrey Hui, Joshua Howland, Jonathan Malmaud, Jacob Austin, Rishabh Singh and Michele Catasta},
  year = {2023},
  doi = {10.48550/arxiv.2302.01973},
  eprint = {2302.01973},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2302.01973}
}

References

36 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmad, W., Chakraborty, S., Ray, B., and Chang, K.-W. Unified pre-training for program understanding and generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2655–2668, Online, June 2021. As…
  2. Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021.
  3. Bavarian, M., Jun, H., Tezak, N., Schulman, J., McLeavey, C., Tworek, J., and Chen, M. Efficient training of language models to fill in the middle. arXiv preprint arXiv:2207.14255, 2022.
  4. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. Unsupervised cross-lingual representation learning at scale. In Annual Meeting of the Association for Computational Linguistics, 2019.
  5. Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M. H., Zi, Y., Anderson, C. J., Feldman, M. Q., et al. A scalable and extensible approach to benchmarking nl2code for 18 programming languages. arXiv preprint arXiv:2208.08227, 2022.
  6. Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., and Zhou, M. CodeBERT: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1536–1547, Online, November 2020. Associat…
  7. Chakraborty, S., Ahmed, T., Ding, Y., Devanbu, P., and Ray, B. Natgen: generative pre-training by “naturalizing” source code. Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2022. find
  8. Chen, M., Tworek, J., Jun, H., Yuan, Q., Ponde, H., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., S… — Evaluating Large Language Models Trained on Code
  9. Fried, D., Aghajanyan, A., Lin, J., Wang, S. I., Wallace, E., Shi, F., Zhong, R., tau Yih, W., Zettlemoyer, L., and Lewis, M. Incoder: A generative model for code infilling and synthesis. ArXiv, abs/2204.05999, 2022. find
  10. Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., and Steinhardt, J. Measuring coding challenge competence with APPS. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. U…
  11. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
  12. Husain, H., Wu, H., Gazit, T., Allamanis, M., and Brockschmidt, M. Codesearchnet challenge: Evaluating the state of semantic code search. ArXiv, abs/1909.09436, 2019.
  13. Christopoulou, F., Lampouras, G., Gritta, M., Zhang, G., Guo, Y., Li, Z.-Y., Zhang, Q., Xiao, M., Shen, B., Li, L., Yu, H., yu Yan, L., Zhou, P., Wang, X., Ma, Y., Iacobacci, I., Wang, Y., Liang, G., Wei, J., Jiang, X., Wang, Q., and Liu, Q. Pangu-coder: Program synthesis with function-level languag… — PanGu-Coder: Program Synthesis with Function-Level Language Modeling
  14. Kocetkov, D., Li, R., Allal, L. B., Li, J., Mou, C., Ferrandis, C. M., Jernite, Y., Mitchell, M., Hughes, S., Wolf, T., et al. The stack: 3 tb of permissively licensed source code. arXiv preprint arXiv:2211.15533, 2022. find
  15. Kudo, T. and Richardson, J. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226, 2018.
  16. Chung, H. W., Garcia, X., Roberts, A., Tay, Y., Firat, O., Narang, S., and Constant, N. Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=kXwdL… ↗
  17. Clement, C., Drain, D., Timcheck, J., Svyatkovskiy, A., and Sundaresan, N. PyMT5: multi-mode translation of natural language and python code with transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9052– 9065, Online, November 2020. As…
  18. Lai, Y., Li, C., Wang, Y., Zhang, T., Zhong, R., Zettlemoyer, L., Yih, S., Fried, D., yi Wang, S., and Yu, T. Ds-1000: A natural and reliable benchmark for data science code generation. ArXiv, abs/2211.11501, 2022. find
  19. Li, Y., Choi, D. H., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Tom, Eccles, Keeling, J., Gimeno, F., Lago, A. D., Hubert, T., Choy, P., de, C., d’Autume, M., Babuschkin, I., Chen, X., Huang, P.-S., Welbl, J., Gowal, S., Alexey, Cherepanov, Molloy, J., Mankowitz, D. J., Robson, E. S., K… — Competition-Level Code Generation with AlphaCode
  20. Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C. B., Drain, D., Jiang, D., Tang, D., Li, G., Zhou, L., Shou, L., Zhou, L., Tufano, M., Gong, M., Zhou, M., Duan, N., Sundaresan, N., Deng, S. K., Fu, S., and Liu, S. Codexglue: A machine learning benchmark dataset for code… find
  21. Shazeer, N. and Stern, M. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp. 4596–4604. PMLR, 2018. find
  22. Tay, Y., Dehghani, M., Tran, V. Q., Garcia, X., Bahri, D., Schuster, T., Zheng, H. S., Houlsby, N., and Metzler, D. Unifying language learning paradigms. arXiv preprint arXiv:2205.05131, 2022.
  23. Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., and Xiong, C. A conversational paradigm for program synthesis. arXiv preprint arXiv:2203.13474, 2022. find
  24. Wang, S., Li, Z., Qian, H., Yang, C., Wang, Z., Shang, M., Kumar, V., Tan, S., Ray, B., Bhatia, P., Nallapati, R., Ramanathan, M. K., Roth, D., and Xiang, B. Recode: Robustness evaluation of code generation models. 2022a. find
  25. Orlanski, G. and Gittens, A. Reading stackoverflow encourages cheating: Adding question text improves extractive code generation. ArXiv, abs/2106.04447, 2021. find
  26. Wang, X., Tsvetkov, Y., and Neubig, G. Balancing training for multilingual neural machine translation. arXiv preprint arXiv:2004.06748, 2020. find
  27. Orlanski, G., Yang, S., and Healy, M. Evaluating how fine-tuning on bimodal data effects code generation. ArXiv, abs/2211.07842, 2022. — [arXiv:2211.07842](https://arxiv.org/abs/2211.07842) find
  28. Wang, Y., Wang, W., Joty, S., and Hoi, S. C. CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 8696–8708, Online and Punta Cana, Dominican Republic,… find
  29. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P. J., et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1–67, 2020.
  30. Roberts, A., Chung, H. W., Levskaya, A., Mishra, G., Bradbury, J., Andor, D., Narang, S., Lester, B., Gaffney, C., Mohiuddin, A., Hawthorne, C., Lewkowycz, A., Salcianu, A., van Zee, M., Austin, J., Goodman, S., Soares, L. B., Hu, H., Tsvyashchenko, S., Chowdhery, A., Bastings, J., Bulian, J., Garci… — Scaling Up Models and Data with $\texttt{t5x}$ and $\texttt{seqio}$
  31. Wang, Z., Cuenca, G., Zhou, S., Xu, F. F., and Neubig, G. Mconala: A benchmark for code generation from multiple natural languages. ArXiv, abs/2203.08388, 2022b. find
  32. Yasunaga, M. and Liang, P. Break-it-fix-it: Unsupervised learning for program repair. In International Conference on Machine Learning (ICML), 2021. find
  33. Yin, P., Deng, B., Chen, E., Vasilescu, B., and Neubig, G. Learning to mine aligned code and natural language pairs from stack overflow. 2018 IEEE/ACM 15th International Conference on Mining Software Repositories (MSR), pp. 476–486, 2018.
  34. Roziere, B., Lachaux, M.-A., Chanussot, L., and Lample, G. Unsupervised translation of programming languages. Advances in Neural Information Processing Systems, 33, 2020.
  35. Rozière, B., Lachaux, M.-A., Szafraniec, M., and Lample, G. Dobf: A deobfuscation pre-training objective for programming languages. In Neural Information Processing Systems, 2021. find
  36. Schuster, T., Kalyan, A., Polozov, A., and Kalai, A. T. Programming puzzles. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021. URL https://openreview.net/forum?id=fe_hCc4RBrg. *Figure 8. Sample problem translated from Python to C++ using BabelC… ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/orlanski-2023-measuring-impact-programming-language-distribution/