Elamur Research⌕
☰
home / papers / 2025 / Should Code Models Learn Pedagogically? A Preliminary Evaluation of Curriculum Learning for Real-World Software Engineering Tasks

Should Code Models Learn Pedagogically? A Preliminary Evaluation of Curriculum Learning for Real-World Software Engineering Tasks

Paper · 2025 · Kyi Shin Khant, Hong Yi Lin and Patanamon Thongtanunam

Summary

The authors test whether curriculum learning (CL) helps pre-trained code models on real-world software engineering tasks. CL means training from simple examples to complex ones. Previously, the benefits of CL had been shown mostly on synthetic or artificially complicated code. Here, code complexity is measured with two classic metrics: length (the number of whitespace-separated tokens) and cyclomatic complexity (computed with the Lizard tool). Data from the CodeXGLUE benchmark are split by quartiles into four difficulty levels and balanced in size. CodeT5 is then progressively fine-tuned on the code clone detection and code summarization tasks. The comparison covers a forward schedule (from easy to hardest), a reverse schedule, and conventional training on shuffled data. No CL variant outperformed conventional training. In clone detection there are signs of catastrophic forgetting, while in summarization the model barely responds to the order of the data. The authors suggest that it relies on meaningful identifier names (shortcut learning). In addition, performance plateaus already after the first quarter of the data. The work calls into question the universality of the "pedagogical" approach to training code models.

Key points

  • Model: CodeT5; data: CodeXGLUE (code from GitHub). Complexity metrics: length (L) and cyclomatic complexity (C). For program pairs in clone detection, the complexities of the two programs are summed.
  • Code summarization: conventional training yields BLEU 20.77 (L) and 20.44 (C). The CL variants obtained 20.43, 20.89, 20.28 and 20.24, i.e., the difference is negligible. BERTScore F1 shows the same picture.
  • Clone detection: on the hardest subset, the reverse schedule lags behind conventional training by up to 10.44 F1 points. Examples the model saw at the beginning of training are forgotten by the end.
  • The model plateaus already after 25% of the training data, regardless of whether training started with the easy or the hardest part. Subsequent difficulty levels add almost nothing.
  • All study materials are published on Zenodo (records/14059483). The authors recommend examining other complexity metrics (e.g., clone type), other models and programming languages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of MelbourneMelbourne · Australia104 works · 22,674 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksInformation Systems · 1,888 worksComputer Science Applications · 184 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,394 worksCode & programming languages · 194 works

Cite

Cite as Kyi Shin Khant, Hong Yi Lin and Patanamon Thongtanunam. 2025. Should Code Models Learn Pedagogically? A Preliminary Evaluation of Curriculum Learning for Real-World Software Engineering Tasks. arXiv:2502.03806.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2502.03806 · arXiv 2502.03806

@misc{khant2025should,
  title = {{Should Code Models Learn Pedagogically? A Preliminary Evaluation of Curriculum Learning for Real-World Software Engineering Tasks}},
  author = {Kyi Shin Khant, Hong Yi Lin and Patanamon Thongtanunam},
  year = {2025},
  doi = {10.48550/arxiv.2502.03806},
  eprint = {2502.03806},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2502.03806}
}

References

28 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. Fu and C. Tantithamthavorn, “Linevul: A transformer-based line-level vulnerability prediction,” in MSR. New York, NY, USA: ACM, 2022, pp. 608–620. find
  2. M. Tufano, C. Watson, G. Bavota, M. D. Penta, M. White, and D. Poshyvanyk, “An empirical study on learning bug-fixing patches in the wild via neural 2 https://zenodo.org/records/14059483 machine translation,” TOSEM, vol. 28, no. 4, pp. 19:1– 19:29, 2019. ↗
  3. S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. B. Clement, D. Drain, D. Jiang, D. Tang, G. Li, L. Zhou, L. Shou, L. Zhou, M. Tufano, M. Gong, M. Zhou, N. Duan, N. Sundaresan, S. K. Deng, S. Fu, and S. Liu, “Codexglue: A machine learning benchmark dataset for code understanding and g… find
  4. H. Y. Lin and P. Thongtanunam, “Towards automated code reviews: Does learning code structure help?” in SANER. NJ, USA: IEEE, 2023, pp. 703–707. find
  5. J. Austin, A. Odena, M. I. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. J. Cai, M. Terry, Q. V. Le, and C. Sutton, “Program synthesis with large language models,” CoRR, vol. abs/2108.07732, 2021.
  6. Z. Zeng, H. Tan, H. Zhang, J. Li, Y. Zhang, and L. Zhang, “An extensive study on pre-trained models for program understanding and generation,” in ISSTA. New York, NY, USA: ACM, 2022, pp. 39–51. find
  7. Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” in ICML, vol. 382. New York, NY, USA: ACM, 2009, pp. 41–48.
  8. X. Wang, Y. Chen, and W. Zhu, “A survey on curriculum learning,” TPAMI, vol. 44, no. 9, pp. 4555–4576, 2022. find
  9. M. Naı̈r, K. M. Yamani, L. S. L’Hadj, and R. Baghdadi, “Curriculum learning for small code language models,” in ACL - Student Research Workshop, 2024, pp. 531–542. find
  10. C. Liu, S. Lu, W. Chen, D. Jiang, A. Svyatkovskiy, S. Fu, N. Sundaresan, and N. Duan, “Code execution with pre-trained language models,” in ACL, 2023, pp. 4984–4999.
  11. D. Wang, Z. Jia, S. Li, Y. Yu, Y. Xiong, W. Dong, and X. Liao, “Bridging pre-trained models and downstream tasks for source code understanding,” in ICSE. New York, NY, USA: ACM, 2022, pp. 287–298. find
  12. S. Hao, X. Shi, H. Liu, and Y. Shu, “Enhancing code language models for program repair by curricular fine-tuning framework,” in ICSME. NJ, USA: IEEE, 2023, pp. 136–146. find
  13. Y. Wang, W. Wang, S. R. Joty, and S. C. H. Hoi, “Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” in EMNLP. ACL, 2021, pp. 8696–8708. find
  14. L. Mou, G. Li, L. Zhang, T. Wang, and Z. Jin, “Convolutional neural networks over tree structures for programming language processing,” in AAAI. AAAI Press, 2016, pp. 1287–1293. find
  15. S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer, “Summarizing source code using a neural attention model,” in ACL, 2016. find
  16. A. M. Dakhel, V. Majdinasab, A. Nikanjam, F. Khomh, M. C. Desmarais, and Z. M. J. Jiang, “Github copilot AI pair programmer: Asset or liability?” J. Syst. Softw., vol. 203, p. 111734, 2023. find
  17. A. Graves, M. G. Bellemare, J. Menick, R. Munos, and K. Kavukcuoglu, “Automated curriculum learning for neural networks,” in ICML, vol. 70. PMLR, 2017, pp. 1311–1320. find
  18. E. A. Platanios, O. Stretcu, G. Neubig, B. Póczos, and T. M. Mitchell, “Competence-based curriculum learning for neural machine translation,” in NAACL-HLT. ACL, 2019, pp. 1162–1172. find
  19. T. J. McCabe, “A complexity measure,” TSE., vol. 2, no. 4, pp. 308–320, 1976. find
  20. M. Allamanis, E. T. Barr, P. T. Devanbu, and C. Sutton, “A survey of machine learning for big code and naturalness,” CSUR, vol. 51, no. 4, pp. 81:1–81:37, 2018.
  21. N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,” TACL, vol. 12, pp. 157–173, 2024.
  22. M. J. Shepperd, “A critique of cyclomatic complexity as a software metric,” Softw. Eng. J., vol. 3, no. 2, pp. 30–36, 1988. find
  23. T. L. Alves, C. Ypma, and J. Visser, “Deriving metric thresholds from benchmark data,” in ICSM. NJ, USA: IEEE, 2010, pp. 1–10. find
  24. M. Khajezade, J. J. Wu, F. H. Fard, G. Rodrı́guez-Pérez, and M. S. Shehata, “Investigating the efficacy of large language models for code clone detection,” in ICPC. New York, NY, USA: ACM, 2024, pp. 161–165. find
  25. K. Papineni, S. Roukos, T. Ward, and W. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in ACL, 2002, pp. 311–318.
  26. T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, “Bertscore: Evaluating text generation with BERT,” in ICLR, 2020. find
  27. C. V. Nguyen, A. Achille, M. Lam, T. Hassner, V. Mahadevan, and S. Soatto, “Toward understanding catastrophic forgetting in continual learning,” CoRR, vol. abs/1908.01091, 2019. — [arXiv:1908.01091](https://arxiv.org/abs/1908.01091) find
  28. M. Du, F. He, N. Zou, D. Tao, and X. Hu, “Shortcut learning of large language models in natural language understanding,” Commun., vol. 67, no. 1, pp. 110–120, 2024. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/khant-2025-should-code-models-learn-pedagogically/