Elamur Research⌕
☰
home / papers / 2023 / Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors

Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors

Paper · 2023 · Tung Phung, Victor-Alexandru Pădurean, José Cambronero et al.

Summary

Large language models are often proposed as tools for teaching introductory programming. Earlier evaluations, however, either used models that are no longer available, such as Codex, or examined only one use case. This study offers a broader benchmark. It compares ChatGPT (GPT-3.5), GPT-4 and human tutors across six roles an AI helper might take: program repair, hint generation, grading feedback, pair programming, contextualized explanation and task synthesis.

The test material consists of five short Python exercises from geeksforgeeks.org. Each exercise is paired with five real incorrect student submissions, chosen to cover different bug types. Two experienced tutors rated every output using scenario-specific quality criteria, and the same two experts also supplied the human baseline.

GPT-4 was far stronger than ChatGPT and came near tutor-level quality in several scenarios. It still lagged clearly behind humans on grading feedback and on creating new debugging exercises. The work matters because it shows where current LLMs are ready to support programming instruction and where better techniques are still needed.

Key points

  • Data: The benchmark has 5 problems (GCD, FIBONACCI, DIVISORSDIV3, PALINDROME, MERGESTRS) and 25 real buggy programs from geeksforgeeks.org, ranging from 4 to 31 lines. Program correctness was checked with the platform's test suites.
  • Protocol:
    • The LLMs received one query per instance through their web interfaces.
    • The two human experts each produced their own answers independently.
    • Each expert evaluated the outputs of the models and of the other expert.
  • Task synthesis results: Overall success was 10.0% for ChatGPT, 22.0% for GPT-4 and 74.0% for Tutor. The widest gap was on reproducing similar bugs (SimilarBugs: 36% / 40% / 92%).
  • Weakest scenarios for GPT-4: grading feedback and task synthesis. Among the problems, task synthesis was hardest on DIVISORSDIV3 and PALINDROME.
  • Limitations named by the authors: only two experts, Python and English only, and no student-based assessment. Proposed future work includes larger benchmarks, open-source models, and fine-tuning or symbolic methods.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States403 works · 162,838 citations🔬Max Planck Institute for Software SystemsSaarbrücken · Germany6 works · 39,162 citations🎓TU WienVienna · Austria10 works · 943 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksInformation Systems · 1,888 worksComputer Science Applications · 184 works

Cite

Cite as Tung Phung, Victor-Alexandru Pădurean, José Cambronero, Sumit Gulwani, Tobias Kohn, Rupak Majumdar, Adish Singla and Gustavo Soares. 2023. Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors. Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 2.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1145/3568812.3603476 · arXiv 2306.17156

@inproceedings{phung2023generative,
  title = {{Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors}},
  author = {Tung Phung, Victor-Alexandru Pădurean, José Cambronero, Sumit Gulwani, Tobias Kohn, Rupak Majumdar, Adish Singla and Gustavo Soares},
  booktitle = {Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 2},
  year = {2023},
  doi = {10.1145/3568812.3603476},
  eprint = {2306.17156},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1145/3568812.3603476}
}

References

30 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tung Phung, Victor-Alexandru Pădurean, José Cambronero, Sumit Gulwani, Tobias Kohn, Rupak Majumdar, Adish Singla, and Gustavo Soares. Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors. In ICER V.2, 2023.
  2. OpenAI. ChatGPT. https://openai.com/blog/chatgpt, 2023. ↗
  3. OpenAI. GPT-4 Technical Report. CoRR, abs/2303.08774, 2023.
  4. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Túlio Ribeiro, and Yi Zhang. Sparks of Artificial General Intelligence: Early Experiments with GPT-4. CoRR, abs/2303…
  5. David Baidoo-Anu and Leticia Owusu Ansah. Education in the Era of Generative Artificial Intelligence (AI): Understanding the Potential Benefits of ChatGPT in Promoting Teaching and Learning. Available at SSRN 4337484, 2023. find
  6. Weng Marc Lim, Asanka Gunasekara, Jessica Leigh Pallant, Jason Ian Pallant, and Ekaterina Pechenkina. Generative AI and the Future of Education: Ragnarök or Reformation? A Paradoxical Perspective from Management Educators. The International Journal of Management Education, 21(2):100790, 2023. find
  7. James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In ACE, 2022. find
  8. Jialu Zhang, José Cambronero, Sumit Gulwani, Vu Le, Ruzica Piskac, Gustavo Soares, and Gust Verbruggen. Repairing Bugs in Python Assignments Using Large Language Models. CoRR, abs/2209.14876, 2022. — [arXiv:2209.14876](https://arxiv.org/abs/2209.14876) find
  9. Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models. In ICER, 2022. find
  10. Juho Leinonen, Arto Hellas, Sami Sarsa, Brent N. Reeves, Paul Denny, James Prather, and Brett A. Becker. Using Large Language Models to Enhance Programming Error Messages. In SIGCSE, 2023.
  11. Tung Phung, José Cambronero, Sumit Gulwani, Tobias Kohn, Rupak Majumdar, Adish Singla, and Gustavo Soares. Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models. In EDM, 2023. find
  12. Mark Chen et al. Evaluating Large Language Models Trained on Code. CoRR, abs/2107.03374, 2021.
  13. geeksforgeeks.org. GeeksforGeeks: A Computer Science Portal for Geeks. https://www.geeksforgeeks.org/, 2009. ↗
  14. Jooyong Yi, Umair Z. Ahmed, Amey Karkare, Shin Hwei Tan, and Abhik Roychoudhury. A Feasibility Study of Using Automated Program Repair for Introductory Programming Assignments. In ESEC/FSE, 2017. find
  15. Jia Tracy Shen, Michiharu Yamashita, Ethan Prihar, Neil T. Heffernan, Xintao Wu, and Dongwon Lee. MathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics Education. CoRR, abs/2106.07340, 2021. — [arXiv:2106.07340](https://arxiv.org/abs/2106.07340) find
  16. Hiroaki Funayama, Tasuku Sato, Yuichiroh Matsubayashi, Tomoya Mizumoto, Jun Suzuki, and Kentaro Inui. Balancing Cost and Quality: An Exploration of Human-in-the-Loop Frameworks for Automated Short Answer Scoring. In AIED, 2022. find
  17. GitHub. GitHub Copilot: Your AI Pair Programmer. https://github.com/features/copilot, 2022. ↗
  18. Hussein Mozannar, Gagan Bansal, Adam Fourney, and Eric Horvitz. Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming. CoRR, abs/2210.14306, 2022. — [arXiv:2210.14306](https://arxiv.org/abs/2210.14306) find
  19. Saki Imai. Is Github Copilot a Substitute for Human Pair-programming? An Empirical Study. In ICSE Companion Proceedings, 2022. find
  20. Qianou Ma, Tongshuang Wu, and Kenneth R. Koedinger. Is AI the Better Programming Partner? Human-Human Pair Programming vs. Human-AI pAIr Programming. CoRR, abs/2306.05153, 2023. — [arXiv:2306.05153](https://arxiv.org/abs/2306.05153) find
  21. Hannah Potter, Ardi Madadi, René Just, and Cyrus Omar. Contextualized Programming Language Documentation. In SIGPLAN Onward!, 2022. find
  22. Umair Z. Ahmed, Maria Christakis, Aleksandr Efremov, Nigel Fernandez, Ahana Ghosh, Abhik Roychoudhury, and Adish Singla. Synthesizing Tasks for Block-based Programming. In NeurIPS, 2020. find
  23. Ahana Ghosh, Sebastian Tschiatschek, Sam Devlin, and Adish Singla. Adaptive Scaffolding in Block-Based Programming via Synthesizing New Tasks as Pop Quizzes. In AIED, 2022. find
  24. Victor-Alexandru Pădurean, Georgios Tzannetos, and Adish Singla. Neural Task Synthesis for Visual Programming. CoRR, abs/2305.18342, 2023. — [arXiv:2305.18342](https://arxiv.org/abs/2305.18342) find
  25. Mikhail Mirzayanov. Codeforces. https://codeforces.com/. ↗
  26. Y Daniel Liang. Introduction to Programming using Python. 2013. find
  27. Ana Bell, Eric Grimson, and John Guttag. Introduction to Computer Science and Programming in Python. https://ocw.mit.edu/courses/6-0001-introduction-to-computer-science-and-programming-in-python-fall-2016/pages/lecture-slides-code/ , 2016. ↗
  28. OpenAI. ChatGPT model=text-davinci-002. https://chat.openai.com/?model=text-davinci-002-render-sha, 2023. ↗
  29. OpenAI. GPT-4 model=gpt-4. https://chat.openai.com/?model=gpt-4, 2023. ↗
  30. Georg Brandl, Matthäus Chajdas, and Jean Abou-Samra. Pygments. https://pygments.org/. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/phung-2023-generative-ai-programming-education-benchmarking/