Elamur Research⌕
☰
home / papers / 2024 / CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs

CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs

Paper · 2024 · Majeed Kazemitabaar, Runlong Ye, Xiaoning Wang et al.

Summary

Students learning to program need fast, individual help, but office hours scale poorly and are used unevenly. General chatbots such as ChatGPT reply with finished code, which educators worry erodes real understanding and academic integrity. The authors built CodeAid together with the course instructor. It is an LLM-powered helper whose guardrails stop it from handing out ready-made solutions. Instead, it explains concepts, produces interactive pseudo-code with per-line explanations, and annotates buggy student code with suggested fixes. Roughly 700 students used it over a 12-week term in a second-year C and systems programming course. Weekly usage reports to the instructor led to a revision mid-semester. The evidence includes logged interactions with student ratings and qualitative coding of a random sample of responses for correctness, helpfulness and solution leakage. The authors also ran surveys, interviewed students, and later interviewed outside instructors. From this they derive four design considerations (D1–D4) for educational AI assistants, along with their trade-offs. The work is a practical reference for building LLM tools that support learning instead of replacing it.

Key points

  • Deployment and data: CodeAid was deployed to about 700 students for one semester and logged over 8,000 interactions, each rated by students. Additional data came from ten weekly surveys, 22 student interviews, a final anonymous survey comparing CodeAid with ChatGPT, and interviews with eight instructors from six countries.
  • Features: Students could ask general programming questions, ask about their own code, request an explanation of code, get help fixing code, or get help writing code (via pseudo-code rather than real solutions).
  • Models: The tool launched on OpenAI's code-davinci-002 and later moved to gpt-3.5-turbo. The authors suggest the weaker early model may explain some wrong answers and the drop in usage over time.
  • Design considerations: D1 is leveraging what AI uniquely offers. D2 is making questions easy to formulate while keeping students cognitively engaged. D3 is withholding direct answers without killing motivation. D4 is giving students transparency and control to judge and steer responses.
  • Other findings and limits:
    • Women used CodeAid more often than men.
    • Checking indirect answers often required students to write their own test programs.
    • Results come from a single course at one university.
    • The paper reports the size of the thematically analysed sample inconsistently (2,100 vs. 1,749 usages).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States403 works · 162,838 citations🎓University of TorontoToronto · Canada121 works · 179,943 citations🎓University of AucklandAuckland · New Zealand17 works · 1,404 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksInformation Systems · 1,888 worksComputer Science Applications · 184 works

Cite

Cite as Majeed Kazemitabaar, Runlong Ye, Xiaoning Wang, Austin Zachary Henley, Paul Denny, Michelle Craig and Tovi Grossman. 2024. CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs. Proceedings of the CHI Conference on Human Factors in Computing Systems.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1145/3613904.3642773 · arXiv 2401.11314

@inproceedings{kazemitabaar2024codeaid,
  title = {{CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs}},
  author = {Majeed Kazemitabaar, Runlong Ye, Xiaoning Wang, Austin Zachary Henley, Paul Denny, Michelle Craig and Tovi Grossman},
  booktitle = {Proceedings of the CHI Conference on Human Factors in Computing Systems},
  year = {2024},
  doi = {10.1145/3613904.3642773},
  eprint = {2401.11314},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1145/3613904.3642773}
}

References

63 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Joe Michael Allen, Kelly Downey, Kris Miller, Alex Daniel Edgcomb, and Frank Vahid. 2019. Many Small Programs in CS1: Usage Analysis from Multiple Universities. In 2019 ASEE Annual Conference & Exposition ". ASEE Conferences, Tampa, Florida, 1–13. https://peer.asee.org/33084. ↗
  2. Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N Bennett, Kori Inkpen, et al. 2019. Guidelines for human-AI interaction. In Proceedings of the 2019 chi conference on human factors in computing systems. 1–13. find
  3. Brett A Becker, Paul Denny, James Finnie-Ansley, Andrew Luxton-Reilly, James Prather, and Eddie Antonio Santos. 2023. Programming Is Hard-Or at Least It Used to Be: Educational Opportunities and Challenges of AI Code Generation. In Proceedings of the 54th ACM Technical Symposium on Computer Science… find
  4. Andrea J Bingham and Patricia Witkowsky. 2021. Deductive and inductive approaches to qualitative data analysis. Analyzing and interpreting qualitative data: After the interview (2021), 133–146. find
  5. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora De… — On the Opportunities and Risks of Foundation Models
  6. Joel Brandt, Philip J Guo, Joel Lewenstein, Mira Dontcheva, and Scott R Klemmer. 2009. Two studies of opportunistic programming: interleaving web foraging, learning, and writing code. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1589–1598. find
  7. Carla E. Brodley, Benjamin J. Hescott, Jessica Biron, Ali Ressing, Melissa Peiken, Sarah Maravetz, and Alan Mislove. 2022. Broadening Participation in Computing via Ubiquitous Combined Majors (CS+X). In Proceedings of the 53rd ACM Technical Symposium on Computer Science Education - Volume 1 (Provide… — [doi:10.1145/3478431.3499352](https://doi.org/10.1145/3478431.3499352) find
  8. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language Models are Few-shot Learners. Advances in neural information processing systems 33 (2020), 1877–1901.
  9. Peter Brusilovsky, Barbara J Ericson, Cay S Horstmann, and Christian Servin. 2023. The Future of Computing Education Materials. (2023). find
  10. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021).
  11. Jonathan E Collins. 2023. Policy Solutions: Policy questions for ChatGPT and artificial intelligence. Phi Delta Kappan 104, 7 (2023), 60–61. find
  12. Paul Denny, Viraj Kumar, and Nasser Giacaman. 2023. Conversing with copilot: Exploring prompt engineering for solving cs1 problems using natural language. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1. 1136–1142. find
  13. Paul Denny, Andrew Luxton-Reilly, Ewan Tempero, and Jacob Hendrickx. 2011. CodeWrite: Supporting Student-Driven Practice of Java. In Proceedings of the 42nd ACM Technical Symposium on Computer Science Education (Dallas, TX, USA) (SIGCSE ’11). Association for Computing Machinery, New York, NY, USA, 4… — [doi:10.1145/1953163.1953299](https://doi.org/10.1145/1953163.1953299) find
  14. Paul Denny, James Prather, Brett A Becker, James Finnie-Ansley, Arto Hellas, Juho Leinonen, Andrew Luxton-Reilly, Brent N Reeves, Eddie Antonio Santos, and Sami Sarsa. 2023. Computing Education in the Era of Generative AI. arXiv preprint arXiv:2306.02608 (2023). find
  15. Augie Doebling and Ayaan M Kazerouni. 2021. Patterns of academic help-seeking in undergraduate computing students. In Proceedings of the 21st Koli Calling International Conference on Computing Education Research. 1–10. find
  16. James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. 2022. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Proceedings of the 24th Australasian Computing Education Conference (Virtual Event, Australia) (ACE ’22).… — [doi:10.1145/3511861.3511863](https://doi.org/10.1145/3511861.3511863) find
  17. James Finnie-Ansley, Paul Denny, Andrew Luxton-Reilly, Eddie Antonio Santos, James Prather, and Brett A Becker. 2023. My AI Wants to Know if This Will Be on the Exam: Testing OpenAI’s Codex on CS2 Programming Exercises. In Proceedings of the 25th Australasian Computing Education Conference. 97–104. find
  18. Zhikai Gao, Sarah Heckman, and Collin Lynch. 2022. Who Uses Office Hours? A Comparison of In-Person and Virtual Office Hours Utilization. In Proceedings of the 53rd ACM Technical Symposium on Computer Science Education - Volume 1 (Providence, RI, USA) (SIGCSE 2022). Association for Computing Machine… — [doi:10.1145/3478431.3499334](https://doi.org/10.1145/3478431.3499334) find
  19. Philip J Guo. 2013. Online python tutor: embeddable web-based program visualization for cs education. In Proceeding of the 44th ACM technical symposium on Computer science education. 579–584. find
  20. Mark Guzdial. 2023. Scaffolding to Support Humanities Students Programming in a Human Language Context. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 2 (Turku, Finland) (ITiCSE 2023). Association for Computing Machinery, New York, NY, USA, 579–58… — [doi:10.1145/3587103.3594157](https://doi.org/10.1145/3587103.3594157) find
  21. Björn Hartmann, Daniel MacDougall, Joel Brandt, and Scott R Klemmer. 2010. What would other programmers do: suggesting solutions to error messages. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1019–1028. find
  22. Andrew Head, Elena Glassman, Gustavo Soares, Ryo Suzuki, Lucas Figueredo, Loris D’Antoni, and Björn Hartmann. 2017. Writing reusable code feedback at scale with mixed-initiative program synthesis. In Proceedings of the Fourth (2017) ACM Conference on Learning@ Scale. 89–98. find
  23. Hotjar. 2023. Hotjar: Website Heatmaps & Behavior Analytics Tools. https://www.hotjar.com/. Accessed on: 30-July-2023. ↗
  24. Jeremy Hsu. 2023. Should schools ban AI chatbots? New Scientist 257, 3422 (2023), 15. https://doi.org/10.1016/S0262-4079(23)00099-4 — [doi:10.1016/s0262-4079(23)00099-4](https://doi.org/10.1016/s0262-4079%2823%2900099-4) ↗
  25. Michelle Ichinco and Caitlin Kelleher. 2015. Exploring novice programmer example use. In 2015 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 63–71. find
  26. Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. 2023. ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Difference…
  27. Majeed Kazemitabaar, Justin Chow, Carl Ka To Ma, Barbara J Ericson, David Weintrop, and Tovi Grossman. 2023. Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–23. find
  28. Majeed Kazemitabaar, Xinying Hou, Austin Henley, Barbara Ericson, David Weintrop, and Tovi Grossman. 2023. How Novices Use LLM-based Code Generators to Solve CS1 Coding Tasks in a Self-Paced Learning Environment. In Proceedings of the 23rd Koli Calling International Conference on Computing Education… find
  29. Harsh Kumar, Ilya Musabirov, Joseph Jay Williams, and Michael Liut. 2023. QuickTA: Exploring the Design Space of Using Large Language Models to Provide Support to Students. Learning Analytics and Knowledge Conference 2023 (LAK’23). find
  30. Sam Lau and Philip J Guo. 2023. From" Ban It Till We Understand It" to" Resistance is Futile": How University Programming Instructors Plan to Adapt as More Students Use AI Code Generation and Explanation Tools such as ChatGPT and GitHub Copilot. In Proceedings of the 2023 ACM Conference on Internati… find
  31. Michael J. Lee. 2014. Gidget: An online debugging game for learning and engagement in computing education. In 2014 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). 193–194. https://doi.org/10.1109/VLHCC. 2014.6883051 — [doi:10.1109/vlhcc](https://doi.org/10.1109/vlhcc) ↗
  32. Michael J Lee and Amy J Ko. 2015. Comparing the effectiveness of online learning approaches on CS1 learning outcomes. In Proceedings of the eleventh annual international conference on international computing education research. 237–246. find
  33. Juho Leinonen, Paul Denny, Stephen MacNeil, Sami Sarsa, Seth Bernstein, Joanne Kim, Andrew Tran, and Arto Hellas. 2023. Comparing Code Explanations Created by Students and Large Language Models. arXiv:2304.03938 [cs.CY] find
  34. Juho Leinonen, Arto Hellas, Sami Sarsa, Brent Reeves, Paul Denny, James Prather, and Brett A Becker. 2023. Using large language models to enhance programming error messages. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1. 563–569.
  35. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Proceedings of the 34th Internation…
  36. Chen Li, Emily Chan, Paul Denny, Andrew Luxton-Reilly, and Ewan Tempero. 2019. Towards a Framework for Teaching Debugging. In Proceedings of the Twenty-First Australasian Computing Education Conference (Sydney, NSW, Australia) (ACE ’19). Association for Computing Machinery, New York, NY, USA, 79–86.… — [doi:10.1145/3286960.3286970](https://doi.org/10.1145/3286960.3286970) find
  37. Mark Liffiton, Brad Sheese, Jaromir Savelka, and Paul Denny. 2023. CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes. arXiv:2308.06921 [cs.CY] — [arXiv:2308.06921](https://arxiv.org/abs/2308.06921) find
  38. Dastyni Loksa and Amy J Ko. 2016. The role of self-regulation in programming problem solving process and success. In Proceedings of the 2016 ACM conference on international computing education research. 83–91. find
  39. Stephen MacNeil, Andrew Tran, Arto Hellas, Joanne Kim, Sami Sarsa, Paul Denny, Seth Bernstein, and Juho Leinonen. 2023. Experiences from using code explanations generated by large language models in a web software development e-book. In Proceedings of the 54th ACM Technical Symposium on Computer Sci… find
  40. Lauren E Margulieux, Mark Guzdial, and Richard Catrambone. 2012. Subgoal-labeled instructional material improves performance and transfer in learning to develop mobile applications. In Proceedings of the ninth annual international conference on International computing education research. 71–78. find
  41. Renee McCauley, Sue Fitzgerald, Gary Lewandowski, Laurie Murphy, Beth Simon, Lynda Thomas, and Carol Zander. 2008. Debugging: a review of the literature from an educational perspective. Computer Science Education 18, 2 (2008), 67–92. https://doi.org/10.1080/08993400802114581 arXiv:https://doi.org/10… — [doi:10.1080/08993400802114581](https://doi.org/10.1080/08993400802114581) ↗
  42. Tilman Michaeli and Ralf Romeike. 2019. Improving Debugging Skills in the Classroom: The Effects of Teaching a Systematic Debugging Process. In Proceedings of the 14th Workshop in Primary and Secondary Computing Education (Glasgow, Scotland, Uk) (WiPSCE’19). Association for Computing Machinery, New… — [doi:10.1145/3361721.3361724](https://doi.org/10.1145/3361721.3361724) find
  43. Matthew B Miles and A Michael Huberman. 1994. Qualitative data analysis: An expanded sourcebook. sage. find
  44. Diba Mirza, Phillip T Conrad, Christian Lloyd, Ziad Matni, and Arthur Gatin. 2019. Undergraduate teaching assistants in computer science: a systematic literature review. In Proceedings of the 2019 ACM Conference on International Computing Education Research. 31–40. find
  45. Laurie Murphy, Sue Fitzgerald, Raymond Lister, and Renée McCauley. 2012. Ability to ’explain in Plain English’ Linked to Proficiency in Computer-Based Programming. In Proceedings of the Ninth Annual International Conference on International Computing Education Research (Auckland, New Zealand) (ICER… — [doi:10.1145/2361276.2361299](https://doi.org/10.1145/2361276.2361299) find
  46. Engineering National Academies of Sciences, Medicine, et al. 2018. Assessing and responding to the growth of computer science undergraduate enrollments. National Academies Press. find
  47. Kimberly A Neuendorf. 2017. The content analysis guidebook. sage. find
  48. OpenAI. 2022. Introducing ChatGPT. https://openai.com/blog/chatgpt. Accessed on: 30-July-2023. ↗
  49. Tung Phung, José Cambronero, Sumit Gulwani, Tobias Kohn, Rupak Majumdar, Adish Singla, and Gustavo Soares. 2023. Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models. arXiv preprint arXiv:2302.04662 (2023). find
  50. James Prather, Brett A. Becker, Michelle Craig, Paul Denny, Dastyni Loksa, and Lauren Margulieux. 2020. What Do We Think We Think We Are Doing? Metacognition and Self-Regulation in Programming. In Proceedings of the 2020 ACM Conference on International Computing Education Research (Virtual Event, Ne… — [doi:10.1145/3372782.3406263](https://doi.org/10.1145/3372782.3406263) find
  51. James Prather, Brent N. Reeves, Paul Denny, Brett A. Becker, Juho Leinonen, Andrew Luxton-Reilly, Garrett Powell, James Finnie-Ansley, and Eddie Antonio Santos. 2023. “It’s Weird That It Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers. ACM Trans. Comput.-Hum. Inter… — [doi:10.1145/3617367](https://doi.org/10.1145/3617367) find
  52. Brent Reeves, Sami Sarsa, James Prather, Paul Denny, Brett A. Becker, Arto Hellas, Bailey Kimmel, Garrett Powell, and Juho Leinonen. 2023. Evaluating the Performance of Code Generation Models for Solving Parsons Problems With Small Prompt Variations. In Proceedings of the 2023 Conference on Innovati… — [doi:10.1145/3587102.3588805](https://doi.org/10.1145/3587102.3588805) find
  53. Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. 2022. Automatic generation of programming exercises and code explanations using large language models. In Proceedings of the 2022 ACM Conference on International Computing Education Research-Volume 1. 27–43. find
  54. Jaromir Savelka, Arav Agarwal, Marshall An, Chris Bogart, and Majd Sakr. 2023. Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses. In Proceedings of the 2023 ACM Conference on International Computing Education Resea… find
  55. Aaron J. Smith, Kristy Elizabeth Boyer, Jeffrey Forbes, Sarah Heckman, and Ketan Mayer-Patel. 2017. My Digital Hand: A Tool for Scaling Up One-to-One Peer Teaching in Support of Computer Science Learning. In Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education (Seattl… — [doi:10.1145/3017680.3017800](https://doi.org/10.1145/3017680.3017800) find
  56. Margaret Smith, Yujie Chen, Rachel Berndtson, Kristen M Burson, and Whitney Griffin. 2017. "Office Hours Are Kind of Weird": Reclaiming a Resource to Foster Student-Faculty Interaction. InSight: A Journal of Scholarly Teaching 12 (2017), 14–29. find
  57. Lasang Jimba Tamang, Zeyad Alshaikh, Nisrine Ait Khayi, Priti Oli, and Vasile Rus. 2021. A comparative study of free self-explanations and socratic tutoring explanations for source code comprehension. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education. 219–225. find
  58. Priyan Vaithilingam, Tianyi Zhang, and Elena L Glassman. 2022. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In Chi conference on human factors in computing systems extended abstracts. 1–7. find
  59. Arto Vihavainen, Craig S Miller, and Amber Settle. 2015. Benefits of self-explanation in introductory programming. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education. 284–289. find
  60. Lev Semenovich Vygotsky and Michael Cole. 1978. Mind in society: Development of higher psychological processes. Harvard university press. find
  61. Qiaosi Wang, Koustuv Saha, Eric Gregori, David Joyner, and Ashok Goel. 2021. Towards mutual theory of mind in human-ai interaction: How language reflects what students perceive about a virtual teaching assistant. In Proceedings of the 2021 CHI conference on human factors in computing systems. 1–14. find
  62. Jacqueline L. Whalley, Raymond Lister, Errol Thompson, Tony Clear, Phil Robbins, P. K. Ajith Kumar, and Christine Prasad. 2006. An Australasian Study of Reading and Comprehension Skills in Novice Programmers, Using the Bloom and SOLO Taxonomies. In Proc. of the 8th Australasian Conf. on Computing Ed… find
  63. Frank F Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. 2022. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming. 1–10.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kazemitabaar-2024-codeaid-evaluating-classroom-deployment-llm/