Elamur Research⌕
☰
home / papers / 2023 / Can Language Models Employ the Socratic Method? Experiments with Code Debugging

Can Language Models Employ the Socratic Method? Experiments with Code Debugging

Paper · 2023 · Erfan Al-Hossami, Razvan Bunescu, Justin Smith et al.

Summary

The paper investigates whether large language models can act as Socratic tutors for beginner programmers. Instead of supplying a fix, such a tutor asks questions that lead students to locate and repair a bug themselves. Progress on such agents has been limited by a lack of suitable data, so the authors built a manually written benchmark of multi-turn Socratic debugging dialogues in Python. Each exercise pairs a simple problem with test cases and buggy code containing exactly one mistake. The model playing the instructor also receives a bug description and the possible fixes. At every turn, contributors recorded a main tutor reply and several alternatives, so that generated replies can be scored for both precision and recall. The authors benchmarked several models: fine-tuned Flan-T5, plus zero-shot and chain-of-thought prompting of GPT-3.5 and GPT-4. GPT-4 clearly beat GPT-3.5, but it still did not reach the level of human experts. The work provides a reusable resource and evaluation protocol for automated tutoring in introductory CS (CS1).

Key points

  • The benchmark contains 151 main conversations. Problems are novice-level, such as Factorial and Fibonacci. The authors wrote 25 of them, and more were drawn from the Auckland (8), Refactory (2) and FalconCode (3) datasets. The bugs reflect common beginner errors, including off-by-one and boundary mistakes, operator misuse, and misunderstood constructs.
  • Automatic scoring compares generated and reference utterances using BLEU, BERT F1 (based on DeBERTa) and Rouge-L. A one-to-one maximum matching, computed with Edmonds' Blossom algorithm, prevents near-duplicate outputs from inflating the scores. Precision is computed as TP/m and recall as TP/n.
  • In the manual error analysis, GPT-3.5 produced 53 irrelevant utterances compared with 8 for GPT-4. Other error categories were repeated, overly direct and premature replies.
  • For GPT-4 with chain-of-thought prompting, automatic metrics did not agree with human judgments, which shows that manual evaluation remains necessary.
  • Code and data are available at github.com/taisazero/socratic-debugging-benchmark. The paper was published at SIGCSE '24.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York UniversityNew York · United States201 works · 100,180 citations🎓University of North Carolina at CharlotteCharlotte · United States6 works · 125 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksInformation Systems · 1,437 worksComputer Science Applications · 102 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Erfan Al-Hossami, Razvan Bunescu, Justin Smith and Ryan Teehan. 2023. Can Language Models Employ the Socratic Method? Experiments with Code Debugging. arXiv:2310.03210.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.03210 · arXiv 2310.03210

@misc{alhossami2023can,
  title = {{Can Language Models Employ the Socratic Method? Experiments with Code Debugging}},
  author = {Erfan Al-Hossami, Razvan Bunescu, Justin Smith and Ryan Teehan},
  year = {2023},
  doi = {10.48550/arxiv.2310.03210},
  eprint = {2310.03210},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.03210}
}

References

41 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Erfan Al-Hossami, Razvan Bunescu, Ryan Teehan, Laurel Powell, Khyati Mahajan, and Mohsen Dorodchi. 2023. Socratic Questioning of Novice Debuggers: A Benchmark Dataset and Preliminary Evaluations. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA… find
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901.
  3. Tracy Camp, W Richards Adrion, Betsy Bizot, Susan Davidson, Mary Hall, Susanne Hambrusch, Ellen Walker, and Stuart Zweben. 2017. Generation CS: the growth of computer science. ACM Inroads 8, 2 (2017), 44–50. find
  4. Lin Chen, Barbara Di Eugenio, Davide Fossati, Stellan Ohlsson, and David Cosejo. 2011. Exploring Effective Dialogue Act Sequences in One-on-one Computer Science Tutoring Dialogues. In Proceedings of the Sixth Workshop on Innovative Use of NLP for Building Educational Applications. Association for Co…
  5. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 (2022). find
  6. Adrian de Freitas, Joel Coffman, Michelle de Freitas, Justin Wilson, and Troy Weingart. 2023. FalconCode: A Multiyear Dataset of Python Code Samples from an Introductory Computer Science Course. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (SIGCSE 2023). Asso… — [doi:10.1145/3545945.3569822](https://doi.org/10.1145/3545945.3569822) find
  7. Andrew Ettles, Andrew Luxton-Reilly, and Paul Denny. 2018. Common logic errors made by novice programmers. In Proceedings of the 20th Australasian Computing Education Conference (ACE ’18). Association for Computing Machinery, New York, NY, USA, 83–89. https://doi.org/10.1145/3160489.3160493 — [doi:10.1145/3160489.3160493](https://doi.org/10.1145/3160489.3160493) ↗
  8. Zvi Galil. 1986. Efficient algorithms for finding maximum matching in graphs. ACM Computing Surveys (CSUR) 18, 1 (1986), 23–38. find
  9. Prakhar Gupta, Shikib Mehri, Tiancheng Zhao, Amy Pavel, Maxine Eskenazi, and Jeffrey P Bigham. 2019. Investigating Evaluation of Open-Domain Dialogue Systems With Human Generated Multiple References. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue. 379–391. find
  10. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654 (2020).
  11. Yang Hu, Umair Z. Ahmed, Sergey Mechtaev, Ben Leong, and Abhik Roychoudhury. 2019. Re-factoring based Program Repair applied to Programming Assignments. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE/ACM, 388–398. find
  12. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
  13. Timotej Lazar, Martin Možina, and Ivan Bratko. 2017. Automatic extraction of AST patterns for debugging student programs. In Artificial Intelligence in Education: 18th International Conference, AIED 2017, Wuhan, China, June 28–July 1, 2017, Proceedings 18. Springer, 162–174. find
  14. Victor CS Lee, Yuen-Tak Yu, Chung Man Tang, Tak-Lam Wong, and Chung Keung Poon. 2018. ViDA: A virtual debugging advisor for supporting learning in computer programming courses. Journal of Computer Assisted Learning 34, 3 (2018), 243–258. find
  15. Juho Leinonen, Arto Hellas, Sami Sarsa, Brent Reeves, Paul Denny, James Prather, and Brett A. Becker. 2023. Using Large Language Models to Enhance Programming Error Messages. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 (Toronto ON, Canada) (SIGCSE 2023). Ass… find
  16. Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out. Association for Computational Linguistics, Barcelona, Spain, 74–81. https://aclanthology.org/W04-1013 ↗
  17. Stephen MacNeil, Andrew Tran, Arto Hellas, Joanne Kim, Sami Sarsa, Paul Denny, Seth Bernstein, and Juho Leinonen. 2023. Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book. In Proceedings of the 54th ACM Technical Symposium on Computer Sci… — [doi:10.1145/3545945.3569785](https://doi.org/10.1145/3545945.3569785) find
  18. Mary Lou Maher, Celine Latulipe, Heather Lipford, and Audrey Rorrer. 2015. Flipped classroom strategies for CS Education. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education. 218–223. find
  19. Jessica McBroom, Irena Koprinska, and Kalina Yacef. 2021. A survey of automated programming hint generation: The hints framework. ACM Computing Surveys (CSUR) 54, 8 (2021), 1–27. find
  20. Renée McCauley, Sue Fitzgerald, Gary Lewandowski, Laurie Murphy, Beth Simon, Lynda Thomas, and Carol Zander. 2008. Debugging: a review of the literature from an educational perspective. Computer Science Education (June 2008). https://doi.org/10.1080/08993400802114581 Publisher: Routledge. — [doi:10.1080/08993400802114581](https://doi.org/10.1080/08993400802114581) ↗
  21. Diba Mirza, Phillip T Conrad, Christian Lloyd, Ziad Matni, and Arthur Gatin. 2019. Undergraduate teaching assistants in computer science: a systematic literature review. In Proceedings of the 2019 ACM Conference on International Computing Education Research. 31–40. find
  22. OpenAI. 2022. Introducing ChatGPT. https://openai.com/blog/chatgpt ↗
  23. OpenAI. 2023. GPT-4 Blog. https://openai.com/research/gpt-4. [Online; accessed 19-August-2023]. ↗
  24. OpenAI. 2023. GPT-4 Technical Report. arXiv (2023). https://openai.com/research/gpt-4 ↗
  25. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, 3…
  26. D. N. Perkins and Fay Martin. 1986. Fragile Knowledge and Neglected Strategies in Novice Programmers. In Papers Presented at the First Workshop on Empirical Studies of Programmers on Empirical Studies of Programmers (Washington, D.C., USA). Ablex Publishing Corp., USA, 213–229. find
  27. David N Perkins and Fay Martin. 1986. Fragile knowledge and neglected strategies in novice programmers. In Papers presented at the first workshop on empirical studies of programmers on Empirical studies of programmers. 213–229. find
  28. Chris Quintana, Brian J. Reiser, Elizabeth A. Davis, Joseph Krajcik, Eric Fretz, Ravit Golan Duncan, Eleni Kyza, Daniel Edelson, and Elliot Soloway. 2004. A Scaffolding Design Framework for Software to Support Science Inquiry. Journal of the Learning Sciences 13, 3 (2004), 337–386. find
  29. Kelly Rivers and Kenneth R Koedinger. 2017. Data-driven hint generation in vast solution spaces: a self-improving python programming tutor. International Journal of Artificial Intelligence in Education 27 (2017), 37–64. find
  30. Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021. Recipes for Building an Open-Domain Chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational…
  31. Kumar Shridhar, Jakub Macina, Mennatallah El-Assady, Tanmay Sinha, Manu Kapur, and Mrinmaya Sachan. 2022. Automatic Generation of Socratic Sub-questions for Teaching Math Word Problems. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computa… find
  32. James C Spohrer, Edgar Pope, Michael Lipman, Warren Sack, Scott Freiman, David Littman, WL Johnson, and Elliot Soloway. 1985. Bug Catalogue: II, III, IV. Yale University, Department of Computer Science (1985). find
  33. Anaïs Tack and Chris Piech. 2022. The AI Teacher Test: Measuring the Pedagogical Ability of Blender and GPT-3 in Educational Dialogues. In Proceedings of the 15th International Conference on Educational Data Mining, Antonija Mitrovic and Nigel Bosch (Eds.). International Educational Data Mining Soci… find
  34. Lasang Jimba Tamang, Zeyad Alshaikh, Nisrine Ait Khayi, Priti Oli, and Vasile Rus. 2021. A Comparative Study of Free Self-Explanations and Socratic Tutoring Explanations for Source Code Comprehension. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education (Virtual Event, US… — [doi:10.1145/3408877.3432423](https://doi.org/10.1145/3408877.3432423) find
  35. Lev S Vygotsky. 2012. Thought and language. MIT press. find
  36. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35 (2022), 24824–24837.
  37. Judith D Wilson. 1987. A Socratic approach to helping novice programmers debug programs. ACM SIGCSE Bulletin 19, 1 (1987), 179–182. find
  38. David Wood, Jerome S Bruner, and Gail Ross. 1976. The role of tutoring in problem solving. Child Psychology & Psychiatry & Allied Disciplines (1976). find
  39. Mike Wu, Noah Goodman, Chris Piech, and Chelsea Finn. 2021. ProtoTransformer: A meta-learning approach to providing student feedback. arXiv preprint arXiv:2107.14035 (2021). — [arXiv:2107.14035](https://arxiv.org/abs/2107.14035) find
  40. Aman Yadav, Sarah Gretter, Susanne Hambrusch, and Phil Sands. 2016. Expanding computer science education in schools: understanding teacher experiences and challenges. Computer Science Education 26, 4 (2016), 235–254. find
  41. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. In International Conference on Learning Representations. https://openreview.net/forum?id=SkeHuCVFDr This figure "acm-jdslogo.png" is available in "png" format from: ht… ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/al-hossami-2023-language-models-employ-socratic-method/