Elamur Research⌕
☰
home / papers / 2023 / Socratic Questioning of Novice Debuggers: A Benchmark Dataset and Preliminary Evaluations

Socratic Questioning of Novice Debuggers: A Benchmark Dataset and Preliminary Evaluations

Paper · 2023 · Erfan Al-Hossami, Razvan Bunescu, Ryan Teehan et al.

Summary

The paper addresses the shortage of teaching staff in computer science education. It asks whether language models can tutor beginner programmers Socratically, guiding them to locate and repair bugs themselves rather than handing over corrected code. The authors frame Socratic debugging as a two-party dialogue. An Instructor agent receives the problem statement, test cases, the student's faulty Python code, a bug description, possible fixes and the dialogue history, and must produce the next guiding utterance. Because many different questions can be valid at any point, experienced tutors wrote a main reply plus several alternative replies for each Instructor turn. This makes it possible to score model outputs automatically for precision and recall. The resulting manually built benchmark was used to test GPT-3.5 and GPT-4. GPT-4 performed far better, but both models stayed below the human expert level. Typical model failures were repeated or irrelevant questions, hints given too early and overly direct hints. The resource gives a structured basis for developing and measuring automated tutoring systems for programming.

Key points

  • The current release of the dataset contains 86 main conversations. Each is built around a novice-level Python exercise, such as Factorial or Fibonacci, with a buggy implementation containing exactly one bug.
  • The bugs model common beginner errors: forgetting zero-based indexing, boundary mistakes, misusing operators, and misunderstanding basic programming constructs.
  • Each Instructor turn includes one main response and alternative responses. Only the main response is carried forward into the conversation history.
  • In the benchmark of GPT-3.5 and GPT-4, GPT-4 clearly outperformed GPT-3.5 but did not reach the human expert score of 70.0.
  • The authors note several limitations: the automatic metrics capture correctness, helpfulness and relevance only partially, misconception coverage is incomplete, and the manual evaluation covered only 5 dialogues. The data is published at github.com/taisazero/socratic-debugging-benchmark.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York UniversityNew York · United States233 works · 104,003 citations🎓University of North Carolina at CharlotteCharlotte · United States10 works · 553 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksInformation Systems · 1,888 worksComputer Science Applications · 184 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,394 works

Cite

Cite as Erfan Al-Hossami, Razvan Bunescu, Ryan Teehan, Laurel Powell, Khyati Mahajan and Mohsen Dorodchi. 2023. Socratic Questioning of Novice Debuggers: A Benchmark Dataset and Preliminary Evaluations. Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.bea-1.57

@inproceedings{alhossami2023socratic,
  title = {{Socratic Questioning of Novice Debuggers: A Benchmark Dataset and Preliminary Evaluations}},
  author = {Erfan Al-Hossami, Razvan Bunescu, Ryan Teehan, Laurel Powell, Khyati Mahajan and Mohsen Dorodchi},
  booktitle = {Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023)},
  year = {2023},
  doi = {10.18653/v1/2023.bea-1.57},
  url = {https://doi.org/10.18653/v1/2023.bea-1.57}
}

References

63 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sterling Alic, Dorottya Demszky, Zid Mancenido, Jing Liu, Heather Hill, and Dan Jurafsky. 2022. Computationally identifying funneling and focusing questions in classroom discourse. find
  2. Mehrdad Alizadeh, Barbara Di Eugenio, Rachel Hars-ley, Nick Green, Davide Fossati, and Omar AlZoubi. 2015. A study of analogy in computer science tutorial dialogues. Trees, 53(19.2):1–6. find
  3. Zeyad Alshaikh, Lasagn Tamang, and Vasile Rus. 2020a. A socratic tutor for source code comprehension. In Artificial Intelligence in Education: 21st International Conference, AIED 2020, Ifrane, Morocco, July 6–10, 2020, Proceedings, Part II 21, pages 15–19. Springer. find
  4. Zeyad Alshaikh, Lasang Jimba Tamang, and Vasile Rus. 2020b. Experiments with a socratic intelligent tutoring system for source code understanding. In The Thirty-Third International Florida Artificial Intelligence Research Society Conference (FLAIRS-32). find
  5. Stella Biderman and Edward Raff. 2022. Fooling moss detection with pretrained language models. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, CIKM ’22, page 2933–2943, New York, NY, USA. Association for Computing Machinery. find
  6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  7. Cory J Butz, Shan Hua, and R Brien Maguire. 2006. A web-based bayesian intelligent tutoring system for computer programming. Web Intelligence and Agent Systems: An International Journal, 4(1):77–97. find
  8. Andrew Caines, Helen Yannakoudakis, Helen Allen, Pascual Pérez-Paredes, Bill Byrne, and Paula Buttery. 2022. The teacher-student chatroom corpus version 2: more lessons, new annotation, automatic detection of sequence shifts. In Proceedings of the 11th Workshop on NLP for Computer Assisted Language… find
  9. Andrew Caines, Helen Yannakoudakis, Helena Edmondson, Helen Allen, Pascual Pérez-Paredes, Bill Byrne, and Paula Buttery. 2020. The teacher-student chatroom corpus. In Proceedings of the 9th Workshop on NLP for Computer Assisted Language Learning, pages 10–20, Gothenburg, Sweden. LiU Electronic Press…
  10. Tracy Camp, W Richards Adrion, Betsy Bizot, Susan Davidson, Mary Hall, Susanne Hambrusch, Ellen Walker, and Stuart Zweben. 2017. Generation cs: the growth of computer science. ACM Inroads, 8(2):44– 50. find
  11. Lin Chen, Barbara Di Eugenio, Davide Fossati, Stellan Ohlsson, and David Cosejo. 2011. Exploring effective dialogue act sequences in one-on-one computer science tutoring dialogues. In Proceedings of the Sixth Workshop on Innovative Use of NLP for Building Educational Applications, pages 65–75, Portl…
  12. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  13. Konstantina Chrysafiadi and Maria Virvou. 2012. Evaluating the integration of fuzzy logic into the student model of a web-based learning environment. Expert systems with applications, 39(18):13127–13134. find
  14. Robert Costello. 2012. Adaptive intelligent personalised learning (aipl) environment. Ph.D. thesis. find
  15. Tyne Crow, Andrew Luxton-Reilly, and Burkhard Wuen-sche. 2018. Intelligent tutoring systems for programming education: a systematic review. In Proceedings of the 20th Australasian Computing Education Conference, pages 53–62. find
  16. Dorottya Demszky, Jing Liu, Zid Mancenido, Julie Cohen, Heather Hill, Dan Jurafsky, and Tatsunori Hashimoto. 2021. Measuring conversational uptake: A case study on student-teacher interactions. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th In…
  17. Mohsen M Dorodchi, Nasrin Dehbozorgi, Aileen Benedict, Erfan Al-Hossami, and Alexandria Benedict. 2020. Scaffolding a team-based active learning course to engage students: A multidimensional approach. In 2020 ASEE Virtual Annual Conference Content Access. find
  18. James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. 2022. The robots are coming: Exploring the implications of openai codex on introductory programming. In Australasian Computing Education Conference, ACE ’22, page 10–19, New York, NY, USA. Association for Comp… find
  19. Zvi Galil. 1986. Efficient algorithms for finding maximum matching in graphs. ACM Computing Surveys (CSUR), 18(1):23–38. find
  20. Alex Gerdes, Bastiaan Heeren, Johan Jeuring, and L Thomas Van Binsbergen. 2017. Ask-elle: an adaptable programming tutor for haskell giving automated feedback. International Journal of Artificial Intelligence in Education, 27(1):65–100. find
  21. Prakhar Gupta, Shikib Mehri, Tiancheng Zhao, Amy Pavel, Maxine Eskenazi, and Jeffrey P Bigham. 2019. Investigating evaluation of open-domain dialogue systems with human generated multiple references. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 379–391. find
  22. Timotej Lazar, Martin Možina, and Ivan Bratko. 2017. Automatic extraction of ast patterns for debugging student programs. In Artificial Intelligence in Education: 18th International Conference, AIED 2017, Wuhan, China, June 28–July 1, 2017, Proceedings 18, pages 162–174. Springer. find
  23. Victor CS Lee, Yuen-Tak Yu, Chung Man Tang, Tak-Lam Wong, and Chung Keung Poon. 2018. Vida: A virtual debugging advisor for supporting learning in computer programming courses. Journal of Computer Assisted Learning, 34(3):243–258. find
  24. Teemu Lehtinen, Aleksi Lukkarinen, and Lassi Haara-nen. 2021. Students struggle to explain their own program code. In Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education V. 1, pages 206–212. find
  25. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  26. Mary Lou Maher, Celine Latulipe, Heather Lipford, and Audrey Rorrer. 2015. Flipped classroom strategies for cs education. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education, pages 218–223. find
  27. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654.
  28. Jessica McBroom, Irena Koprinska, and Kalina Yacef. 2021. A survey of automated programming hint generation: The hints framework. ACM Computing Surveys (CSUR), 54(8):1–27. find
  29. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring coding challenge competence with apps. NeurIPS.
  30. Diba Mirza, Phillip T Conrad, Christian Lloyd, Ziad Matni, and Arthur Gatin. 2019. Undergraduate teaching assistants in computer science: a systematic literature review. In Proceedings of the 2019 ACM Conference on International Computing Education Research, pages 31–40. find
  31. Sebastian Hobert. 2019. Say Hello to ‘Coding Tutor’! Design and Evaluation of a Chatbot-based Learning System Supporting Students to Learn to Program. find
  32. ICIS 2019 Proceedings. find
  33. Barry Peddycord Iii, Andrew Hicks, and Tiffany Barnes. 2014. Generating hints for programming problems using intermediate output. In Educational Data Mining 2014. Citeseer. find
  34. Johan Jeuring, L. Thomas van Binsbergen, Alex Gerdes, and Bastiaan Heeren. 2014. Model solutions and properties for diagnosing student programs in ask-elle. In Proceedings of the Computer Science Education Research Conference, CSERC ’14, page 31–40, New York, NY, USA. Association for Computing Machi… find
  35. Nam Ju Kim, Brian R Belland, and Andrew E Walker. 2018. Effectiveness of computer-based scaffolding in the context of problem-based learning for STEM education: Bayesian meta-analysis. Educational Psychology Review, 30:397–429. find
  36. Elham Mousavinasab, Nahid Zarifsanaiey, Sharareh R. Niakan Kalhori, Mahnaz Rakhshan, Leila Keikha, and Marjan Ghazi Saeedi. 2021. Intelligent tutoring systems: a systematic review of characteristics, applications, and evaluation methods. Interactive Learning Environments, 29(1):142–163. find
  37. National Council of Teachers of Mathematics. 2014. Principles to actions: Ensuring mathematical success for all. NCTM, National Council of Teachers of Mathematics, Reston, VA. find
  38. OpenAI. 2022. Introducing chatgpt. find
  39. OpenAI. 2023. Gpt-4 technical report. arXiv.
  40. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
  41. Roy D Pea. 1986. Language-independent conceptual “bugs” in novice programming. Journal of educational computing research, 2(1):25–36. find
  42. David N Perkins and Fay Martin. 1986. Fragile knowledge and neglected strategies in novice programmers. In Papers presented at the first workshop on empirical studies of programmers on Empirical studies of programmers, pages 213–229. find
  43. Kumar Shridhar, Jakub Macina, Mennatallah El-Assady, Tanmay Sinha, Manu Kapur, and Mrinmaya Sachan. 2022. Automatic generation of socratic subquestions for teaching math word problems. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4136–4149, Abu Dha…
  44. Yizhou Qian and James Lehman. 2017. Students’ misconceptions and other difficulties in introductory programming: A literature review. ACM Transactions on Computing Education (TOCE), 18(1):1–24. find
  45. Katherine Stasaski, Kimberly Kao, and Marti A Hearst. 2020. Cima: A large open access dialogue dataset for tutoring. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 52–64.
  46. Chris Quintana, Brian J. Reiser, Elizabeth A. Davis, Joseph Krajcik, Eric Fretz, Ravit Golan Duncan, Eleni Kyza, Daniel Edelson, and Elliot Soloway. 2004. A scaffolding design framework for software to support science inquiry. Journal of the Learning Sciences, 13(3):337–386. find
  47. Anaïs Tack and Chris Piech. 2022. The AI teacher test: Measuring the pedagogical ability of blender and GPT-3 in educational dialogues. In Proceedings of the 15th International Conference on Educational Data Mining, pages 522–529, Durham, United Kingdom. International Educational Data Mining Society…
  48. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1– 67.
  49. Lasang Jimba Tamang, Zeyad Alshaikh, Nisrine Ait Khayi, Priti Oli, and Vasile Rus. 2021. A comparative study of free self-explanations and socratic tutoring explanations for source code comprehension. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education, SIGCSE ’21, page… find
  50. Brian J Reiser. 2004. Scaffolding complex learning: The mechanisms of structuring and problematizing student work. The Journal of the Learning sciences, 13(3):273–304. find
  51. Kelly Rivers and Kenneth R Koedinger. 2017. Data-driven hint generation in vast solution spaces: a self-improving python programming tutor. International Journal of Artificial Intelligence in Education, 27:37– 64. find
  52. Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021. Recipes for building an open-domain chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational…
  53. Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. 2022. Automatic generation of programming exercises and code explanations using large language models. In Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1, ICER ’22, page 27–43, New York, NY, USA.… find
  54. Erik Saule. 2018. Experiences on teaching parallel and distributed computing for undergraduates. In 2018 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pages 361–368. find
  55. Gladys Tyen, Mark Brenchley, Andrew Caines, and Paula Buttery. 2022. Towards an open-domain chatbot for language practice. In Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022), pages 234–249, Seattle, Washington. Association for Computational… find
  56. Lev S Vygotsky. 2012. Thought and language. MIT press. find
  57. Steve Walsh. 2006. Investigating classroom discourse. Routledge. find
  58. Judith D Wilson. 1987. A socratic approach to helping novice programmers debug programs. ACM SIGCSE Bulletin, 19(1):179–182. find
  59. David Wood, Jerome S Bruner, and Gail Ross. 1976. The role of tutoring in problem solving. Child Psychology & Psychiatry & Allied Disciplines. find
  60. Terry Wood. 1994. Patterns of interaction and the culture of mathematics classrooms. In Cultural perspectives on the mathematics classroom, pages 149–168. Springer. find
  61. Mike Wu, Noah Goodman, Chris Piech, and Chelsea Finn. 2021. Prototransformer: A meta-learning approach to providing student feedback. arXiv preprint arXiv:2107.14035. — [arXiv:2107.14035](https://arxiv.org/abs/2107.14035) find
  62. Aman Yadav, Sarah Gretter, Susanne Hambrusch, and Phil Sands. 2016. Expanding computer science education in schools: understanding teacher experiences and challenges. Computer Science Education, 26(4):235–254. find
  63. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hossami-2023-socratic-questioning-novice-debuggers-benchmark/