Elamur Research⌕
☰
home / papers / 2024 / Instruct, Not Assist: LLM-based Multi-Turn Planning and Hierarchical Questioning for Socratic Code Debugging

Instruct, Not Assist: LLM-based Multi-Turn Planning and Hierarchical Questioning for Socratic Code Debugging

Paper · 2024 · Priyanka Kargupta, Ishika Agarwal, Dilek Hakkani Tur et al.

Summary

LLM tutors tend to act like helpful assistants and simply hand students the fix. This undermines learning. The authors target programming education and propose TreeInstruct, an Instructor agent that teaches debugging through Socratic questioning without revealing the answer.

From the problem, the buggy code and the corrected code, TreeInstruct builds a state space: a set of tasks (misunderstandings and bugs) the student must resolve. A Verifier judges each student reply and tracks which tasks are resolved. Questions form a dynamic tree:

  • Sibling questions follow wrong answers.
  • Child questions probe remaining knowledge gaps.
  • A fresh tree starts once a task is resolved.

The authors also release MULTI-DEBUG, an expert-annotated benchmark with multiple, possibly interdependent bugs per problem. On it and on the existing Socratic Debugging Benchmark, TreeInstruct outperforms baselines with both GPT-4 and Llama-3-8b. A small study with real students confirmed this. The work shows that structured planning can turn general LLMs into scalable tutors that guide students rather than solve problems for them.

Key points

  • MULTI-DEBUG: 150 coding problems with incorrect solutions and bug fixes, covering conceptual and syntactical errors in 1- to 3-bug settings.
  • Expert evaluation: TreeInstruct was preferred about 78% of the time on average, and students fixed their code 24.55% more often than with baselines.
  • Robustness to difficulty: performance fell only 2.44% from the Socratic Debugging Benchmark to MD 1-bug and 1.53% from 1-bug to 3-bug. The Vanilla baseline dropped 14.55% from 1-bug to 3-bug.
  • CodeAid comparison: in 57% of cases, the earlier tool CodeAid reveals pseudocode or actual code, and it is judged helpful in just 55% of cases.
  • Limitations:
    • The top success rate is only 77.27%.
    • Dialogues can loop for 20–30 turns with weak students.
    • Syntactic bugs such as missing colons are hard for LLMs to notice.
  • Availability: code and data are released on GitHub (agarwalishika/TreeInstruct).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Illinois Urbana-ChampaignUrbana · United States228 works · 74,591 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksInformation Systems · 1,888 worksComputer Science Applications · 184 works

Cite

Cite as Priyanka Kargupta, Ishika Agarwal, Dilek Hakkani Tur and Jiawei Han. 2024. Instruct, Not Assist: LLM-based Multi-Turn Planning and Hierarchical Questioning for Socratic Code Debugging. Findings of the Association for Computational Linguistics: EMNLP 2024.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.findings-emnlp.553 · arXiv 2406.11709

@inproceedings{kargupta2024instruct,
  title = {{Instruct, Not Assist: LLM-based Multi-Turn Planning and Hierarchical Questioning for Socratic Code Debugging}},
  author = {Priyanka Kargupta, Ishika Agarwal, Dilek Hakkani Tur and Jiawei Han},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2024},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.553},
  eprint = {2406.11709},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2024.findings-emnlp.553}
}

References

23 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Erfan Al-Hossami, Razvan Bunescu, Justin Smith, and Ryan Teehan. 2023a. Can language models employ the socratic method? experiments with code debugging. arXiv preprint arXiv:2310.03210.
  3. Erfan Al-Hossami, Razvan Bunescu, Ryan Teehan, Laurel Powell, Khyati Mahajan, and Mohsen Dorodchi. 2023b. Socratic questioning of novice debuggers: A benchmark dataset and preliminary evaluations. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA…
  4. Sterling Alic, Dorottya Demszky, Zid Mancenido, Jing Liu, Heather Hill, and Dan Jurafsky. 2022. Computationally identifying funneling and focusing questions in classroom discourse. arXiv preprint arXiv:2208.04715. find
  5. Albert T Corbett and John R Anderson. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted interaction, 4:253–278. find
  6. Kathleen Cotton. 1988. Classroom questioning. School improvement research series, 5:1–22. find
  7. Dorottya Demszky and Heather Hill. 2022. The ncte transcripts: A dataset of elementary math classroom transcripts. arXiv preprint arXiv:2211.11772.
  8. Beth A Herbel-Eisenmann and M Lynn Breyfogle. 2005. Questioning our patterns of questioning. Mathematics teaching in the middle school, 10(9):484–489. find
  9. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. find
  10. Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. 2023. Chatgpt for good? on opportunities and challenges of large language models for education. Learning and individual difference…
  11. Majeed Kazemitabaar, Runlong Ye, Xiaoning Wang, Austin Z Henley, Paul Denny, Michelle Craig, and Tovi Grossman. 2024. Codeaid: Evaluating a classroom deployment of an llm-based programming assistant that balances student and educator needs. arXiv preprint arXiv:2401.11314.
  12. Wenhan Lyu, Yimeng Wang, Tingting Rachel Chung, Yifan Sun, and Yixuan Zhang. 2024. Evaluating the effectiveness of llms in introductory computer science education: A semester-long field study. arXiv preprint arXiv:2404.13414. — [arXiv:2404.13414](https://arxiv.org/abs/2404.13414) find
  13. Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J Guibas, and Jascha Sohl-Dickstein. 2015. Deep knowledge tracing. Advances in neural information processing systems, 28.
  14. Anna N Rafferty, Emma Brunskill, Thomas L Griffiths, and Patrick Shafto. 2016. Faster teaching via pomdp planning. Cognitive science, 40(6):1290–1332. find
  15. Husniah Sahamid. 2016. Developing critical thinking through socratic questioning: An action research study. International Journal of Education and Literacy Studies, 4(3):62–72. find
  16. Brad Sheese, Mark Liffiton, Jaromir Savelka, and Paul Denny. 2024. Patterns of student help-seeking when using a large language model-powered programming assistant. In Proceedings of the 26th Australasian Computing Education Conference, pages 49–57. find
  17. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  18. Rose E Wang and Dorottya Demszky. 2024. Edu-convokit: An open-source library for education conversation data. arXiv preprint arXiv:2402.05111. find
  19. Rose E. Wang, Qingyang Zhang, Carly Robinson, Susanna Loeb, and Dorottya Demszky. 2023. Bridging the novice-expert gap via models of decision-making: A case study on remediating math mistakes. find
  20. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.
  21. Judith D Wilson. 1987. A socratic approach to helping novice programmers debug programs. ACM SIGCSE Bulletin, 19(1):179–182. find
  22. Ya-Ting C Yang, Timothy J Newby, and Robert L Bill. 2005. Using socratic questioning to promote critical thinking skills through asynchronous discussion forums in distance learning environments. The american journal of distance education, 19(3):163–181. find
  23. Michael V. Yudelson, Kenneth R. Koedinger, and Geoffrey J. Gordon. 2013. Individualized bayesian knowledge tracing models. In Artificial Intelligence in Education, pages 171–180, Berlin, Heidelberg. Springer Berlin Heidelberg. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kargupta-2024-instruct-not-assist-llm-based/