Elamur Research⌕
☰
home / papers / 2024 / AutoTutor meets Large Language Models: A Language Model Tutor with Rich Pedagogy and Guardrails

AutoTutor meets Large Language Models: A Language Model Tutor with Rich Pedagogy and Guardrails

Paper · 2024 · Sankalan Pal Chowdhury, Vilém Zouhar and Mrinmaya Sachan

Summary

Using LLMs directly as tutors is attractive, but they tend to abandon sound pedagogy. They may leak answers, loop unproductively, or accept wrong solutions, and their black-box behaviour is hard for teachers to adjust. Classic AutoTutor-style systems have the opposite problem: their pedagogy is solid, but every new topic requires costly expert-written scripts. The authors propose a hybrid. Learning scientists still handcraft the overall pedagogical design as a finite state transducer, while an LLM, protected by internal checks, takes over the domain expert's job of filling its state space (solution steps, hints, prompts). They build MWPTutor, a prototype for math word problems with two variants: one that calls an LLM live and one that uses pre-generated (cached) content that humans can verify. On two math word problem datasets, conversations with an AI-simulated student outscored those of an instructed, free-form GPT-4 on overall tutoring quality. Human raters also judged the utterances as following the intended pedagogy. Manual inspection, however, revealed remaining weaknesses, so the authors withhold testing on real students. The work points toward scalable tutoring that stays interpretable and modular.

Key points

  • MWPTutor separates pedagogy design (human-made state machine) from script authoring (delegated to an LLM). It comes in "live" and "cached" variants, and the cached one can be checked by humans before deployment.
  • By design, MWPTutor never approves an incorrect answer. In contrast, free-form GPT-4 sometimes accepted wrong answers, misled students, or rejected a correct equation and drifted until the 60-utterance limit.
  • In a showcased example, MWPTutor guided the student to the answer in 6 tutor utterances. GPT-4 cycled through 29 utterances without the student ever reaching the solution.
  • The motivation draws on prior costs: authoring a traditional ITS takes over 100 expert hours per hour of student interaction, and LISP Tutor alone used 475 hand-written rules.
  • Code and data are released at github.com/eth-lre/MWPTutor. Planned future work includes graph-structured solution spaces, richer error-specific strategies, and smaller specialized LLMs for answer matching.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓ETH ZurichZurich · Switzerland97 works · 9,612 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksInformation Systems · 1,888 worksMedicine · 288 worksHealth Informatics · 227 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Sankalan Pal Chowdhury, Vilém Zouhar and Mrinmaya Sachan. 2024. AutoTutor meets Large Language Models: A Language Model Tutor with Rich Pedagogy and Guardrails. Proceedings of the Eleventh ACM Conference on Learning @ Scale.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1145/3657604.3662041 · arXiv 2402.09216

@inproceedings{chowdhury2024autotutor,
  title = {{AutoTutor meets Large Language Models: A Language Model Tutor with Rich Pedagogy and Guardrails}},
  author = {Sankalan Pal Chowdhury, Vilém Zouhar and Mrinmaya Sachan},
  booktitle = {Proceedings of the Eleventh ACM Conference on Learning @ Scale},
  year = {2024},
  doi = {10.1145/3657604.3662041},
  eprint = {2402.09216},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1145/3657604.3662041}
}

References

37 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. John R. Anderson, C. Franklin Boyle, and Brian J. Reiser. 1985. Intelligent Tutoring Systems. Science 228, 4698 (1985), 456–462. https://doi.org/10.1126/science.228. 4698.456 arXiv:https://www.science.org/doi/pdf/10.1126/science.228.4698.456 — [doi:10.1126/science.228](https://doi.org/10.1126/science.228) ↗
  2. David Baidoo-Anu and Leticia Owusu Ansah. 2023. Education in the era of generative artificial intelligence (AI): Understanding the potential benefits of ChatGPT in promoting teaching and learning. Journal of AI 7, 1 (2023), 52–62. https://dergipark.org.tr/en/pub/jai/issue/77844/1337500 ↗
  3. Benjamin S Bloom. 1984. The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational researcher 13, 6 (1984), 4–16. https://journals.sagepub.com/doi/abs/10.3102/0013189X013006004 ↗
  4. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Demsz… — On the Opportunities and Risks of Foundation Models
  5. Colin Bryson and Len Hand. 2007. The role of engagement in inspiring teaching and learning. Innovations in Education and Teaching International 44, 4 (2007), 349–362. https://doi.org/10.1080/14703290701602748 arXiv:https://doi.org/10.1080/14703290701602748 — [doi:10.1080/14703290701602748](https://doi.org/10.1080/14703290701602748) ↗
  6. Zhiqiang Cai, Xiangen Hu, and Arthur C. Graesser. 2019. Authoring Conversational Intelligent Tutoring Systems. In Adaptive Instructional Systems, Robert A. Sottilare and Jessica Schwarz (Eds.). Springer International Publishing, Cham, 593–603. https://link.springer.com/chapter/10.1007/978-3-030-2234… — [doi:10.1007/978-3-030-22341-0_46](https://doi.org/10.1007/978-3-030-22341-0_46) ↗
  7. Eason Chen, Ray Huang, Han-Shin Chen, Yuen-Hsien Tseng, and Liang-Yi Li. 2023. GPTutor: A ChatGPT-Powered Programming Tool for Code Explanation. In Artificial Intelligence in Education. Posters and Late Breaking Results Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral… — [doi:10.1007/978-3-031-36336-8_50](https://doi.org/10.1007/978-3-031-36336-8_50) find
  8. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan… find
  9. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.14168 (2021). https://arxiv.org… ↗
  10. Chenhang Cui, Yiyang Zhou, Xinyu Yang, Shirley Wu, Linjun Zhang, James Zou, and Huaxiu Yao. 2023. Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challenges. arXiv:2311.03287 [cs.LG] — [arXiv:2311.03287](https://arxiv.org/abs/2311.03287) find
  11. Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2023. Chain-of-Verification Reduces Hallucination in Large Language Models. arXiv:2309.11495 [cs.CL]
  12. Arthur Graesser, Natalie Person, and Derek Harter. 2000. Teaching Tactics in AutoTutor. (01 2000). http://users.sussex.ac.uk/~bend/its2000/graesser.pdf ↗
  13. Arthur C. Graesser. 2016. Conversations with AutoTutor Help Students Learn. International Journal of Artificial Intelligence in Education 26, 1 (01 Mar 2016), 124–132. https://doi.org/10.1007/s40593-015-0086-4 — [doi:10.1007/s40593-015-0086-4](https://doi.org/10.1007/s40593-015-0086-4) ↗
  14. Arthur C. Graesser, Kurt VanLehn, Carolyn P. Rose, Pamela W. Jordan, and Derek Harter. 2001. Intelligent Tutoring Systems with Conversational Dialogue. AI Magazine 22, 4 (Dec. 2001), 39. https://doi.org/10.1609/aimag.v22i4.1591 — [doi:10.1609/aimag.v22i4.1591](https://doi.org/10.1609/aimag.v22i4.1591) ↗
  15. Kenneth R. Koedinger, Vincent Aleven, Neil Heffernan, Bruce McLaren, and Matthew Hockenberry. 2004. Opening the Door to Non-programmers: Authoring Intelligent Tutor Behavior by Demonstration. In Intelligent Tutoring Systems, James C. Lester, Rosa Maria Vicari, and Fábio Paraguaçu (Eds.). Springer Be… find
  16. Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large Language Models are Zero-Shot Reasoners. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc.,…
  17. Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. 2023. Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study. arXiv:2305.13860 [cs.SE]
  18. Jakub Macina, Nico Daheim, Sankalan Pal Chowdhury, Tanmay Sinha, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. 2023. MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems. In The 2023 Conference on Empirical Methods in Natural Language Processi…
  19. Florence Martin and Doris Bolliger. 2018. Engagement Matters: Student Perceptions on the Importance of Engagement Strategies in the Online Learning Environment. 22 (03 2018), 205–222. https://doi.org/10.24059/olj.v22i1.1092 — [doi:10.24059/olj.v22i1.1092](https://doi.org/10.24059/olj.v22i1.1092) ↗
  20. Benjamin Nye, Arthur Graesser, and Xiangen Hu. 2014. AutoTutor and Family: A Review of 17 Years of Natural Language Tutoring. International Journal of Artificial Intelligence in Education 24 (12 2014). https://doi.org/10.1007/s40593-014-0029-5 ↗
  21. Maxwell Nye, Anders Andreassen, Guy Gur-Ari, Henryk Witold Michalewski, Jacob Austin, David Bieber, David Martin Dohan, Aitor Lewkowycz, Maarten Paul Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. Show Your Work: Scratchpads for Intermediate Computation with Language Models. https://a…
  22. OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL]
  23. José Paladines and Jaime Ramirez. 2020. A Systematic Literature Review of Intelligent Tutoring Systems With Dialogue in Natural Language. IEEE Access 8 (2020), 164246–164267. https://doi.org/10.1109/ACCESS.2020.3021383 — [doi:10.1109/access.2020.3021383](https://doi.org/10.1109/access.2020.3021383) ↗
  24. Jim Parsons and Leah Taylor. 2011. Improving Student Engagement. Current Issues in Education 14, 1 (May 2011). https://cie.asu.edu/ojs/index.php/cieatasu/ article/view/745 ↗
  25. Tung Phung, José Cambronero, Sumit Gulwani, Tobias Kohn, Rupak Majumdar, Adish Singla, and Gustavo Soares. 2023. Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models. In Proceedings of the 16th International Conference on Educational Data Mining, Mingyu Feng,… find
  26. Tung Phung, Victor-Alexandru Pădurean, José Cambronero, Sumit Gulwani, Tobias Kohn, Rupak Majumdar, Adish Singla, and Gustavo Soares. 2023. Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors. In Proceedings of the 2023 ACM Conference on International Computing Edu…
  27. Jaakko Rajala, Jenni Hukkanen, Maria Hartikainen, and Pia Niemelä. 2023. "Call me Kiran" – ChatGPT as a Tutoring Chatbot in a Computer Science Course. In Proceedings of the 26th International Academic Mindtrek Conference. 83–94. https://dl.acm.org/doi/pdf/10.1145/3616961.3616974 — [doi:10.1145/3616961.3616974](https://doi.org/10.1145/3616961.3616974) ↗
  28. Brian Reiser. 2004. Scaffolding Complex Learning: The Mechanisms of Structuring and Problematizing Student Work. Journal Of The Learning Sciences 13 (01 2004), 273–304. https://doi.org/10.1207/s15327809jls1303_2 — [doi:10.1207/s15327809jls1303_2](https://doi.org/10.1207/s15327809jls1303_2) ↗
  29. Kathryn M. Rich, T. Andrew Binkowski, Carla Strickland, and Diana Franklin. 2018. Decomposition: A K-8 Computational Thinking Learning Trajectory. In Proceedings of the 2018 ACM Conference on International Computing Education Research (Espoo, Finland) (ICER ’18). Association for Computing Machinery,… — [doi:10.1145/3230977.3230979](https://doi.org/10.1145/3230977.3230979) find
  30. Robin Schmucker, Meng Xia, Amos Azaria, and Tom Mitchell. 2023. Ruf-fle&Riley: Towards the Automated Induction of Conversational Tutoring Systems. arXiv:2310.01420 [cs.CL] — [arXiv:2310.01420](https://arxiv.org/abs/2310.01420) find
  31. D. Sleeman and J.S. Brown. 1982. Intelligent Tutoring Systems. Academic Press. https://books.google.ch/books?id=pjqcAAAAMAAJ ↗
  32. Ting Song and Kurt Becker. 2014. Expert vs. novice: Problem decomposition/re-composition in engineering design. In 2014 International Conference on interactive collaborative learning (ICL). IEEE, 181–190. https://ieeexplore.ieee.org/abstract/ document/7017768 ↗
  33. Shashank Sonkar, Naiming Liu, Debshila Basu Mallick, and Richard G. Baraniuk. 2023. CLASS: A Design Framework for building Intelligent Tutoring Systems based on Learning Science principles. arXiv:2305.13272 [cs.CL]
  34. Ferhat Yarkin and Justin P. Coon. 2022. Simple Gray Coding and LLR Calculation for MDS Modulation Systems. CoRR abs/2201.08237 (2022). arXiv:2201.08237 https://arxiv.org/abs/2201.08237 — [arXiv:2201.08237](https://arxiv.org/abs/2201.08237) ↗
  35. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023. Meta-Math: Bootstrap Your Own Mathematical Questions for Large Language Models. arXiv:2309.12284 [cs.CL]
  36. Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A. Smith. 2023. How Language Model Hallucinations Can Snowball. arXiv:2305.13534 [cs.CL]
  37. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. arXiv:1904.09675 [cs.CL] find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chowdhury-2024-autotutor-meets-large-language-models/