Elamur Research⌕
☰
home / papers / 2023 / CLASS: A Design Framework for Building Intelligent Tutoring Systems Based on Learning Science principles

CLASS: A Design Framework for Building Intelligent Tutoring Systems Based on Learning Science principles

Paper · 2023 · Shashank Sonkar, Naiming Liu, Debshila Mallick et al.

Summary

Building intelligent tutoring systems (ITS) has traditionally depended on rules or constraints written by human experts. Most newer LLM-based tutors simply prompt commercial APIs, which limits scalability and puts them behind a paywall. The authors propose CLASS, a framework for fine-tuning an open LLM into a tutor using learning science principles: problem decomposition, scaffolding and socio-constructivist learning. CLASS relies on two synthetic datasets generated by GPT-4:

  • Scaffolding dataset: hard problems split into subproblems, each with hints, a wrong student answer and corrective feedback.
  • Conversational dataset: simulated student–tutor dialogues that use a fixed response template, which makes the model's decisions inspectable.

An indexed search over educational content is added to limit hallucinations. As a proof of concept, they fine-tune Vicuna-13b into SPOCK, a tutor for introductory college biology. Four biology experts evaluated SPOCK while role-playing students. They praised how it decomposes questions and how encouraging its tone is. The work offers a reusable, open recipe for subject-specific tutors, though evaluation with real students is still missing.

Key points

  • SPOCK is built on Vicuna-13b, a 13-billion-parameter open model. Code and models are released at github.com/luffycodes/Tutorbot-Spock.
  • Problems in the scaffolding dataset target Bloom's taxonomy levels 4–6. Each entry includes subproblems, hints, an incorrect response and feedback.
  • A subproblem-augmented dual-retrieval method searches using both the main problem and its subproblems. It raises LLaMA's MMLU accuracy by 3.5%, outperforming retrieval based on the main problem alone.
  • The evaluation protocol covers factual correctness, relevance, completeness (including off-topic guardrails) and motivation. Four biology subject-matter experts rated SPOCK on a 1–5 scale.
  • The authors acknowledge two limitations: residual factual errors and hallucinations, and weak performance on math or numeric tasks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Rice UniversityHouston · United States15 works · 2,036 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksInformation Systems · 1,888 works

Cite

Cite as Shashank Sonkar, Naiming Liu, Debshila Mallick and Richard Baraniuk. 2023. CLASS: A Design Framework for Building Intelligent Tutoring Systems Based on Learning Science principles. Findings of the Association for Computational Linguistics: EMNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-emnlp.130 · arXiv 2305.13272

@inproceedings{sonkar2023class,
  title = {{CLASS: A Design Framework for Building Intelligent Tutoring Systems Based on Learning Science principles}},
  author = {Shashank Sonkar, Naiming Liu, Debshila Mallick and Richard Baraniuk},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.130},
  eprint = {2305.13272},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.findings-emnlp.130}
}

References

41 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abubakar Abid, Ali Abdalla, Ali Abid, Dawood Khan, Abdulrahman Alfozan, and James Zou. 2019. Gradio: Hassle-Free Sharing and Testing of ML Models in the Wild. arXiv preprint arXiv:1906.02569. — [arXiv:1906.02569](https://arxiv.org/abs/1906.02569) find
  2. John R Anderson, Albert T Corbett, Kenneth R Koedinger, and Ray Pelletier. 1995. Cognitive tutors: Lessons learned. The journal of the learning sciences, 4(2):167–207. find
  3. Maria Bannert. 2003. Effekte metakognitiver Lernhil-fen auf den Wissenserwerb in vernetzten Lernumge-bungen. Zeitschrift für Pädagogische Psychologie, 17(1):13–25. find
  4. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of Artificial General Intelligence: Early experiments with GPT-4. arXiv preprint arXiv:2303.12712.
  5. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* Chat-GPT Quality. tutor with dialogue in natural language. Behavior… find
  6. Joshua B Halpern. 2017. Libretexts: a flexible online open system for disseminating educational materials relevant to geophysics at all levels. In AGU Fall Meeting Abstracts, volume 2017, pages ED31A–0275. find
  7. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring Massive Multitask Language Understanding. CoRR, abs/2009.03300.
  8. Ted Kesler, Pablo PL Tinio, and Brian T Nolan. 2016. What’s our position? A critical media literacy study of popular culture websites with eighth-grade special education students. Reading & Writing Quarterly, 32(1):1–26. find
  9. John Koetsier. 2023. OpenAI’s newest chatbot had a bug that exposed ’a very small’ amount of users’ data. find
  10. Naiming Liu, Shashank Sonkar, Zichao Wang, Simon Woodhead, and Richard G Baraniuk. 2023. Novice Learner and Expert Tutor: Evaluating Math Reasoning Abilities of Large Language Models with Misconceptions. arXiv preprint arXiv:2310.02439.
  11. Naiming Liu, Zichao Wang, Richard Baraniuk, and Andrew Lan. 2022. Open-ended knowledge tracing for computer science education. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3849–3862. find
  12. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
  13. Jakub Macina, Nico Daheim, Sankalan Pal Chowdhury, Tanmay Sinha, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. 2023. MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems. arXiv preprint arXiv:2305.14536.
  14. Mary Ann Clark, Jung Choi, and Matthew Douglas. 2021. Biology 2e. OpenStax. find
  15. Mashable. 2023. ChatGPT The Bard is giving out free Windows 11 keys. find
  16. Jack Conklin. 2005. A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives complete edition. find
  17. Antonija Mitrovic, Stellan Ohlsson, and Devon K Barrow. 2013. The effect of positive feedback in a constraint-based intelligent tutoring system. Computers & Education, 60(1):264–272. find
  18. Albert T Corbett and John R Anderson. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted interaction, 4:253–278. find
  19. K Anders Ericsson. 2017. Protocol analysis. A companion to cognitive science, pages 425–432. find
  20. Shi Feng, Alejandra J Magana, and Dominic Kao. 2021. A systematic review of literature on the effectiveness of intelligent tutoring systems in STEM. In 2021 IEEE Frontiers in Education Conference (FIE), pages 1–9. IEEE. find
  21. Arthur C Graesser, Shulan Lu, George Tanner Jackson, Heather Hite Mitchell, Mathew Ventura, Andrew Olney, and Max M Louwerse. 2004. find
  22. AutoTutor: A OpenAI. 2023. GPT-4 Technical Report.
  23. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
  24. Ansh Radhakrishnan, Karina Nguyen, Anna Chen, Carol Chen, Carson Denison, Danny Hernandez, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilė Lukošiūtė, et al. 2023. Question decomposition improves the faithfulness of model-generated reasoning. arXiv preprint arXiv:2307.11768. find
  25. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505–3506. find
  26. Valerie J Shute, Chen Sun, and Jodi Asbell-Clarke. 2017. Demystifying computational thinking. Educational research review, 22:142–158. find
  27. Shashank Sonkar and Richard G Baraniuk. 2023. Deduction under Perturbed Evidence: Probing Student Simulation Capabilities of Large Language Models. arXiv preprint arXiv:2305.14507. — [arXiv:2305.14507](https://arxiv.org/abs/2305.14507) find
  28. Shashank Sonkar, MyCo Le, Xinghe Chen, Naiming Liu, Debshila Basu Mallick, and Richard G Baraniuk. 2023. Code Soliloquies for Accurate Calculations in Large Language Models. arXiv preprint arXiv:2309.12161. find
  29. Shashank Sonkar, Andrew E Waters, Andrew S Lan, Phillip J Grimaldi, and Richard G Baraniuk. 2020. qdkt: Question-centric Deep Knowledge Tracing. arXiv preprint arXiv:2005.12442. — [arXiv:2005.12442](https://arxiv.org/abs/2005.12442) find
  30. C Addison Stone. 1998. The metaphor of scaffolding: Its utility for the field of learning disabilities. Journal of learning disabilities, 31(4):344–364. find
  31. Pramuditha Suraweera and Antonija Mitrovic. 2002. KERMIT: A constraint-based tutor for database modeling. In Intelligent Tutoring Systems: 6th International Conference, ITS 2002 Biarritz, France and San Sebastian, Spain, June 2–7, 2002 Proceedings 6, pages 377–387. Springer. find
  32. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford Alpaca: An Instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca. ↗
  33. Danielle Thomas, Xinyu Yang, Shivang Gupta, Ade-tunji Adeniran, Elizabeth Mclaughlin, and Kenneth Koedinger. 2023. When the Tutor Becomes the Student: Design and Evaluation of Efficient Scenario-Based Lessons for Tutors. In LAK23: 13th International Learning Analytics and Knowledge Conference, LAK20… find
  34. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  35. Fons JR Van de Vijver and Kwok Leung. 2021. Methods and data analysis for cross-cultural research, volume 116. Cambridge University Press. find
  36. Lev Semenovich Vygotsky and Michael Cole. 1978. Mind in society: Development of higher psychological processes. Harvard university press. find
  37. Jeannette M Wing. 2006. Computational thinking. Communications of the ACM, 49(3):33–35. find
  38. Rainer Winkler and Matthias Söllner. 2018. Unleashing the potential of chatbots in education: A state-ofthe-art analysis. In Academy of management annual meeting (AOM). find
  39. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… — Transformers: State-of-the-Art Natural Language Processing
  40. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023a. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena.
  41. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023b. Judging llm-as-a-judge with mt-bench and chatbot arena.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sonkar-2023-class-design-framework-building-intelligent/