Elamur Research⌕
☰
home / papers / 2020 / Automated Personalized Feedback Improves Learning Gains in An Intelligent Tutoring System

Automated Personalized Feedback Improves Learning Gains in An Intelligent Tutoring System

Paper · 2020 · Ekaterina Kochmar, Dung Do Vu, Robert Belfer et al.

Summary

Dialogue-based intelligent tutoring systems (ITS) struggle to respond well to the wide variety of wrong answers students give, and hand-written feedback rules do not scale. Working on Korbit, a large dialogue-based ITS for data science and machine learning, the authors build three kinds of fully automatic feedback generators. Text hints come from reference solutions. The system finds question keywords, uses spaCy dependency parsing to pick a span that avoids them, and wraps that span in a discourse phrase. A Random Forest then ranks the candidates using one of three feature sets: linguistic features alone (baseline), those plus simple student-performance statistics (shallow personalization), or those plus features from up to four earlier dialogue turns (deep personalization). Further interventions come from Wikipedia explanations filtered by a Decision Tree and from math hints, which work by scoring alternative parse trees of LaTeX input. In a live study with 183 students, the deep model gave the largest learning gains and significantly outperformed the non-personalized baseline on second attempts. The work suggests that data-driven personalized feedback can improve learning without expert authoring and can be moved to other domains.

Key points

  • Training and evaluation data: The hint-ranking models were trained on 450 logged student–system interactions and first compared offline with 50-fold cross-validation. The live evaluation then used 796 annotated interactions from 183 students taking the machine learning course in January–February 2020.
  • Learning gains across all attempts: The share of correct answers after feedback was 48.53% for deep personalization, 46.51% for shallow and 39.47% for the baseline.
  • Learning gains before the second attempt: The figures were 60.47%, 51.43% and 37.93% respectively. The deep-vs-baseline difference was significant by a z-test (p = 0.03005).
  • Wikipedia-based explanations: These were built from a 2 GB Wikipedia crawl on machine learning. Explanations taken directly from an article's first sentence and those generated from the rest of the article produced similar learning gains. Students judged one or both types helpful in 83.33% of cases.
  • Mathematical hints: Two domain experts reviewed 86 interactions containing a wrong equation. They rated over 90% of the hints as "very" or "somewhat" useful. Unlike other math ITS that rely on hand-crafted test cases, this feedback is fully automated.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom287 works · 125,174 citations🎓McGill UniversityMontreal · Canada97 works · 11,755 citations🔬Mila - Quebec Artificial Intelligence InstituteMontreal · Canada73 works · 14,704 citations🎓École de Technologie SupérieureMontreal · Canada3 works · 165 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksInformation Systems · 1,888 worksComputer Science Applications · 184 works

Cite

Cite as Ekaterina Kochmar, Dung Do Vu, Robert Belfer, Varun Gupta, Iulian Vlad Serban and Joelle Pineau. 2020. Automated Personalized Feedback Improves Learning Gains in An Intelligent Tutoring System. Lecture Notes in Computer Science.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1007/978-3-030-52240-7_26 · arXiv 2005.02431

@inproceedings{kochmar2020automated,
  title = {{Automated Personalized Feedback Improves Learning Gains in An Intelligent Tutoring System}},
  author = {Ekaterina Kochmar, Dung Do Vu, Robert Belfer, Varun Gupta, Iulian Vlad Serban and Joelle Pineau},
  booktitle = {Lecture Notes in Computer Science},
  year = {2020},
  doi = {10.1007/978-3-030-52240-7\_26},
  eprint = {2005.02431},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1007/978-3-030-52240-7_26}
}

References

27 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AbuEl-Reesh, J.Y., Abu-Naser, S.S.: An Intelligent Tutoring System for Learning Classical Cryptography Algorithms (CCAITS). International Journal of Academic and Applied Research (IJAAR) 2(2), 1–11 (2018) find
  2. Agha, M., Jarghon, A., Abu-Naser, S.: An Intelligent Tutoring Systems For Teating SQL. International Journal of Academic Information Systems Research (IJAISR) pp. 1–7 (02 2018) find
  3. Ahn, J.W., Chang, M., Watson, P., Tejwani, R., Sundararajan, S., Abuelsaad, T., Prabhu, S.: Adaptive Visual Dialog for Intelligent Tutoring Systems. In: International Conference on Artificial Intelligence in Education. pp. 413–418. Springer (2018) find
  4. Al-Dahdooh, R., Abu-Naser, S.: Development and Evaluation of the Oracle Intelligent Tutoring System (OITS). European Academic Research 4, 8711–8721 (01 2017) find
  5. Al-Nakhal, M., Abu-Naser, S.: Adaptive Intelligent Tutoring System for Learning Computer Theory. European Academic Research 4, 8770–8782 (01 2017) find
  6. Al Rekhawi, H., Abu-Naser, S.: Android Applications UI Development Intelligent Tutoring System. International Journal of Engineering and Information Systems (IJEAIS) pp. 1–14 (01 2018) find
  7. Albacete, P., Jordan, P., Katz, S., Chounta, I.A., McLaren, B.M.: The Impact of Student Model Updates on Contingent Scaffolding in a Natural-Language Tutoring System. In: International Conference on Artificial Intelligence in Education. pp. 37–47. Springer (2019) find
  8. Anderson, J.R., Boyle, C.F., Reiser, B.J.: Intelligent tutoring systems. Science 228(4698), 456–462 (1985) find
  9. Bdenbender, J., Frischauf, A., Goguadze, G., Er, M., Libbrecht, P., Ullrich, C.: Using Computer Algebra Systems as Cognitive Tools. vol. 2363, pp. 802–810 (06 2002) find
  10. Cazden, C.: Peekaboo as an Instructional Model: Discourse Development at Home and at School. Papers and Reports on Child Language Development, No. 17 (1979) find
  11. Chi, M., Koedinger, K., Gordon, G., Jordan, P., Vanlehn, K.: Instructional Factors Analysis: A Cognitive Model For Multiple Instructional Interventions. In: EDM 2011 - Proceedings of the 4th International Conference on Educational Data Mining. pp. 61–70 (01 2011) find
  12. Goguadze, G., Palomo, A.G., Melis, E.: Interactivity of Exercises in ActiveMath. In: ICCE. pp. 109–115 (2005) find
  13. Graesser, A.C., Cai, Z., Morgan, B., Wang, L.: Assessment with computer agents that engage in conversational dialogues and trialogues with learners. Computers in Human Behavior 76, 607 – 616 (2017). https://doi.org/https://doi.org/10.1016/j.chb.2017.03.041, http://www.sciencedirect.com/science/artic… — [doi:10.1016/j.chb.2017.03.041](https://doi.org/10.1016/j.chb.2017.03.041) find
  14. Graesser, A.C., Chipman, P., Haynes, B.C., Olney, A.: AutoTutor: An intelligent tutoring system with mixed-initiative dialogue. IEEE Transactions on Education 48(4), 612–618 (2005) find
  15. Graesser, A.C., VanLehn, K., Rosé, C.P., Jordan, P.W., Harter, D.: Intelligent tutoring systems with conversational dialogue. AI magazine 22(4), 39–39 (2001) find
  16. Hennecke, M.: Online Diagnose in intelligenten mathematischen Lehr-Lern-Systemen. VDI-Verlag (1999) find
  17. Leelawong, K., Biswas, G.: Designing learning by teaching agents: The Betty’s Brain system. International Journal of Artificial Intelligence in Education 18(3), 181–208 (2008) find
  18. Lin, C.F., chu Yeh, Y., Hung, Y.H., Chang, R.I.: Data mining for providing a personalized learning path in creativity: An application of decision trees. Computers & Education 68, 199 – 210 (2013). https://doi.org/https://doi.org/10.1016/j.compedu.2013.05.009, http://www.sciencedirect.com/science/art… — [doi:10.1016/j.compedu.2013.05.009](https://doi.org/10.1016/j.compedu.2013.05.009) find
  19. Melis, E., Siekmann, J.: ActiveMath: An Intelligent Tutoring System for Mathematics. In: Rutkowski, L., Siekmann, J.H., Tadeusiewicz, R., Zadeh, L.A. (eds.) Artificial Intelligence and Soft Computing - ICAISC 2004. pp. 91–101. Springer Berlin Heidelberg, Berlin, Heidelberg (2004) find
  20. Munshi, A., Biswas, G.: Personalization in OELEs: Developing a Data-Driven Framework to Model and Scaffold SRL Processes. In: International Conference on Artificial Intelligence in Education. pp. 354–358. Springer (2019) find
  21. Nye, B.D., Graesser, A.C., Hu, X.: AutoTutor and family: A review of 17 years of natural language tutoring. International Journal of Artificial Intelligence in Education 24(4), 427–469 (2014) find
  22. Passier, H., Jeuring, J.: Feedback in an interactive equation solver (2006) find
  23. Qwaider, S.R., Abu-Naser, S.S.: Excel Intelligent Tutoring System. International Journal of Academic Information Systems Research (IJAISR) 2(2), 8–18 (2018) find
  24. Rus, V., Stefanescu, D., Baggett, W., Niraula, N., Franceschetti, D., Graesser, A.C.: Macro-adaptation in conversational intelligent tutoring matters. In: International Conference on Intelligent Tutoring Systems. pp. 242–247. Springer (2014) find
  25. Rus, V., Stefanescu, D., Niraula, N., Graesser, A.C.: DeepTutor: towards macro-and micro-adaptive conversational intelligent tutoring at scale. In: Proceedings of the first ACM conference on Learning@ scale conference. pp. 209–210 (2014) find
  26. Ventura, M., Chang, M., Foltz, P., Mukhi, N., Yarbro, J., Salverda, A.P., Behrens, J., Ahn, J.w., Ma, T., Dhamecha, T.I., et al.: Preliminary evaluations of a dialogue-based digital tutor. In: International Conference on Artificial Intelligence in Education. pp. 480–483. Springer (2018) find
  27. Wu, L., Looi, C.K.: Agent Prompts: Scaffolding Students for Productive Reflection in an Intelligent Learning Environment. In: Aleven, V., Kay, J., Mostow, J. (eds.) Intelligent Tutoring Systems. pp. 426–428. Springer Berlin Heidelberg, Berlin, Heidelberg (2010) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kochmar-2020-automated-personalized-feedback-improves-learning/