Elamur Research⌕
☰
home / papers / 2026 / Using Interaction Log Data to Evaluate and Improve Feedback Accuracy in an Intelligent Language Tutoring System

Using Interaction Log Data to Evaluate and Improve Feedback Accuracy in an Intelligent Language Tutoring System

Paper · 2026 · Mariia Soliar, Leona Colling, Stephen Bodnar et al.

Summary

Intelligent tutoring systems record detailed learner interactions, but these logs are rarely used to check whether the system's own feedback is correct. This paper does that for FeedBook, a web-based English ITS for 7th-grade learners in German secondary schools. FeedBook uses expert-written rules to turn target answers in gap-filling grammar exercises into anticipated learner errors, each paired with a teacher-written metalinguistic feedback message. The authors drew on authentic logs (5,646 logs from 368 students). First, they passed the logged answers through a pipeline that mimics the feedback mechanism and annotated whether the feedback was accurate, specific, or a generic fallback. This exposed frequent errors, mostly in question formation, that received wrong or unspecific responses. They then revised the relevant rules and tested the new rule set on a separate, unseen dataset. The updated rules performed significantly better than the original ones, and the gains held on new data. The study shows that log-based evaluation can complement expert intuition when maintaining transparent rule-based feedback. This matters for semi-open exercises, in which the acceptable answers are already defined and false positives from LLMs pose a risk.

Key points

  • Data: 5,646 interaction logs from 368 students, collected in the Interact4School and AI2Teach projects. The data come from two collection periods, one used for training (rule refinement) and one for testing generalization.

  • System and tools: FeedBook's offline feedback generation relies on UIMA, ClearNLP, Morpha and Sfst. If no rule-based hypothesis matches a learner's answer, the system shows a default "try again" message.

  • Rule refinement focus: The authors prioritized rules for interrogative constructions, such as question words and auxiliaries in simple past and simple present. They found that even a few rule changes gave statistically significant improvements that generalized to unseen data.

  • Error sources found: Most incorrect feedback in the baseline was misapplied spelling feedback. Some failures came from NLP annotation errors, for example wrong lemmas from UIMA Java SDK 2.9.0 causing default feedback for do/does/did confusion.

  • Acknowledged limitations: The two data collections differed in several ways that were not controlled for:

    • exercises were manual in the first period and automatically generated in the second;
    • difficulty was assigned adaptively in the second period;
    • capitalization errors were ignored in the second period;
    • front-end answer preprocessing was added in the second period.

    These differences may have favored the test results.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TübingenTübingen · Germany75 works · 41,081 citations🔬Leibniz-Institut für WissensmedienTübingen · Germany2 works

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Science Applications · 102 works

Cite

Cite as Mariia Soliar, Leona Colling, Stephen Bodnar and Detmar Meurers. 2026. Using Interaction Log Data to Evaluate and Improve Feedback Accuracy in an Intelligent Language Tutoring System. Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bea-1.14

@inproceedings{soliar2026using,
  title = {{Using Interaction Log Data to Evaluate and Improve Feedback Accuracy in an Intelligent Language Tutoring System}},
  author = {Mariia Soliar, Leona Colling, Stephen Bodnar and Detmar Meurers},
  booktitle = {Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.bea-1.14},
  url = {https://doi.org/10.18653/v1/2026.bea-1.14}
}

References

74 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tutoring System. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 288–299. find
  2. Alan Agresti. 2012. Categorical data analysis, volume 792. John Wiley & Sons. find
  3. Luiz Amaral, Detmar Meurers, and Ramon Ziai. 2011. Analyzing learner language: towards a flexible natural language processing architecture for intelligent language tutors. Computer Assisted Language Learning, 24(1):1–16. find
  4. Wei Dai, Jionghao Lin, Hua Jin, Tongguang Li, Yi-Shan Tsai, Dragan Gašević, and Guanliang Chen. 2023. Can large language models provide feedback to students? A case study on ChatGPT. In 2023 IEEE International Conference on Advanced Learning Technologies (ICALT), pages 323–325. IEEE. find
  5. David Baidoo-Anu and Leticia Owusu Ansah. 2023. Education in the era of generative artificial intelligence (AI): Understanding the potential benefits of ChatGPT in promoting teaching and learning. Journal of AI, 7(1):52–62. find
  6. Jasper Degraeuwe and Patrick Goethals. 2024. Leading by example: The use of generative artificial intelligence to create pedagogically suitable example sentences. In Swedish Language Technology Conference and NLP4CALL, pages 33–48. find
  7. Seyyed Kazem Banihashem, Omid Noroozi, Stan van Ginkel, Leah P Macfadyen, and Harm JA Biemans. 2022. A systematic review of the role of learning analytics in enhancing feedback practices in higher education. Educational Research Review, page 100489. find
  8. Florian Berens, Katharina Wendebourg, Mareike Kholin, Leona Colling, Julia Schmidt-Peterson, Manuel Hopp, Tanja Heck, Stephen Bodnar, Walid El Hefny, Florian Nuxoll, Christoph Deeg, Katja Krey, Josef Schrader, Hannes Schröter, Benjamin Nagengast, Detmar Meurers, and Ulrich Trautwein. 2024. AI2Teach:… find
  9. Dorina Bór. 2023. Improving the user experience of an intelligent tutoring system for proving mathematical statements: Master thesis. find
  10. Christopher Bryant, Zheng Yuan, Muhammad Reza Qorib, Hannan Cao, Hwee Tou Ng, and Ted Briscoe. 2023. Grammatical error correction: A survey of the state of the art. Computational Linguistics, 49(3):643–701. find
  11. Anderson Pinheiro Cavalcanti, Arthur Barbosa, Ruan Carvalho, Fred Freitas, Yi-Shan Tsai, Dragan Gašević, and Rafael Ferreira Mello. 2021. Automatic feedback in online learning environments: A systematic literature review. Computers and Education: Artificial Intelligence, 2:100027. find
  12. Jinho D Choi and Martha Palmer. 2012. Fast and robust part-of-speech tagging using dynamic model selection. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 363–367. find
  13. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. find
  14. Leona Colling, Tanja Heck, and Detmar Meurers. 2023. Reconciling Adaptivity and Task Orientation in the Student Dashboard of an Intelligent Language Robert DeKeyser. 2007. Skill acquisition theory. In Bill VanPatten and Jessica Williams, editors, Theories in Second Language Acquisition: An introduct… find
  15. Bingyu Dong, Jie Bai, Tao Xu, and Yun Zhou. 2024. Large language models in education: A systematic review. In 2024 6th International Conference on Computer Science and Technologies in Education (CSTE), pages 131–134. IEEE. find
  16. Rod Ellis. 2006. Current issues in the teaching of grammar: An SLA perspective. TESOL quarterly, 40(1):83–107. find
  17. Emrah Emirtekin. 2025. Large Language Model-Powered Automated Assessment: A Systematic Review. Applied Sciences, 15(10):5683. find
  18. Ali Kürşat Erümit, İsmail Çetin, Mehmet Kokoç, Temel Kösa, Vasif Nabiyev, and Emine Selin Aygün. 2019. Designing a usability assessment process for adaptive intelligent tutoring systems: A case study. Turkish Online Journal of Qualitative Inquiry, 10(1):141–179. find
  19. Ali Mohammad Fazilatfar, Mohadeseh Sedghi, and Mohammad Javad Ali Beigi. 2024. An attempt to improve grammatical structure retention: Impact of computer-mediated feedback on iranian EFL learners. Journal of Studies in Language Learning and Teaching, 1(2):289–306. find
  20. Yuejiao Fei, Leyang Cui, Sen Yang, Wai Lam, Zhenzhong Lan, and Shuming Shi. 2023. Enhancing grammatical error correction systems with explanations. arXiv preprint arXiv:2305.15676. find
  21. David Ferrucci, Adam Lally, Karin Verspoor, and Eric Nyberg. 2009. Unstructured Information Management Architecture (UIMA) Version 1.0. OASIS Standard. find
  22. Qing-Ke Fu, Di Zou, Haoran Xie, and Gary Cheng. 2022. A review of AWE feedback: types, learning outcomes, and implications. Computer Assisted Language Learning, 37(1-2):179–221. find
  23. Jim Greer and Mary Mark. 2016. Evaluation methods for intelligent tutoring systems revisited. International Journal of Artificial Intelligence in Education, 26:387–392. find
  24. Tanja Heck, Detmar Meurers, and Florian Nuxoll. 2022. Automatic exercise generation to support macro-adaptivity in intelligent language tutoring systems. Intelligent CALL, granular systems and learner data: short papers from EUROCALL 2022, page 162. find
  25. Trude Heift. 2010. Prompting in CALL: A longitudinal study of learner uptake. Modern Language Journal, 94(2):198–216. find
  26. Trude Heift. 2013. Multiple learner errors and meaningful feedback. CALICO Journal, 20(3):533–548. find
  27. Jennifer Hill and Rahul Simha. 2016. Automatic generation of context-based fill-in-the-blank exercises using co-occurrence likelihoods and Google ngrams. In Proceedings of the 11th workshop on innovative use of NLP for building educational applications, pages 23–30. find
  28. Andrea Horbach, Ronja Laarmann-Quante, Lucas Liebenow, Thorben Jansen, Stefan Keller, Jennifer Meyer, Torsten Zesch, and Johanna Fleckenstein. 2022. Bringing automatic scoring into the classroom–measuring the impact of automated analytic feedback on student writing performance. In Swedish Language T… find
  29. Xinyi Huang, Di Zou, Gary Cheng, Xieling Chen, and Haoran Xie. 2023. Trends, research issues and applications of artificial intelligence in language education. Educational Technology & Society, 26(1):112–131. find
  30. Choi Inn-Chull. 2019. Exploring the potential of a computerized corrective feedback system based on a process-oriented qualitative error analysis. J Eng Teach Movie Media, 20(1):89–117. find
  31. Timur Jaganov, John Blake, Julián Villegas, and Nicholas Carr. 2025. Large Language Model-Driven Dynamic Assessment of Grammatical Accuracy in English Language Learner Writing. arXiv preprint arXiv:2505.00931.
  32. Abdulkadir Karaci, Zeynep Piri, Halil İbrahim Akyüz, and Göksal Bilgiici. 2018. Student perceptions of an intelligent tutoring system: A technology acceptance model perspective. Online Submission, 182(22):31–36. find
  33. Anisia Katinskaia, Sardana Ivanova, and Roman Yangarber. 2019. Multiple admissibility in language learning: Judging grammaticality using unlabeled data. In Workshop on Balto-Slavic Natural Language Processing, pages 12–22. find
  34. Semere Kiros Bitew, Johannes Deleu, A Seza Doğruöz, Chris Develder, and Thomas Demeester. 2023. Learning from Partially Annotated Data: Example-aware Creation of Gap-filling Exercises for Language Learning. arXiv e-prints, pages arXiv–2306. find
  35. Fabienne M Van der Kleij, Remco CW Feskens, and Theo JHM Eggen. 2015. Effects of feedback in a computer-based learning environment on students’ learning outcomes: A meta-analysis. Review of educational research, 85(4):475–511. find
  36. Susanne Knoop and Sabrina Wilske. 2013. WordGap - Automatic Generation of Gap-Filling Vocabulary Exercises for Mobile Learning. In Proceedings of Second Workshop NLP Computer-Assisted Language Learning, pages 39–47. find
  37. Masamune Kobayashi, Masato Mita, and Mamoru Komachi. 2024. Large Language Models Are State-ofthe-Art Evaluator for Grammatical Error Correction. In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), pages 68–77. Association for Computational L… find
  38. Kenneth R. Koedinger, Emma Brunskill, Ryan S.J.d. Baker, Elizabeth A. McLaughlin, and John Stamper. 2013. New potentials for data-driven intelligent tutoring system development and optimization. AI Magazine, 34(3):27–41. find
  39. Ehsan Latif, Vincent Liu, and Xiaoming Zhai. 2026. A systematic review of intelligent and robot tutoring systems: evolution, pedagogical design, and AI-driven classification. Smart Learning Environments, 13(1):1. find
  40. Elizabeth Lavolette, Charlene Polio, and Jimin Kahng. 2015. The accuracy of computer-assisted feedback and students’ responses to it. Language, Learning & Technology, 19(2). find
  41. Claudia Leacock, Martin Chodorow, Michael Gamon, and Joel Tetreault. 2014. Automated grammatical error detection for language learners. Morgan & Claypool Publishers. find
  42. Shawn Loewen. 2013. The role of feedback. In The Routledge handbook of second language acquisition, chapter 3, pages 24–40. Routledge. find
  43. Alison Mackey. 1999. INPUT, INTERACTION, AND SECOND LANGUAGE DEVELOPMENT: An Empirical Study of Question Formation in ESL. Studies in Second Language Acquisition, 21(4):557–587. find
  44. Ahmad Marouf, Rami Al-Dahdooh, Mahmoud Jamal Abu Ghali, Ali Osama Mahdi, Basem S Abunasser, and Samy S Abu-Naser. 2024. Enhancing education with artificial intelligence: The role of intelligent tutoring systems. International Journal of Engineering and Information Systems (IJEAIS), pages 10–16. find
  45. Detmar Meurers, Kordula De Kuthy, Florian Nuxoll, Björn Rudzewitz, and Ramon Ziai. 2019. Scaling up intervention studies to investigate real-life foreign language learning in school. Annual Review of Applied Linguistics, 39:161–188. find
  46. Manfred Pienemann, Malcolm Johnston, and Geoff Brindley. 1988. Constructing an acquisition-based procedure for second language assessment. Studies in second language acquisition, 10(2):217–243. find
  47. Guido Minnen, John Carroll, and Darren Pearce. 2001. Applied morphological processing of English. Natural Language Engineering, 7(3):207–233. find
  48. Cristobal Romero and Sebastian Ventura. 2013. Data mining in education. Wiley Interdisciplinary Reviews: Data mining and knowledge discovery, 3(1):12–27. find
  49. Elham Mousavinasab, Nahid Zarifsanaiey, Sharareh R. Niakan Kalhori, Mahnaz Rakhshan, Leila Keikha, and Marjan Ghazi Saeedi. 2021. Intelligent tutoring systems: a systematic review of characteristics, applications, and evaluation methods. Interactive Learning Environments, 29(1):142–163. find
  50. Cristobal Romero and Sebastian Ventura. 2020. Educational data mining and learning analytics: An updated survey. Wiley interdisciplinary reviews: Data mining and knowledge discovery, 10(3):e1355. find
  51. Ryo Nagata. 2019. Toward a task of feedback comment generation for writing learning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3206–3215. find
  52. Ryo Nagata, Masato Hagiwara, Kazuaki Hanawa, Masato Mita, Artem Chernodub, and Olena Nahorna. 2021. Shared task on feedback comment generation for language learners. In Proceedings of the 14th International Conference on Natural Language Generation, pages 320–324. find
  53. Ryo Nagata, Kentaro Inui, and Shin’ichiro Ishikawa. 2020. Creating corpora for research in feedback comment generation. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 340–345. find
  54. Kostiantyn Omelianchuk, Andrii Liubonko, Oleksandr Skurzhanskyi, Artem Chernodub, Oleksandr Korni-ienko, and Igor Samokhin. 2024. Pillars of grammatical error correction: Comprehensive inspection of contemporary approaches in the era of large language models. In Proceedings of the 19th Workshop on I… find
  55. Subhadarshi Panda, Frank Palma Gomez, Michael Flor, and Alla Rozovskaya. 2022. Automatic generation of distractors for fill-in-the-blank exercises with round-trip neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Student Research… find
  56. Cora Parrisius, Ines Pieronczyk, Carolyn Blume, Katharina Wendebourg, Diana Pili-Moss, Mirjam Assmann, Sabine Beilharz, Stephen Bodnar, Leona Colling, Heiko Holz, et al. 2022. Using an intelligent tutoring system within a task-based learning approach in English as a foreign language classes to foste… find
  57. Kenneth A Petersen. 2010. Implicit corrective feedback in computer-guided interaction: Does mode matter? Georgetown University. find
  58. Björn Rudzewitz, Ramon Ziai, Kordula De Kuthy, and Detmar Meurers. 2017. Developing a web-based workbook for English supporting the interaction of students and teachers. In Proceedings of the joint workshop on NLP for Computer Assisted Language Learning and NLP for Language Acquisition, pages 36–46.
  59. Björn Rudzewitz, Ramon Ziai, Kordula De Kuthy, Verena Möller, Florian Nuxoll, and Detmar Meurers. 2018. Generating feedback for English foreign language exercises. In Proceedings of the thirteenth workshop on innovative use of NLP for building educational applications, pages 127–136.
  60. Simón Ruiz, Patrick Rebuschat, and Detmar Meurers. 2023. Supporting individualized practice through intelligent CALL. In Practice and automatization in second language research, pages 119–143. Routledge. find
  61. Helmut Schmid. 2005. A programming language for finite state transducers. In Finite-State Methods and Natural Language Processing (FSMNLP), volume 4002 of Lecture Notes in Computer Science, pages 308–309. Springer. find
  62. Julika Siemer and Marios C Angelides. 1998. A comprehensive method for the evaluation of complete intelligent tutoring systems. Decision support systems, 22(1):85–102. find
  63. Yixiao Song, Kalpesh Krishna, Rajesh Bhatt, Kevin Gimpel, and Mohit Iyyer. 2024. GEE! Grammar Error Explanation with Large Language Models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 754–781, Mexico City, Mexico. Association for Computational Linguistics. find
  64. Maja Stahl, Leon Biermann, Andreas Nehring, and Henning Wachsmuth. 2024. Exploring LLM prompting strategies for joint essay scoring and feedback generation. In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), pages 283– 298, Mexico City, Mex… find
  65. John Stamper, Ruiwei Xiao, and Xinying Hou. 2024. Enhancing LLM-based feedback: Insights from intelligent tutoring systems and the learning sciences. In International Conference on Artificial Intelligence in Education, pages 32–43. Springer. find
  66. Aru Ukenova and Gulmira Bekmanova. 2023. A review of intelligent interactive learning methods. Frontiers in Computer Science, 5. find
  67. Kurt VanLehn. 2011. The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational psychologist, 46(4):197–221. find
  68. Elena Volodina, Christopher Bryant, Andrew Caines, Orphée De Clercq, Jennifer-Carmen Frey, Elizaveta Ershova, Alexandr Rosen, and Olga Vinogradova. 2023. MultiGED-2023 shared task at NLP4CALL: Multilingual Grammatical Error Detection. In Proceedings of the 12th Workshop on NLP for Computer Assisted… find
  69. Chih Hung Wu, Hao-Chiang Koong Lin, Tao-Hua Wang, Tzu-Hsuan Huang, and Yueh-Min Huang. 2022. Affective mobile language tutoring system for supporting language learning. Frontiers in Psychology, 13:833327. find
  70. Lu Yang and Rui Li. 2024. ChatGPT for L2 learning: Current status and implications. System, 124:103351. find
  71. Jingheng Ye, Shang Qin, Yinghui Li, Hai-Tao Zheng, Shen Wang, and Qingsong Wen. 2025. Corrections Meet Explanations: A Unified Framework for Explainable Grammatical Error Correction. arXiv preprint arXiv:2502.15261. find
  72. Chak Yan Yeung, John SY Lee, and Benjamin K Tsou. 2019. Difficulty-aware distractor generation for gap-fill items. In Proceedings of the 17th annual workshop of the Australasian language technology association, pages 159–164. find
  73. Nana Yoshimi, Tomoyuki Kajiwara, Satoru Uchida, Yuki Arase, and Takashi Ninomiya. 2023. Distractor generation for fill-in-the-blank exercises by question type. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 276–… find
  74. M Zerkouk, M Mihoubi, and B Chikhaoui. 2025. A comprehensive review of AI-based intelligent tutoring systems: Applications and challenges. arXiv preprint arXiv:2507.18882. — [arXiv:2507.18882](https://arxiv.org/abs/2507.18882) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/soliar-2026-interaction-log-data-evaluate-improve/