Elamur Research⌕
☰
home / papers / 2026 / AITutor-EvalKit: Exploring the Capabilities of AI Tutors

AITutor-EvalKit: Exploring the Capabilities of AI Tutors

Paper · 2026 · Numaan Naeem, Kaushal Kumar Maurya, Kseniia Petukhova et al.

Summary

AITutor-EvalKit is an open-source toolkit for judging the pedagogical quality of LLM-based tutors. Educators currently struggle to trust these systems, partly because no tool assesses tutor responses on the fly. The toolkit targets Student Mistake Remediation in middle-school math dialogues. It scores a tutor's reply along four dimensions from an established taxonomy: Mistake Identification, Mistake Location, Providing Guidance, and Actionability.

The backend offers three evaluator types:

  • LoMTL, a new lightweight model that fine-tunes one small LLM with LoRA and treats the four ternary labels as jointly learned tasks;
  • Prometheus2 as an open-source LLM judge;
  • GPT-5 as a commercial judge.

A browser-based frontend visualizes the verdicts and can also gather user annotations. LoMTL is trained on MRBench and performs about as well as the best BEA 2025 shared task systems while remaining cheap and scalable. In a user study, participants generally rated LoMTL's judgments as more accurate than those of either LLM judge and found the interface easy to use. This gives researchers and educational stakeholders a transparent, extensible way to probe AI tutors.

Key points

  • Data: MRBench contains 491 dialogues (300 development, 191 test). Each dialogue pairs seven LLM-generated tutor replies with one or two human ones. Every reply is labelled "Yes", "To some extent" or "No" on each dimension. A 10-dialogue demo subset comes from the test set.
  • Model: LoMTL is built on google/gemma-2-2b-it and adds sampling and balanced batching. A single model covers all four dimensions, whereas many shared task entries relied on ensembles or per-dimension models.
  • User study: participants contributed 95 single-response and 115 pairwise annotations. Eleven of them had teaching experience and eight had previously used an AI tutor.
  • Resources: MIT-licensed code is on GitHub (kaushal0494/AITutor-EvalKit), alongside a public demo at demo-ai-tutor.vercel.app and a video walkthrough.
  • Limitations: the toolkit currently covers English only, math only, single-turn evaluation and one taxonomy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksInformation Systems · 1,437 worksComputer Science Applications · 102 works

Cite

Cite as Numaan Naeem, Kaushal Kumar Maurya, Kseniia Petukhova and Ekaterina Kochmar. 2026. AITutor-EvalKit: Exploring the Capabilities of AI Tutors. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 3: System Demonstrations).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-demo.32

@inproceedings{naeem2026aitutor,
  title = {{AITutor-EvalKit: Exploring the Capabilities of AI Tutors}},
  author = {Numaan Naeem, Kaushal Kumar Maurya, Kseniia Petukhova and Ekaterina Kochmar},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 3: System Demonstrations)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-demo.32},
  url = {https://doi.org/10.18653/v1/2026.eacl-demo.32}
}

References

38 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, and 1 others. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219.
  2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  3. Jiyuan An, Xiang Fu, Bo Liu, Xuquan Zong, Cunliang Kong, Shuliang Liu, Shuo Wang, Zhenghao Liu, Liner Yang, Hanghang Fan, and Erhong Yang. 2025. BLCU-ICALL at BEA 2025 shared task: Multi-strategy evaluation of AI tutors. In Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educa… find
  4. Anthropic. 2024. The Claude 3 Model Family: Opus, Sonnet, Haiku. In https://api.semanticscholar.org/CorpusID:268232499. ↗
  5. Steven Bird. 2006. NLTK: the natural language toolkit. In Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, pages 69–72.
  6. Benjamin Samuel Bloom. 1984. The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring. Educational Researcher, 13:16 – 4. find
  7. Jo Boaler. 2013. Ability and mathematics: The mindset revolution that is reshaping education. Forum. find
  8. Donald J. Boyd, Pam L. Grossman, Hamilton Lankford, Susanna Loeb, and James Humphrey Wyckoff. 2008. Teacher Preparation and Student Achievement. Educational Evaluation and Policy Analysis, 31:416– 440. find
  9. Jonathan P Chang, Caleb Chiam, Liye Fu, Andrew Z Wang, Justine Zhang, and Cristian Danescu-Niculescu-Mizil. 2020. Convokit: A toolkit for the analysis of conversations. arXiv preprint arXiv:2005.04246.
  10. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The LLaMA 3 herd of models. arXiv preprint arXiv:2407.21783.
  11. Yuming Fan, Chuangchuang Tan, and Wenyu Song. 2025. BJTU at BEA 2025 shared task: Task-aware prompt tuning and data augmentation for evaluating AI math tutors. In Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2025), pages 1073–1077, Vienna, Aust… find
  12. Kunal Handa, Margarett Clapper, Jessica Boyle, Rose Wang, Diyi Yang, David Yeager, and Dorottya Demszky. 2023. “Mistakes Help Us Grow”: Facilitating and Evaluating Growth Mindset Supportive Language in Classrooms. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Process… find
  13. Baraa Hikal, Mohamed Basem, Islam Oshallah, and Ali Hamdi. 2025. MSA at BEA 2025 shared task: Disagreement-aware instruction tuning for multidimensional evaluation of LLMs as math tutors. In Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2025), p… find
  14. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, and 1 others. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825. find
  15. Irina Jurenka, Markus Kunesch, Kevin R McKee, Daniel Gillick, Shaojian Zhu, Sara Wiltberger, Shubham Milind Phal, Katherine Hermann, Daniel Kasenberg, Avishkar Bhoopchand, and 1 others. 2024. Towards responsible development of generative AI for education: An evaluation-driven approach. arXiv preprin… find
  16. Sean Kelly, Robert Bringe, Esteban Aucejo, and Jane Cooley Fruehwirth. 2020. Using global observation protocols to inform research on teaching effectiveness and school improvement: Strengths and emerging limitations. Education Policy Analysis Archives, 28:62–62. find
  17. Sal Khan. 2024. Khanmigo. find
  18. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models. In Proceedings of the 2024 Conference on Empirical Metho…
  19. Ekaterina Kochmar, Kaushal Maurya, Kseniia Petukhova, KV Aditya Srivatsa, Anaïs Tack, and Justin Vasselli. 2025. Findings of the BEA 2025 shared task on pedagogical ability assessment of AI-powered tutors. In Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applicat… find
  20. Nataliya Kosmyna, Eugene Hauptmann, Ye Tong Yuan, Jessica Situ, Xian-Hao Liao, Ashly Vivian Beres-nitzky, Iris Braunstein, and Pattie Maes. 2025. Your brain on chatgpt: Accumulation of cognitive debt when using an ai assistant for essay writing task. arXiv preprint arXiv:2506.08872. — [arXiv:2506.08872](https://arxiv.org/abs/2506.08872) find
  21. Jakub Macina, Nico Daheim, Sankalan Chowdhury, Tanmay Sinha, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. 2023a. MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems. In Findings of the Association for Computational Linguistics: EMNLP 2023, p…
  22. Jakub Macina, Nico Daheim, Ido Hakimi, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. 2025. MathTutorBench: A benchmark for measuring open-ended pedagogical capabilities of LLM tutors. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 204–221, Suzhou,… find
  23. Jakub Macina, Nico Daheim, Lingzhi Wang, Tanmay Sinha, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. 2023b. Opportunities and Challenges in Neural Dialog Tutoring. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2357– 2372, Dub… find
  24. Yuren Mao, Yuhang Ge, Yijiang Fan, Wenyi Xu, Yu Mi, Zhonghao Hu, and Yunjun Gao. 2025. A survey on lora of large language models. Frontiers of Computer Science, 19(7):197605. find
  25. Kaushal Kumar Maurya, Kv Aditya Srivatsa, Kseniia Petukhova, and Ekaterina Kochmar. 2025. Unifying AI tutor evaluation: An evaluation taxonomy for pedagogical ability assessment of LLM-powered AI tutors. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association… find
  26. Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2024. Large language models: A survey. arXiv preprint arXiv:2402.06196.
  27. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Mathieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn: M… find
  28. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, and 1 others. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXi…
  29. Making: A Case Study on Remediating Math Mistakes. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2174– 2199, Mexico City, Mexico. Association for Computational Linguisti… find
  30. Rose E Wang, Ana T Ribeiro, Carly D Robinson, Susanna Loeb, and Dorottya Demszky. 2024b. Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise. EdWorkingPaper No. 24-1054. Annenberg Institute for School Reform at Brown University. find
  31. T Wolf. 2019. Huggingface’s transformers: State-ofthe-art natural language processing. arXiv preprint arXiv:1910.03771.
  32. Kwang Suk Yoon, Teresa Duncan, Silvia Wen-Yu Lee, Beth Scarloss, and Kathy L Shapley. 2007. Reviewing the evidence on how teacher professional development affects student achievement. issues & answers. rel 2007-no. 033. ERIC. find
  33. Jihyeon Roh and Jinhyun Bang. 2025. bea-jh at BEA 2025 shared task: Evaluating AI-powered tutors through pedagogically-informed reasoning. In Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2025), pages 1049–1059, Vienna, Austria. Association for… find
  34. Anaïs Tack, Ekaterina Kochmar, Zheng Yuan, Serge Bibauw, and Chris Piech. 2023. find
  35. The BEA 2023 shared task on generating AI teacher responses in educational dialogues. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 785– 795, Toronto, Canada. Association for Computational Linguistics. find
  36. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, and 1 others. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. find
  37. Rose Wang and Dorottya Demszky. 2024. Edu-ConvoKit: An Open-Source Library for Education Conversation Data. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: System Demonstrations), pages 61–69… find
  38. Rose Wang, Qingyang Zhang, Carly Robinson, Susanna Loeb, and Dorottya Demszky. 2024a. Bridging the Novice-Expert Gap via Models of Decision- find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/naeem-2026-aitutor-evalkit-exploring-capabilities-ai/