Elamur Research⌕
☰
home / papers / 2026 / Confirming Correct, Missing the Rest: LLM Tutoring Agents Struggle Where Feedback Matters Most

Confirming Correct, Missing the Rest: LLM Tutoring Agents Struggle Where Feedback Matters Most

Paper · 2026 · Tahreem Yasir, Wenbo Li, Sam Gilson et al.

Summary

LLM tutors are often proposed as conversational add-ons to intelligent tutoring systems (ITS). It was unclear whether they can separate three kinds of student steps that ITS routinely distinguish: optimal moves, logically sound but longer detours, and outright errors. The authors build a benchmark for this three-way, step-level diagnosis in propositional logic. They took 516 proof states from the logs of a logic tutor used in an undergraduate Discrete Mathematics course. An LLM played the student and proposed next steps. Each step was labeled against a knowledge graph that enumerates every valid derivation for the 32 underlying problems. Seven LLM feedback agents then judged and commented on these steps under three feedback conditions, yielding 10,836 solution–feedback pairs. Feedback quality was assessed both automatically and by human raters.

The models almost always recognize optimal steps. However, they tend to reject sound alternatives and accept wrong steps, which are the cases where tailored guidance is most needed. These weaknesses appear across all models and regardless of the solution context provided. Correct diagnoses also often come with unhelpful feedback. The authors therefore recommend hybrid tutors in which KG-grounded models handle diagnosis and LLMs handle dialogue.

Key points

  • Data: 516 unique proof states drawn from 32 problems across five practice levels of a propositional logic tutor (Spring 2023). Step complexity ranges from 0 to 11 (mean 3.07).
  • Ground truth: for each problem, a knowledge graph built from a closed set of 15 inference rules covers all valid paths. This supports an optimal / valid-alternative / incorrect labeling, unlike binary-correctness benchmarks such as ProofWriter, FOLIO or ProntoQA.
  • Setup: seven models, three feedback conditions (Peer, Teacher, Judge), zero-shot prompting at temperature 0.0. Human evaluation covered 100 pairs per condition on four rubric dimensions.
  • Findings: accuracy is near ceiling on optimal steps, with systematic over-rejection of valid alternatives and over-validation of errors. Failures depend more on the model than on problem difficulty, and accurate labels do not guarantee actionable feedback.
  • Resources: code and data are released at github.com/tahreemm/BEA_2026. Stated limitations include a single institution, simulated rather than real students, and single-turn feedback only.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓North Carolina State UniversityRaleigh · United States14 works · 406 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksInformation Systems · 1,730 works

Cite

Cite as Tahreem Yasir, Wenbo Li, Sam Gilson, Sutapa Tithi, Xiaoyi Tian and Tiffany Barnes. 2026. Confirming Correct, Missing the Rest: LLM Tutoring Agents Struggle Where Feedback Matters Most. Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bea-1.56

@inproceedings{yasir2026confirming,
  title = {{Confirming Correct, Missing the Rest: LLM Tutoring Agents Struggle Where Feedback Matters Most}},
  author = {Tahreem Yasir, Wenbo Li, Sam Gilson, Sutapa Tithi, Xiaoyi Tian and Tiffany Barnes},
  booktitle = {Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.bea-1.56},
  url = {https://doi.org/10.18653/v1/2026.bea-1.56}
}

References

40 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Vincent Aleven, Bruce Mclaren, Jonathan Sewall, and Kenneth Koedinger. 2009. A New Paradigm for Intelligent Tutoring Systems: Example-Tracing Tutors. I. J. Artificial Intelligence in Education, 19:105–154. find
  2. Qinjin Jia, Jialin Cui, Ruijie Xi, Chengyuan Liu, Parvez Rashid, Ruochi Li, and Edward Gehringer. 2024. On assessing the faithfulness of llm-generated feedback on student assignments. In Proceedings of the 17th International Conference on Educational Data Mining, pages 491–499. find
  3. Tiffany Barnes and John Stamper. 2008. Toward Automatic Hint Generation for Logic Proof Tutoring Using Historical Student Data. pages 373–382. find
  4. PN Johnson-Laird and PC Wason. 1970. Insight into a logical relation. The Quarterly Journal of Experimental Psychology, 22(1):49–61. find
  5. Luca Benedetto, Giovanni Aradelli, Antonia Donvito, Alberto Lucchetti, Andrea Cappelli, and Paula Buttery. 2024. Using LLMs to simulate students’ responses to exam questions. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 11351–11368, Miami, Florida, USA. Association… find
  6. Kenneth R Koedinger and Vincent Aleven. 2007. Exploring the assistance dilemma in experiments with cognitive tutors. Educational psychology review, 19(3):239–264. find
  7. Conrad Borchers and Tianze Shou. 2025. Can large language models match tutoring system adaptivity? a benchmarking study. In International Conference on Artificial Intelligence in Education, pages 407–420. Springer. find
  8. Liuqiao Chen, Yan Huang, Zhenglin Jiang, and Ying Chen. 2025. Generative Feedback for Code Learning: A Study on LLM-Driven Formative Assessment. In Proceedings of the 2025 International Conference on AI-enabled Education, AIEE ’25, pages 253–259, New York, NY, USA. Association for Computing Machiner… find
  9. Cornelia Große and Alexander Renkl. 2006. Effects of multiple solution methods mathematics learning. Learning and Instruction, 16(2), 122-138. Learning and Instruction - LEARN INSTR, 16:122–138. find
  10. Shuchen Guo, Ehsan Latif, Yifan Zhou, Xuan Huang, and Xiaoming Zhai. 2024. Using Generative AI and Multi-Agents to Provide Automatic Feedback. arXiv preprint. ArXiv:2411.07407 [cs]. — [arXiv:2411.07407](https://arxiv.org/abs/2411.07407) find
  11. Adit Gupta, Jennifer Reddig, Tommaso Calo, Daniel Weitekamp, and Christopher J MacLellan. 2025. Beyond final answers: Evaluating large language models for math tutoring. In International Conference on Artificial Intelligence in Education, pages 323–337. Springer. find
  12. Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, Lucy Sun, Alex Wardle-Solano, Hannah Szabo, Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, and 16 others. 2024. FOLIO: Natur…
  13. Amogh Inamdar, Uzay Macar, Michel Vazirani, Michael Tarnow, Zarina Mustapha, Natalia Dittren, Sam Sadeh, Nakul Verma, and Ansaf Salleb-Aouissi. 2025. Logiclearner: A tool for the guided practice of propositional logic proofs. arXiv preprint arXiv:2503.19280. — [arXiv:2503.19280](https://arxiv.org/abs/2503.19280) find
  14. Kenneth R Koedinger, Albert T Corbett, and Charles Perfetti. 2012. The knowledge-learning-instruction framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive science, 36(5):757–798. find
  15. Rongxin Liu, Julianna Zhao, Benjamin Xu, Christopher Perez, Yuliia Zhukovets, and David Malan. 2025a. Improving AI in CS50: Leveraging Human Feedback for Better Learning. pages 715–721. find
  16. Rongxin Liu, Julianna Zhao, Benjamin Xu, Christopher Perez, Yuliia Zhukovets, and David J Malan. 2025b. Improving ai in cs50: Leveraging human feedback for better learning. In Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1, pages 715–721. find
  17. Xinyi Lu and Xu Wang. 2024. Generative students: Using LLM-simulated student profiles to support question item evaluation. In Proceedings of the Eleventh ACM Conference on Learning @ Scale (L@S ’24), pages 16–27, Atlanta, GA, USA. ACM. find
  18. Jakub Macina, Nico Daheim, Sankalan Chowdhury, Tanmay Sinha, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. 2023. MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems. In Findings of the Association for Computational Linguistics: EMNLP 2023, pa…
  19. Hamed Mahdavi, Pouria Mahdavinia, Samira Malek, Pegah Mohammadipour, Alireza Hashemi, Majid Daliri, Alireza Farhadi, Amir Khasahmadi, Niloofar Mireshghallah, and Vasant Honavar. 2025. Ref-Grader: Automated Grading of Mathematical Competition Proofs using Agentic Workflows. arXiv preprint. ArXiv:2510… — [arXiv:2510.09021](https://arxiv.org/abs/2510.09021) find
  20. Mehak Maniktala, Min Chi, and Tiffany Barnes. 2023. Enhancing a student productivity model for adaptive problem-solving assistance. User Modeling and User-Adapted Interaction, 33(1):159–188. find
  21. Amogh Mannekote, Adam Davies, Jina Kang, and Kristy Elizabeth Boyer. 2025. Can llms reliably simulate human learner actions? a simulation authoring framework for open-ended learning environments. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 29044–29052. find
  22. Kaushal Kumar Maurya, Kv Aditya Srivatsa, Kseniia Petukhova, and Ekaterina Kochmar. 2025. Unifying AI Tutor Evaluation: An Evaluation Taxonomy for Pedagogical Ability Assessment of LLM-Powered AI Tutors. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association…
  23. Tung Phung, Victor-Alexandru Pădurean, Anjali Singh, Christopher Brooks, José Cambronero, Sumit Gulwani, Adish Singla, and Gustavo Soares. 2024. Automating Human Tutor-Style Programming Feedback: Leveraging GPT-4 Tutor Model for Hint Generation and GPT-3.5 Student Model for Hint Validation. In Proc… find
  24. Jennifer M. Reddig, Arav Arora, and Christopher J. MacLellan. 2025. Generating In-Context, Personalized Feedback for Intelligent Tutors with Large Language Models. International Journal of Artificial Intelligence in Education, 35(6):3459–3500. find
  25. Bárbara Rodrigues, Rui Pinto, and Gil Gonçalves. 2025. A systematic literature review of ai-driven intelligent tutoring systems in engineering education: Emphasizing personalization, feedback, and student monitoring. IEEE Access, 13:190152–190177. find
  26. Abulhair Saparov and He He. 2023. Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought. arXiv preprint. ArXiv:2210.01240 [cs].
  27. Iulian Vlad Serban and 1 others. 2020. Automated personalized feedback improves learning gains in an intelligent tutoring system. In International Conference on Artificial Intelligence in Education, pages 140–146. Springer. Personalized hints improve learning gains; feedback from ZPD. find
  28. Preya Shabrina, Behrooz Mostafavi, Mark Abdelshiheed, Min Chi, and Tiffany Barnes. 2024. Investigating the impact of backward strategy learning in a logic tutor: Aiding subgoal learning towards improved problem solving. International Journal of Artificial Intelligence in Education, 34(3):825–861. find
  29. Shamini Shetye. 2024. An evaluation of khanmigo, a generative ai tool, as a computer-assisted language learning app. Studies in Applied Linguistics and TESOL, 24(1). find
  30. Yao Shi, Rongkeng Liang, and Yong Xu. 2025. Educationq: Evaluating llms’ teaching capabilities through multi-agent dialogue framework. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 32799–32828.
  31. John Stamper, Ruiwei Xiao, and Xinying Hou. 2024. Enhancing llm-based feedback: Insights from intelligent tutoring systems and the learning sciences. In International Conference on Artificial Intelligence in Education, pages 32–43. Springer. find
  32. Oyvind Tafjord, Bhavana Dalvi Mishra, and Peter Clark. 2020. Proofwriter: Generating implications, proofs, and abductive statements over natural language. arXiv preprint arXiv:2012.13048.
  33. Sutapa Dey Tithi, Arun Kumar Ramesh, Clara Di-Marco, Xiaoyi Tian, Nazia Alam, Kimia Fazeli, and Tiffany Barnes. 2025. The promise and limits of LLMs in constructing proofs and hints for logic problems in intelligent tutoring systems. arXiv preprint. ArXiv:2505.04736 [cs]. — [arXiv:2505.04736](https://arxiv.org/abs/2505.04736) find
  34. Kurt VanLehn. 2006. The behavior of tutoring systems. International journal of artificial intelligence in education, 16(3):227–265. find
  35. Kurt VanLehn. 2011. The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46(4):197–221. Step-based ITS achieve effect sizes of 0.75-0.80, comparable to human tutoring (0.79). find
  36. Devika Venugopalan, Ziwen Yan, Conrad Borchers, Jionghao Lin, and Vincent Aleven. 2025. Combining large language models with tutoring system intelligence: A case study in caregiver homework support. In Proceedings of the 15th International Learning Analytics and Knowledge Conference, pages 373–383. find
  37. Daniel Weitekamp, Erik Harpstead, and Ken R. Koedinger. 2020. An Interaction Design for Machine Teaching to Develop AI Tutors. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI ’20, pages 1–11, New York, NY, USA. Association for Computing Machinery. find
  38. Tahreem Yasir, Sutapa Dey Tithi, Benyamin Tabarsi, Dmitri Droujkov, Sam Gilson Yasitha Rajapaksha, Xiaoyi Tian, Arun Ramesh, Tiffany Barnes, and 1 others. 2026. When verification hurts: Asymmetric effects of multi-agent feedback in logic proof tutoring. arXiv preprint arXiv:2603.27076. — [arXiv:2603.27076](https://arxiv.org/abs/2603.27076) find
  39. Mohamed Zerkouk and Belkacem Chikhaoui. 2025. A comprehensive review of ai-based intelligent tutoring systems: Applications and challenges. arXiv preprint arXiv:2507.18882. ITS can improve student performance; CLT informs stepwise hint design. — [arXiv:2507.18882](https://arxiv.org/abs/2507.18882) find
  40. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yasir-2026-confirming-correct-missing-rest-llm/