Elamur Research⌕
☰
home / papers / 2026 / Towards Reward Modeling for AI Tutors in Math Mistake Remediation

Towards Reward Modeling for AI Tutors in Math Mistake Remediation

Paper · 2026 · Kseniia Petukhova and Ekaterina Kochmar

Summary

Judging whether an AI tutor's reply is pedagogically sound is difficult. Reference-based NLG metrics cannot tell whether a response notices the student's error, guides their reasoning, or simply gives away the solution. Adding up MRBench's eight per-dimension labels with equal weights also produces many ties and treats every aspect as equally important.

For math mistake remediation, the authors collect human pairwise preferences over MRBench tutor responses and use them to build a ranked hierarchy of pedagogical aspects. Inspired by the minimal-revision approach of CLAIR, they then generate synthetic response pairs that differ only along one targeted aspect, such as mistake identification and location, scaffolding, actionability, or clarity and coherence. They train Bradley-Terry reward models on a Qwen2.5-0.5B backbone using weighted-sum rankings derived from MRBench, the synthetic pairs, and mixtures of both.

Synthetic data alone reaches 0.69 pairwise accuracy on human preferences, and the best mixture reaches 0.74, outperforming much larger general-purpose reward models. The result is a compact, openly released tool for evaluating and aligning tutoring LLMs.

Key points

  • MRBench (192 dialogs, 1,596 human and LLM tutor responses built on MathDial and Bridge) is annotated on eight dimensions. Equal-weight summing of these labels, as in Macina et al. (2025), often produces tied scores.
  • Human preference annotation yielded a priority order of aspects. Factuality and non-contradiction come first, and clarity and coherence rank lowest. This hierarchy guided both LLM-based annotation and the design of the synthetic contrastive pairs.
  • On the human preference test set, the Qwen2.5-0.5B Bradley-Terry model scores 0.74 (weighted-sum plus targeted synthetic groups) and 0.69 (synthetic data only).
  • The external baselines score lower: Skywork-Reward-V2-Llama-3.1-8B reaches 0.64, RewardAnything-8B-v1 reaches 0.68, and the model of Macina et al. (2025) reaches 0.69.
  • The released resources include the human preference annotations, the synthetic augmentation dataset, the reward model, and the code (GitHub: Kpetyxova/Towards_Reward_Modeling_for_Tutors). The authors note that the method could transfer to other domains.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksSocial Sciences · 2,009 worksInformation Systems · 1,888 worksEducation · 221 works

Cite

Cite as Kseniia Petukhova and Ekaterina Kochmar. 2026. Towards Reward Modeling for AI Tutors in Math Mistake Remediation. Proceedings of the Fifteenth Language Resources and Evaluation Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/5i2e38498j4m · arXiv 2603.24375

@inproceedings{petukhova2026reward,
  title = {{Towards Reward Modeling for AI Tutors in Math Mistake Remediation}},
  author = {Kseniia Petukhova and Ekaterina Kochmar},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/5i2e38498j4m},
  eprint = {2603.24375},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.63317/5i2e38498j4m}
}

References

24 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Benjamin S Bloom. 1984. The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational researcher, 13(6):4–16. find
  2. Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345. find
  3. Lin Chin-Yew. 2004. Rouge: A package for automatic evaluation of summaries. In Proceedings of the Workshop on Text Summarization Branches Out, 2004.
  4. Karel D’Oosterlinck, Winnie Xu, Chris Develder, Thomas Demeester, Amanpreet Singh, Christopher Potts, Douwe Kiela, and Shikib Mehri. 2025. Anchored preference optimization and contrastive revisions: Addressing underspecification in alignment. Transactions of the Association for Computational Linguis…
  5. Xiang Gao, Yizhe Zhang, Michel Galley, Chris Brockett, and Bill Dolan. 2020. Dialogue response ranking training with large-scale human feedback data. arXiv preprint arXiv:2009.06978. find
  6. Ekaterina Kochmar, Kaushal Kumar Maurya, Kseniia Petukhova, KV Srivatsa, Anaïs Tack, and Justin Vasselli. 2025. Findings of the BEA 2025 shared task on pedagogical ability assessment of AI-powered tutors. arXiv preprint arXiv:2507.10579.
  7. Chris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, Yang Liu, and Yahui Zhou. 2025. Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy. arXiv preprint arXiv:2507.01352.
  8. Rongxin Liu, Carter Zenke, Charlie Liu, Andrew Holmes, Patrick Thornton, and David J Malan. 2024. Teaching CS50 with AI: leveraging generative artificial intelligence in computer science education. In Proceedings of the 55th ACM technical symposium on computer science education V. 1, pages 750–756. find
  9. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG evaluation using GPT-4 with better human alignment (2023). arXiv preprint arXiv:2303.16634, 12.
  10. Jakub Macina, Nico Daheim, Sankalan Pal Chowdhury, Tanmay Sinha, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. 2023. Mathdial: A dialogue tutoring dataset with rich pedagogical properties grounded in math reasoning problems. arXiv preprint arXiv:2305.14536.
  11. Jakub Macina, Nico Daheim, Ido Hakimi, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. 2025. Mathtutorbench: A benchmark for measuring open-ended pedagogical capabilities of llm tutors. arXiv preprint arXiv:2502.18940.
  12. Kaushal Kumar Maurya, Kv Aditya Srivatsa, Kseniia Petukhova, and Ekaterina Kochmar. 2025. Unifying AI tutor evaluation: An evaluation taxonomy for pedagogical ability assessment of LLM-powered AI tutors. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association…
  13. Alizée Pace, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. 2024. West-of-n: Synthetic preference generation for improved reward modeling. arXiv preprint arXiv:2401.12086.
  14. Sankalan Pal Chowdhury, Vilém Zouhar, and Mrinmaya Sachan. 2024. AutoTutor meets large language models: A language model tutor with rich pedagogy and guardrails. In Proceedings of the Eleventh ACM Conference on Learning@ Scale, pages 5–15. 2025. RewardAnything: Generalizable Principle-Following Rewa…
  15. Maja Popović. 2017. chrF++: words helping character n-grams. In Proceedings of the second conference on machine translation, pages 612– 618.
  16. Matt Post. 2018. A call for clarity in reporting BLEU scores. arXiv preprint arXiv:1804.08771.
  17. Jiaming Shen, Ran Xu, Yennie Jun, Zhen Qin, Tianqi Liu, Carl Yang, Yi Liang, Simon Baumgartner, and Michael Bendersky. 2024. Boosting reward model with preference-conditional multi-aspect synthetic data generation. arXiv preprint arXiv:2407.16008.
  18. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295. find
  19. Rose E Wang, Qingyang Zhang, Carly Robinson, Susanna Loeb, and Dorottya Demszky. 2023. Bridging the novice-expert gap via models of decision-making: A case study on remediating math mistakes. arXiv preprint arXiv:2310.10648. find
  20. Sebastian Wollny, Jan Schneider, Daniele Di Mitri, Joshua Weidlich, Marc Rittberger, and Hendrik Drachsler. 2021. Are we there yet?-A systematic literature review on chatbots in education. Frontiers in artificial intelligence, 4:654924. find
  21. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai D… find
  22. Zhuohao Yu, Jiali Zeng, Weizheng Gu, Yidong Wang, Jindong Wang, Fandong Meng, Jie Zhou, Yue Zhang, Shikun Zhang, and Wei Ye. 11. Language Resource References Macina, Jakub and Daheim, Nico and Chowdhury, Sankalan Pal and Sinha, Tanmay and Kapur, Manu and Gurevych, Iryna and Sachan, Mrinmaya. 2023. M… — MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problems
  23. Maurya, Kaushal Kumar and Srivatsa, Kv Aditya and Petukhova, Kseniia and Kochmar, Ekaterina. 2025. Unifying AI Tutor Evaluation: An Evaluation Taxonomy for Pedagogical Ability Assessment of LLM-Powered AI Tutors. Association for Computational Linguistics.
  24. Wang, Rose E and Zhang, Qingyang and Robinson, Carly and Loeb, Susanna and Demszky, Dorottya. 2023. Bridging the novice-expert gap via models of decision-making: A case study on remediating math mistakes. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/petukhova-2026-reward-modeling-ai-tutors-math/