Elamur Research⌕
☰
home / papers / 2025 / Findings of the BEA 2025 Shared Task on Pedagogical Ability Assessment of AI-powered Tutors

Findings of the BEA 2025 Shared Task on Pedagogical Ability Assessment of AI-powered Tutors

Paper · 2025 · Ekaterina Kochmar, Kaushal Kumar Maurya, Kseniia Petukhova et al.

Summary

The BEA 2025 shared task asks whether LLM-based tutors respond to students in pedagogically sound ways. Earlier work used inconsistent criteria, so the organizers built a common benchmark from math tutoring dialogues drawn from MathDial and Bridge. Each dialogue ends with a student error. It is paired with candidate replies from seven LLMs (GPT-4, Gemini, Sonnet, Mistral, Llama-3.1-8B, Llama-3.1-405B, Phi-3) and from human expert or novice tutors.

Each reply is rated on a three-level scale (No / To some extent / Yes) for four dimensions adopted from Maurya et al. (2025): whether the tutor recognizes that an error occurred, whether it pinpoints where the error is, whether it offers the student guidance, and whether it leaves the student with a clear, actionable next step. These dimensions are grounded in learning-science principles. A fifth track asks systems to recognize which tutor wrote a response.

More than 50 teams took part, using any external data and either classical ML or LLM approaches. Scores show the problem is far from solved. The top macro F1 is 71.81 for mistake identification and only 58.34 for guidance, while tutor identification reached 96.98. All data are publicly released, giving the field a reusable standard for evaluating AI tutors.

Key points

  • Data split: the development set covers 300 dialogues with 2,476 annotated tutor responses, about 75% from MathDial and 25% from Bridge. The test set has 191 dialogues with 1,547 responses, released without labels.
  • Tracks: five tracks ran on CodaBench, each allowing up to 5 submissions per team. Four track pedagogical dimensions as 3-class problems, and the fifth is a 9-class tutor identification task.
  • Best results:
    • Mistake identification: macro F1 71.81.
    • Providing guidance: macro F1 58.34, the lowest of the four pedagogical tracks.
    • Tutor identification: F1 96.98.
  • Per-tutor identification accuracy: averaged across submissions, Llama-3.1-8B was the hardest tutor to identify (61.4%). Phi-3 (79.5%) and the Expert tutor (79.1%) were the easiest.
  • Limitations: the task covers only math and only the mistake-remediation scenario, and it judges replies within a short context window rather than the full dialogue.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Mohamed bin Zayed University of Artificial IntelligenceAbu Dhabi · United Arab Emirates145 works · 6,811 citations🎓Nara Institute of Science and TechnologyIkoma · Japan49 works · 4,601 citations🎓KU LeuvenLeuven · Belgium52 works · 8,121 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksInformation Systems · 1,888 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,394 works

Cite

Cite as Ekaterina Kochmar, Kaushal Kumar Maurya, Kseniia Petukhova, KV Aditya Srivatsa, Anaïs Tack and Justin Vasselli. 2025. Findings of the BEA 2025 Shared Task on Pedagogical Ability Assessment of AI-powered Tutors. Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2025).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.bea-1.77 · arXiv 2507.10579

@inproceedings{kochmar2025findings,
  title = {{Findings of the BEA 2025 Shared Task on Pedagogical Ability Assessment of AI-powered Tutors}},
  author = {Ekaterina Kochmar, Kaushal Kumar Maurya, Kseniia Petukhova, KV Aditya Srivatsa, Anaïs Tack and Justin Vasselli},
  booktitle = {Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2025)},
  year = {2025},
  doi = {10.18653/v1/2025.bea-1.77},
  eprint = {2507.10579},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2025.bea-1.77}
}

References

7 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, and 1 others. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219.
  2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774.
  3. Jiyuan An, Xiang Fu, Bo Liu, Xuquan Zong, Cunliang Kong, Shuliang Liu, Shuo Wang, Zhenghao Liu, Liner Yang, Hanghang Fan, and Erhong Yang. 2025. BLCU-ICALL at BEA 2025 Shared Task: Multi-Strategy Evaluation of AI Tutors. In Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educa… find
  4. Anthropic. 2024. The Claude 3 Model Family: Opus, Sonnet, Haiku. find
  5. Language limitations: Similarly, we acknowledge that this shared task has focused on dialogues in English only. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, and 1 others. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609.
  6. Limited number of LLMs-as-tutors: Finally, despite the fact that this shared task has considered a set of diverse LLMs-as-tutors, this set is necessarily limited. find
  7. Souvik Bhattacharyya, Billodal Roy, Niranjan M Kumar, and Pranav Gupta. 2025. LexiLogic at BEA 2025 Shared Task: Fine-tuning Transformer Language Models for the Pedagogical Skill Evaluation of LLM-based tutors. In Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational App… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kochmar-2025-findings-bea-2025-shared-task/