Elamur Research⌕
☰
home / papers / 2023 / The BEA 2023 Shared Task on Generating AI Teacher Responses in Educational Dialogues

The BEA 2023 Shared Task on Generating AI Teacher Responses in Educational Dialogues

Paper · 2023 · Anaïs Tack, Ekaterina Kochmar, Zheng Yuan et al.

Summary

This paper reports on the first shared task that tests whether generative language models can act as AI teachers by producing the next teacher turn in a real teacher–student conversation. The organizers built the data from the Teacher-Student Chatroom Corpus (TSCC), which contains ESL tutoring chats. They cut each dialogue into context windows of at most 100 tokens that end in a teacher reply. To prevent reference responses from leaking between splits, they used a careful selection procedure based on the Hungarian algorithm. Eight teams competed on CodaLab using models ranging from GPT-2 and T5-base to GPT-3/GPT-4, LLaMA and Alpaca. Submissions were ranked automatically with BERTScore and DialogRPT. The three best then went through human comparison on three criteria: sounding like a teacher, understanding the student, and helping them. NAISTeacher won both rankings. It used GPT-3.5 Turbo with several prompts and picked the final output by DialogRPT reranking. The authors conclude that current automatic metrics capture pedagogical quality poorly, so better education-specific evaluation is needed.

Key points

  • Data: 7,047 candidate samples were extracted from TSCC with spaCy tokenization. The final split has 3,052 training samples (2,747 train / 305 held-out) and 273 test samples, with conflict checks to avoid nested-sample leakage.
  • Competition: It ran in two phases on CodaLab, development (March 24 – April 30, 2023) and evaluation (May 1–5, 2023). In the evaluation phase, 10 participants from eight teams made 19 valid submissions.
  • Evaluation protocol: Automatic scoring used BERTScore and DialogRPT. Human pairwise judgments followed the Tack and Piech (2022) AI teacher test and covered the top three systems plus the original teacher reference.
  • Winner: NAISTeacher (GPT-3.5 Turbo, prompt ensemble, DialogRPT-based response ranking) was ranked first by both automatic metrics and human raters.
  • Metric weaknesses: A trivial baseline that always answers "Hello" reached the best BERTScore precision and DialogRPT updown. Adding a "teacher:" prefix also shifted scores. Teams further noted that DialogRPT tends to favour full answers over scaffolding that guides the student.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States567 works · 335,190 citations🎓KU LeuvenLeuven · Belgium52 works · 8,121 citations🎓King's College LondonLondon · United Kingdom50 works · 1,852 citations🤝IMECLeuven · Belgium6 works · 1,917 citations

Also on the byline: Central University of Ecuador.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksInformation Systems · 1,888 works

🧭 Topics 1Research subjects and languages the work is about.Dialogue systems · 664 works

Cite

Cite as Anaïs Tack, Ekaterina Kochmar, Zheng Yuan, Serge Bibauw and Chris Piech. 2023. The BEA 2023 Shared Task on Generating AI Teacher Responses in Educational Dialogues. Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.bea-1.64 · arXiv 2306.06941

@inproceedings{tack2023bea,
  title = {{The BEA 2023 Shared Task on Generating AI Teacher Responses in Educational Dialogues}},
  author = {Anaïs Tack, Ekaterina Kochmar, Zheng Yuan, Serge Bibauw and Chris Piech},
  booktitle = {Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023)},
  year = {2023},
  doi = {10.18653/v1/2023.bea-1.64},
  eprint = {2306.06941},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.bea-1.64}
}

References

30 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adaeze Adigwe and Zheng Yuan. 2023. The ADAIO System at the BEA-2023 Shared Task: Shared Task Generating AI Teacher Responses in Educational Dialogues. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications, page to appear, Toronto, Canada. Association for… find
  2. Alexis Baladón, Ignacio Sastre, Luis Chiruzzo, and Aiala Rosá. 2023. RETUYT-InCo at BEA 2023 Shared Task: Tuning Open-Source LLMs for Generating Teacher Responses. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications, page to appear, Toronto, Canada. Ass… find
  3. Serge Bibauw, Wim Van den Noortgate, Thomas François, and Piet Desmet. 2022. Dialogue systems for language learning: A meta-analysis. Language Learning & Technology, 26(1). find
  4. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora De… — On the Opportunities and Risks of Foundation Models
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  6. Andrew Caines, Helen Yannakoudakis, Helen Allen, Pascual Pérez-Paredes, Bill Byrne, and Paula Buttery. 2022. The Teacher-Student Chatroom Corpus version 2: More lessons, new annotation, automatic detection of sequence shifts. In Proceedings of the 11th Workshop on NLP for Computer Assisted Language… find
  7. Andrew Caines, Helen Yannakoudakis, Helena Edmondson, Helen Allen, Pascual Pérez-Paredes, Bill Byrne, and Paula Buttery. 2020. The teacher-student chatroom corpus. In Proceedings of the 9th Workshop on NLP for Computer Assisted Language Learning, pages 10–20, Gothenburg, Sweden. LiU Electronic Press…
  8. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. find
  9. Xiang Gao, Yizhe Zhang, Michel Galley, Chris Brockett, and Bill Dolan. 2020a. Dialogue response ranking training with large-scale human feedback data. arXiv preprint arXiv:2009.06978. find
  10. Xiang Gao, Yizhe Zhang, Michel Galley, Chris Brockett, and Bill Dolan. 2020b. Dialogue Response Ranking Training with Large-Scale Human Feedback Data. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 386–395, Online. Association for Computation… find
  11. Yann Hicke, Abhishek Masand, Wentao Guo, and Tushaar Gangavarapu. 2023. Assessing the efficacy of large language models in generating accurate teacher responses. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications, page to appear, Toronto, Canada. Assoc… find
  12. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
  13. Thomas Huber, Christina Niklaus, and Siegfried Handschuh. 2023. Enhancing Educational Dialogues: A Reinforcement Learning Approach for Generating AI Teacher Responses. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications, page to appear, Toronto, Canada.… find
  14. H. W. Kuhn. 1955. The Hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(1-2):83–97. find
  15. Margaret Li, Jason Weston, and Stephen Roller. 2019. ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons. find
  16. Amin Omidvar and Aijun An. 2023. Empowering Conversational Agents using Semantic In-Context Learning. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications, page to appear, Toronto, Canada. Association for Computational Linguistics. find
  17. Adrien Pavao, Isabelle Guyon, Anne-Catherine Letour-nel, Xavier Baró, Hugo Escalante, Sergio Escalera, Tyler Thomas, and Zhen Xu. 2022. CodaLab Competitions: An open source platform to organize scientific challenges. Technical report. find
  18. Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021. MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers. In Advances in Neural Information Processing Systems 34 Pre-Proceedings (NeurIPS 2021), pag…
  19. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  20. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551.
  21. Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. 2022. Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization…
  22. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. find
  23. Anaïs Tack and Chris Piech. 2022. The AI Teacher Test: Measuring the Pedagogical Ability of Blender and GPT-3 in Educational Dialogues. In Proceedings of the 15th International Conference on Educational Data Mining, volume 15, pages 522–529, Durham, United Kingdom. International Educational Data Min…
  24. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Stanford Alpaca: An instruction-following LLaMA model. find
  25. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  26. Justin Vasselli, Christopher Vasselli, Adam Nohejl, and Taro Watanabe. 2023. NAISTeacher: A Prompt and Rerank Approach to Generating Teacher Utterances in Educational Dialogues. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications, page to appear, Toront… find
  27. Sebastian Wollny, Jan Schneider, Daniele Di Mitri, Joshua Weidlich, Marc Rittberger, and Hendrik Drachsler. 2021. Are We There Yet? - A Systematic Literature Review on Chatbots in Education. Frontiers in Artificial Intelligence, 4:654924. find
  28. Yi-Ting Yeh, Maxine Eskenazi, and Shikib Mehri. 2021. A Comprehensive Assessment of Dialog Evaluation Metrics. In The First Workshop on Evaluations and Assessments of Neural Conversation Systems, pages 15–33, Online. Association for Computational Linguistics.
  29. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In International Conference on Learning Representations. find
  30. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2019. Dialogpt: Large-scale generative pre-training for conversational response generation. arXiv preprint arXiv:1911.00536.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tack-2023-bea-2023-shared-task-generating/