Elamur Research⌕
☰
home / papers / 2024 / BIPED: Pedagogically Informed Tutoring System for ESL Education

BIPED: Pedagogically Informed Tutoring System for ESL Education

Paper · 2024 · Soonwoo Kwon, Sojung Kim, Minju Park et al.

Summary

LLM-based conversational tutors for English learners usually cover only toy topics or rely on a small set of coarse teaching moves. This paper aims for tutors that adapt richer pedagogical strategies to each learner. The authors collected BIPED, a bilingual dataset of real one-on-one human tutoring sessions built around reading an authentic news article, with work on vocabulary, grammar and cultural background. After the sessions, they analysed the transcripts and derived a taxonomy of 34 tutor acts and 9 student acts. They then annotated every utterance with these acts, its content, and whether student answers were correct.

On top of this data they built two tutors that first select a tutor act and then write a reply matching it:

  • one uses GPT-4 with BIPED examples supplied in context;
  • one fine-tunes SOLAR-KO 10.7B, a model trained on Korean text, with multi-task instruction tuning.

The fine-tuned model came closest to human tutors' wording and reply length, and it used the widest range of teaching acts. The work offers a resource and a recipe for more pedagogically grounded ESL tutoring.

Key points

  • Dataset scale: BIPED has 112 fully annotated sessions with 4 experienced ESL tutors and 60 students. Each student had three 30-minute audio sessions, transcribed with Whisper.
  • Session statistics: sessions average about 105.7 turns. Tutors use about 31.7 words per turn and students about 8.2.
  • Evidence of teaching value: the mean normalized learning gain was 0.089, and student satisfaction ratings were about 4.6–4.7 out of 5.
  • Fine-tuned model results:
    • Tutor-act accuracy was 0.259 across 34 classes.
    • It had the lowest invariability (5.273), meaning more varied act choices.
    • It had the best sBLEU (15.874) and BERTScore (0.716), though slightly lower embedding similarity than GPT-3.5 1-shot.
    • Adding the extra instruction tasks beat both the ablation and the plain conditional-generation baselines.
  • Limitations: evaluation is static rather than interactive with real students, and models larger than 10.7B (e.g., LLaMa 70b) were not tested.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Korea Advanced Institute of Science and TechnologyDaejeon · South Korea59 works · 3,320 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksInformation Systems · 1,888 works

🧭 Topics 1Research subjects and languages the work is about.Grammatical error correction & learner language · 165 works

Cite

Cite as Soonwoo Kwon, Sojung Kim, Minju Park, Seunghyun Lee and Kyuseok Kim. 2024. BIPED: Pedagogically Informed Tutoring System for ESL Education. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.acl-long.186 · arXiv 2406.03486

@inproceedings{kwon2024biped,
  title = {{BIPED: Pedagogically Informed Tutoring System for ESL Education}},
  author = {Soonwoo Kwon, Sojung Kim, Minju Park, Seunghyun Lee and Kyuseok Kim},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.186},
  eprint = {2406.03486},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2024.acl-long.186}
}

References

37 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R Abdelghani, YH Wang, X Yuan, T Wang, H Sauzéon, and PY Oudeyer. 2022. Gpt-3-driven pedagogical agents for training children’s curious question-asking skills. arXiv, 2211. find
  2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv:2303.08774.
  3. Sterling Alic, Dorottya Demszky, Zid Mancenido, Jing Liu, Heather Hill, and Dan Jurafsky. 2022. Computationally identifying funneling and focusing questions in classroom discourse. arXiv:2208.04715. find
  4. H Douglas Brown. 2014. Principles of language learning and teaching: A course in second language acquisition. Pearson. find
  5. H Douglas Brown and Heekyeong Lee. 2015. Teaching by Principles: An Interactive Approach to Language Pedagogy. P. Ed Australia. find
  6. Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. Multiwoz–a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. arXiv:1810.00278.
  7. Andrew Caines, Helen Yannakoudakis, Helen Allen, Pascual Pérez-Paredes, Bill Byrne, and Paula Buttery. 2022. The teacher-student chatroom corpus version 2: more lessons, new annotation, automatic detection of sequence shifts. In Swedish Language Technology Conference and NLP4CALL, pages 23–35. find
  8. Dorottya Demszky and Heather Hill. 2022. The ncte transcripts: A dataset of elementary math classroom transcripts. arXiv:2211.11772.
  9. Benedict Du Boulay and Rosemary Luckin. 2016. Modelling human teaching tactics and strategies for tutoring systems: 14 years on. International Journal of Artificial Intelligence in Education, 26:393–404. find
  10. Richard Hake. 1998. Interactive-engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses. American Journal of Physics - AMER J PHYS, 66. find
  11. Jin-Xia Huang, Yohan Lee, and Oh-Woog Kwon. 2022. Direct: Toward dialogue-based reading comprehension tutoring. IEEE Access, 11:8978–8987. find
  12. Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. 2023. Chatgpt for good? on opportunities and challenges of large language models for education. Learning and individual difference…
  13. L. Junbum. 2024. Solar-ko-10.7b. find
  14. Seungjun Lee, Yoonna Jang, Chanjun Park, Jungseob Lee, Jaehyung Seo, Hyeonseok Moon, Sugyeong Eo, Seounghoon Lee, Bernardo Yahya, and Heuiseok Lim. 2023. PEEP-talk: A situational dialogue-based chatbot for English education. In Proceedings of the 61st Annual Meeting of the Association for Computatio…
  15. Jionghao Lin, Wei Tan, Ngoc Dang Nguyen, David Lang, Lan Du, Wray Buntine, Richard Beare, Guanliang Chen, and Dragan Gašević. 2023. Robust educational dialogue act classifiers with low-resource and imbalanced datasets. In International Conference on Artificial Intelligence in Education, pages 114–1… find
  16. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations. find
  17. Jakub Macina, Nico Daheim, Sankalan Pal Chowdhury, Tanmay Sinha, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. 2023a. Mathdial: A dialogue tutoring dataset with rich pedagogical properties grounded in math reasoning problems. arXiv:2305.14536.
  18. Jakub Macina, Nico Daheim, Lingzhi Wang, Tanmay Sinha, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. 2023b. Opportunities and challenges in neural dialog tutoring. arXiv:2301.09919.
  19. Jakub Macina, Nico Daheim, Lingzhi Wang, Tanmay Sinha, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. 2023c. Opportunities and challenges in neural dialog tutoring. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2357–2372, Dubr…
  20. Li Nguyen, Zheng Yuan, and Graham Seed. 2022. Building educational technologies for code-switching: Current practices, difficulties and future directions. Languages, 7(3):220. find
  21. Minju Park, Sojung Kim, Seunghyun Lee, Soonwoo Kwon, and Kyuseok Kim. 2024. Empowering personalized learning through a conversation-based tutoring system with student modeling. In Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems, CHI EA ’24, New York, NY, USA. Asso… find
  22. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Brussels, Belgium. Association for Computational Linguistics.
  23. Kun Qian, Ryan Shea, Yu Li, Luke Kutszik Fryer, and Zhou Yu. 2023. User adaptive language learning chatbots with a curriculum. arXiv:2304.05489. find
  24. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pages 28492–28518. PMLR.
  25. Tulika Saha, Aditya Patra, Sriparna Saha, and Pushpak Bhattacharyya. 2020. Towards emotion-aided multimodal dialogue act classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4361–4372. find
  26. Borhan Samei, Vasile Rus, Benjamin D. Nye, and Donald M. Morrison. 2015. Hierarchical dialogue act classification in online tutoring sessions. In Educational Data Mining. find
  27. Katherine Stasaski, Kimberly Kao, and Marti A Hearst. 2020. Cima: A large open access dialogue dataset for tutoring. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 52–64.
  28. Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational linguistics, 26(3):339–373.
  29. Abhijit Suresh, Jennifer Jacobs, Charis Harty, Margaret Perkoff, James H Martin, and Tamara Sumner. 2022. The talkmoves dataset: K-12 mathematics lesson transcripts annotated for teacher and student discursive moves. arXiv:2204.09652.
  30. Anaïs Tack and Chris Piech. 2022. The ai teacher test: Measuring the pedagogical ability of blender and gpt-3 in educational dialogues. arXiv:2205.07540.
  31. Wei Tan, Jionghao Lin, David Lang, Guanliang Chen, Dragan Gašević, Lan Du, and Wray Buntine. 2023. Does informativeness matter? active learning for educational dialogue act classification. In International Conference on Artificial Intelligence in Education, pages 176–188. Springer. find
  32. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  33. Steve Walsh. 2006. Investigating classroom discourse. Routledge. find
  34. Steve Walsh. 2013. Classroom discourse and teacher development. Edinburgh University Press. find
  35. Lingzhi Wang, Mrinmaya Sachan, Xingshan Zeng, and Kam-Fai Wong. 2023a. Strategize before teaching: A conversational tutoring system with pedagogy self-distillation. arXiv:2302.13496. find
  36. Lingzhi Wang, Mrinmaya Sachan, Xingshan Zeng, and Kam-Fai Wong. 2023b. Strategize before teaching: A conversational tutoring system with pedagogy self-distillation. In Findings of the Association for Computational Linguistics: EACL 2023, pages 2268– 2274, Dubrovnik, Croatia. Association for Computat… find
  37. Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kwon-2024-biped-pedagogically-informed-tutoring-system/