Elamur Research⌕
☰
home / papers / 2024 / A Conversational Intelligent Tutoring System for Improving English Proficiency of Non-Native Speakers via Debriefing of Online Meeting Transcriptions

A Conversational Intelligent Tutoring System for Improving English Proficiency of Non-Native Speakers via Debriefing of Online Meeting Transcriptions

Paper · 2024 · Juan Antonio Pérez-Ortiz, Miquel Esplà-Gomis, Víctor M. Sánchez-Cartagena et al.

Summary

DeMINT is a work-in-progress tutoring chatbot for non-native English speakers. It turns their own online meetings into language lessons. Professionals who use English as a lingua franca seldom get constructive feedback on their mistakes, and human tutors are costly and hard to schedule. The system therefore runs an offline pipeline over meeting recordings before any conversation starts:

  • speaker diarization with pyannote.audio,
  • transcription with a speech-to-text model fine-tuned to keep grammatical errors,
  • error correction and annotation,
  • error explanation backed by grammar resources.

A chatbot built on pre-trained LLMs then talks the learner through their mistakes. It uses in-context learning, retrieval from external memory, knowledge tracing and a fine-tuned "empathetic teacher" model.

A key engineering contribution is the error-preserving recognizer. Off-the-shelf Whisper tends to silently fix learner grammar, so the authors built a speech corpus of ungrammatical sentences and adapted Whisper with LoRA. A prototype was piloted with L1-Spanish/L2-English students. Code and models are open, which matters because few open conversational tutoring systems exist for L2 learning.

Key points

  • The speech training set has 32,000 utterances: 28,592 from C4 200M, 814 real COREFL recordings, and 2,594 COREFL essays voiced with StyleTTS2 in multiple voices. Validation and test sets have 1,000 utterances each, and clean C4 200M sentences were mixed in.
  • whisper-large-v3 and distil-large-v3 were fine-tuned with LoRA (r=16, α=32, on q_proj/v_proj). On the authors' own error-containing test set, WER fell from 41.82 to 31.47 for the full model and from 39.48 to 38.81 for the distilled one. The distilled model is the one deployed.
  • Grammatical errors are annotated with ERRANT, which uses a linguistically extended Levenshtein alignment plus rule-based edit labels. The annotations are stored in M2 format inside a shared JSON schema.
  • The teacher model is Llama-3.1-8B, tuned with 8-bit QLoRA (r=8, α=16) via LLaMA-Factory on one A100.
  • The models are released as Transducens/error-preserving-whisper (and its distilled variant), with code at github.com/transducens/demint. The current scope is grammar only, not vocabulary or pronunciation.

Interesting ideas

  • I1. Strong internal language models quietly "normalize" ungrammatical input, so the authors treat faithfulness to the errors as an explicit training and evaluation goal.
  • I2. The training data mixes synthetic corruptions with their clean counterparts, so the model does not learn to expect errors everywhere.
  • I3. Text-only data becomes richer training material by rendering each sentence through many synthetic voices, multiplying the variants cheaply.
  • I4. The fine-tuned recognizer is scored on its target test set and on several standard benchmarks. This exposes trade-offs, such as worse WER on Peoples Speech, alongside the gains.
  • I5. Differences between two texts are aligned with a linguistically aware edit distance, and each edit gets a rule-based type label.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): Add a check that the tagger and parser analyse learner or noisy sentences as written, rather than producing a tree for the "corrected" sentence the model expected.
  • I2 (learn): When generating corrupted silver data to induce repair rules, keep paired clean sentences as negative controls so the rules do not fire on correct text.
  • I3 (learn): In the same spirit, render one underlying meaning graph through several surface paraphrases to stress-test whether parsers agree on the frames.
  • I4 (learn): Report each training step against both the target data and unrelated control treebanks, to catch regressions hidden by in-domain improvement.
  • I5 (back): Compare compressor output with the original through typed edits (agreement, tense, determiner) rather than a single score, making reconstruction failures explainable.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AlicanteAlicante · Spain21 works · 1,098 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksDialogue systems · 624 works

Cite

Cite as Juan Antonio Pérez-Ortiz, Miquel Esplà-Gomis, Víctor M. Sánchez-Cartagena, Felipe Sánchez-Martínez, Roman Chernysh, Gabriel Mora-Rodríguez and Lev Berezhnoy. 2024. A Conversational Intelligent Tutoring System for Improving English Proficiency of Non-Native Speakers via Debriefing of Online Meeting Transcriptions. Proceedings of the 13th Workshop on Natural Language Processing for Computer Assisted Language Learning.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{perezortiz2024conversational,
  title = {{A Conversational Intelligent Tutoring System for Improving English Proficiency of Non-Native Speakers via Debriefing of Online Meeting Transcriptions}},
  author = {Juan Antonio Pérez-Ortiz, Miquel Esplà-Gomis, Víctor M. Sánchez-Cartagena, Felipe Sánchez-Martínez, Roman Chernysh, Gabriel Mora-Rodríguez and Lev Berezhnoy},
  booktitle = {Proceedings of the 13th Workshop on Natural Language Processing for Computer Assisted Language Learning},
  year = {2024},
  doi = {10.3384/ecp211014},
  url = {https://doi.org/10.3384/ecp211014}
}

References

56 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nathalie Aichhorn and Jonas Puck. 2017. “I just don’t feel comfortable speaking English”: Foreign language anxiety as a catalyst for spoken-language barriers in MNCs. International Business Review, 26(4):749–763. find
  2. Tri Dao. 2023. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv:2307.08691 [cs.LG] . ↗
  3. Serge Bibauw, Thomas François, and Piet Desmet. 2019. Discussing with a computer to practice a foreign language: research synthesis and conceptual framework of dialogue-based CALL. Computer Assisted Language Learning, 32(8):827–877. ↗
  4. Dorottya Demszky, Jing Liu, Zid Mancenido, Julie Cohen, Heather Hill, Dan Jurafsky, and Tatsunori Hashimoto. 2021. Measuring Conversational Uptake: A Case Study on Student-Teacher Interactions. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL). ↗
  5. Hervé Bredin. 2023. pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe. In Proc. INTERSPEECH 2023. find
  6. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient fine-tuning of quantized LLMs. In Advances in Neural Information Processing Systems, volume 36, pages 10088–10115. Curran Associates, Inc. find
  7. Christopher Bryant, Mariano Felice, Øistein E. Andersen, and Ted Briscoe. 2019. The BEA-2019 shared task on grammatical error correction. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 52–75, Florence, Italy. Association for Computational Linguistics.
  8. Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2024. The Faiss library. arXiv:2401.08281 [cs.LG] . ↗
  9. Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017. Automatic annotation and evaluation of error types for grammatical error correction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 793–805, Vancouver, Canada. Association for Computational Linguistics.
  10. Jinming Du and Ben Kei Daniel. 2024. Transforming language education: A systematic review of AI-powered chatbots for English as a foreign language speaking practice. Computers and Education: Artificial Intelligence, 6:100230. ↗
  11. Christopher Bryant, Zheng Yuan, Muhammad Reza Qorib, Hannan Cao, Hwee Tou Ng, and Ted Briscoe. 2023. Grammatical Error Correction: A Survey of the State of the Art. Computational Linguistics, 49(3):643–701. ↗
  12. Yuejiao Fei, Leyang Cui, Sen Yang, Wai Lam, Zhenzhong Lan, and Shuming Shi. 2023. Enhancing grammatical error correction systems with explanations. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7489–7501, Toronto, Canada. Association for Computational Linguistics. find
  13. Andrew Caines, Helen Yannakoudakis, Helen Allen, Pascual Pérez-Paredes, Bill Byrne, and Paula Buttery. 2022. The Teacher-Student Chatroom Corpus version 2: more lessons, new annotation, automatic detection of sequence shifts. In Proceedings of the 11th Workshop on NLP for Computer Assisted Language Learning, pages 23–35, Louvain-la-Neuve, Belgium. LiU Electronic Press. find
  14. Mariano Felice, Christopher Bryant, and Ted Briscoe. 2016. Automatic extraction of learner errors in ESL sentences using linguistically enhanced alignments. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 825–835, Osaka, Japan. ↗
  15. The COLING 2016 Organizing Committee. find
  16. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  17. Edresson Casanova, Kelly Davis, Eren Gölge, Görkem Göknar, Iulian Gulea, Logan Hart, Aya Aljafari, Joshua Meyer, Reuben Morais, Samuel Olayemi, and Julian Weber. 2024. XTTS: a massively multilingual zero-shot text-to-speech model. arXiv:2406.04904 [eess.AS] . ↗
  18. Weijiao Huang, Khe Foon Hew, and Luke K. Fryer. 2022. Chatbots for language learning: Are they really useful? a systematic review of chatbot-supported language learning. Journal of Computer Assisted Learning, 38(1):237–257. ↗
  19. Anna Izabela Cislowska and Beatriz Peña Acuña. 2024. Integration of chatbots in additional language education: A systematic review. European Journal of Educational Research, 13(4):1607–1625. ↗
  20. Jaeho Jeon. 2024. Exploring AI chatbot affordances in the EFL classroom: young learners’ experiences and perspectives. Computer Assisted Language Learning, 37(1-2):1–26. find
  21. Daniel Dahlmeier, Hwee Tou Ng, and Siew Mei Wu. 2013. Building a large annotated corpus of learner English: The NUS corpus of learner English. In Jakub Macina, Nico Daheim, Sankalan Chowdhury, Tanmay Sinha, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. 2023. MathDial: A dialogue tutoring dataset with rich pedagogical properties grounded in math reasoning problems. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5602–5621, Singapore. Association for Computational Linguistics.
  22. Jiyou Jia. 2009. CSIEC: A computer assisted English learning chatbot based on textual knowledge and reasoning. Knowledge-Based Systems, 22(4):249– 255. ↗
  23. Masahiro Kaneko and Naoaki Okazaki. 2024. Controlled generation with prompt insertion for natural language explanations in grammatical error correction. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 3955–3961, Torino, Italia. ELRA and ICCL. find
  24. Janick Michot, Manuela Hürlimann, Jan Deriu, Luzia Sauer, Katsiaryna Mlynchyk, and Mark Cieliebak. 2024. Error-preserving Automatic Speech Recognition of Young English Learners’ Language. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6444–6454, Bangkok, Thailand. Association for Computational Linguistics. ↗
  25. Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. 2022. Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP. arXiv:2212.14024 [cs.CL] . ↗
  26. Marvin Minsky. 1986. The Society of Mind. Simon & Schuster, New York. ↗
  27. Tomoya Mizumoto, Mamoru Komachi, Masaaki Nagata, and Yuji Matsumoto. 2011. Mining revision log of language learning sns for automated Japanese error correction of second language learners. In Proceedings of the 5th International Joint Conference on Natural Language Processing (IJCNLP), pages 147–155. find
  28. Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2023. DSPy: Compiling declarative language model calls into self-improving pipelines. arXiv:2310.03714 [cs.CL] . ↗
  29. Amr M. Mohamed. 2024. Exploring the potential of an AI-based chatbot (ChatGPT) in enhancing English as a foreign language (EFL) teaching: perceptions of EFL faculty members. Education and Information Technologies, 29(3):3195–3217. ↗
  30. Lasha Labadze, Maya Grigolia, and Lela Machaidze. 2024. Role of AI chatbots in education: systematic literature review. International Journal of Educational Technology in Higher Education, 21(1):28. ↗
  31. Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. 2017. JFLEG: A fluency corpus and benchmark for grammatical error correction. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 229–234, Valencia, Spain. Association for Computational Linguistics. ↗
  32. Vladimir I. Levenshtein. 1966. Binary codes capable of correcting deletions, insertions and reversals. Soviet Physics Doklady, 10:707–710. find
  33. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459– 9474. Curran Associates, Inc.
  34. Catherine Nickerson. 2005. English as a lingua franca in international business contexts. English for Specific Purposes, 24(4):367–380. ↗
  35. Kostiantyn Omelianchuk, Andrii Liubonko, Oleksandr Skurzhanskyi, Artem Chernodub, Oleksandr Korni-ienko, and Igor Samokhin. 2024. Pillars of grammatical error correction: Comprehensive inspection of contemporary approaches in the era of large language models. In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), pages 17–33, Mexico City, Mexico. Association for Computational Linguistics. ↗
  36. Yinghao Aaron Li, Cong Han, Vinay Raghavan, Gavin Mischler, and Nima Mesgarani. 2023. StyleTTS 2: Towards human-level text-to-speech through style diffusion and adversarial training with large speech language models. In Advances in Neural Information Processing Systems, volume 36, pages 19594– 19621. Curran Associates, Inc. ↗
  37. Cristóbal Lozano, Ana Díaz-Negrillo, and Marcus Callies. 2020. Designing and compiling a learner corpus of written and spoken narratives: COREFL. In Christiane Bongartz and Jacopo Torregrossa, editors, What’s in a Narrative? Variation in Story-Telling at the Interface between Language and Literacy, pages 21–46. Peter Lang. find
  38. Larry Pitts. 2015. ESL Role Plays: 50 Engaging Role Plays for ESL and EFL Classes. ECQ Publishing. find
  39. Alexis Plaquet and Hervé Bredin. 2023. Powerset multi-class cross entropy loss for neural speaker diarization. In Proc. INTERSPEECH 2023. find
  40. Yixiao Song, Kalpesh Krishna, Rajesh Bhatt, Kevin Gimpel, and Mohit Iyyer. 2024. GEE! grammar error explanation with large language models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 754–781, Mexico City, Mexico. Association for Computational Linguistics. find
  41. Muhammad Reza Qorib and Hwee Tou Ng. 2023. System combination via quality estimation for grammatical error correction. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12746–12759, Singapore. Association for Computational Linguistics. find
  42. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 28492–28518. PMLR. ↗
  43. Felix Stahlberg and Shankar Kumar. 2021. Synthetic data generation for grammatical error correction with tagged corruption models. In Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications, pages 37–47, Online. Association for Computational Linguistics. ↗
  44. Katherine Stasaski, Kimberly Kao, and Marti A. Hearst. 2020. CIMA: A large open access dialogue dataset for tutoring. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 52–64, Seattle, WA, USA → Online. Association for Computational Linguistics. find
  45. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. ↗
  46. Chrysi Rapanta, Cláudia Gonçalves, João Rui Pereira, Dilar Cascalheira, Beatriz Gil, Rita Morais, Anna Čermáková, Julia Peck, Benjamin Brummernhen-rich, Regina Jucks, Mercè Garcia-Milà, Andrea Miralda-Banda, José Luna, Maria Vrikki, Maria Evagorou, and Fabrizio Macagno. 2021. Multicultural classroom discourse dataset on teachers’ and students’ dialogic empathy. Data in Brief, 39:107518. find
  47. Toshikazu Tajiri, Mamoru Komachi, and Yuji Matsumoto. 2012. Tense and aspect error correction for esl learners using global context. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 198–202.
  48. Yuki Takashima, Yusuke Fujita, Shinji Watanabe, Shota Horiguchi, Paola García, and Kenji Nagamatsu. 2021. End-to-end speaker diarization conditioned on speech activity and overlap detection. In 2021 IEEE Spoken Language Technology Workshop (SLT), pages 849–856. ↗
  49. Keshav Santhanam, Omar Khattab, Christopher Potts, and Matei Zaharia. 2022a. PLAID: An efficient engine for late interaction retrieval. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, CIKM ’22, page 1747–1756, New York, NY, USA. Association for Computing Machinery. ↗
  50. Marcos Treviso, Nuno M. Guerreiro, Sweta Agrawal, Ricardo Rei, José Pombal, Tania Vaz, Helena Wu, Beatriz Silva, Daan van Stigt, and André F. T. Martins. 2024. xTower: A multilingual LLM for explaining and correcting translation errors. arXiv:2406.19482 [cs.CL] . ↗
  51. Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2022b. Col-BERTv2: Effective and efficient retrieval via lightweight late interaction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3715–3734, Seattle, United States. Association for Computational Linguistics. find
  52. Hyejin Yang, Heyoung Kim, Jang Ho Lee, and Dongkwang Shin. 2022. Implementation of an AI chatbot as an English conversation partner in EFL speaking classes. ReCALL, 34(3):327–343. ↗
  53. Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A new dataset and method for automatically grading ESOL texts. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 180–189, Portland, Oregon, USA. Association for Computational Linguistics.
  54. Maganat Shegebayev. 2023. Rise of English as business lingua franca at the turn of the century: An overview. In Stanley D. Brunn and Roland Kehrein, editors, Language, Society and the State in a Changing World, pages 357–365. Springer International Publishing, Cham. ↗
  55. Shunan Zhang, Cheng Shan, John Sie Yuen Lee, ShaoPeng Che, and Jang Hyun Kim. 2023. Effect of chatbot-assisted language learning: A meta-analysis. Education and Information Technologies, 28(11):15223–15243. ↗
  56. Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung, Moya Chen, Kushal Arora, Joshua Lane, Morteza Behrooz, William Ngan, Spencer Poff, Naman Goyal, Arthur Szlam, Y-Lan Boureau, Melanie Kambadur, and Jason Weston. 2022. BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage. Mingchen Zhuge, Haozhe Liu, Francesco Faccio, Dylan R. Ashley, Róbert Csordás, Anand Gopalakrishnan, Abdullah Hamdi, Hasan Abed Al Kader Hammoud, Vincent Herrmann, Kazuki Irie, Louis Kirsch, Bing Li, Guohao Li, Shuming Liu, Jinjie Mai, Piotr Pi˛ekos, Aditya Ramesh, Imanol Schlag, Weimin Shi, Aleksandar Stanić, Wenyi Wang, Yuhui Wang, Mengmeng Xu, Deng-Ping Fan, Bernard Ghanem, and Jürgen Schmidhuber. 2023. Mindstorms in natural language-based societies of mind. In R0-FoMo: Robustness of Few-shot and Zero-shot Learning in Foundation Models, NeurIPS 2023. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ortiz-2024-conversational-intelligent-tutoring-system-improving/