Elamur Research⌕
☰
home / papers / 2024 / Error-preserving Automatic Speech Recognition of Young English Learners' Language

Error-preserving Automatic Speech Recognition of Young English Learners' Language

Paper · 2024 · Janick Michot, Manuela Hürlimann, Jan Deriu et al.

Summary

The paper builds speech recognition for a voice chatbot that will help Swiss primary pupils practise spoken English. Such a tool needs transcripts that keep the learners' mistakes intact, because corrective feedback depends on them. Off-the-shelf ASR fails this need in two ways. It is mostly trained on adult native speakers reading aloud. Its built-in language model also tends to silently fix grammatical and lexical slips.

The authors recorded about 85 hours of spontaneous English from pupils in grades 4–6, collected through game-like speaking tasks. Each recording has a verbatim transcript with learner errors marked by special symbols. They propose WEPR (Word-Based Error Preservation Rate), a metric computed only over reference words that carry an error annotation. They then compare seven pre-trained ASR systems with a wav2vec2 XLSR 300M model fine-tuned on their corpus.

The fine-tuned model preserves errors better than every baseline, Whisper-Large included. It also clearly beats English models of similar size on WER. The work shows that targeted in-domain data is needed when downstream tasks depend on faithful, uncorrected input.

Key points

  • The corpus has 85 hours of audio: 45,004 utterances from 327 distinct speakers. Pupils were aged 9–14, from 8 schools in German-speaking Switzerland, and recordings ran from March to November 2023. The expected proficiency is roughly CEFR A1–A2, although levels were not annotated.
  • Speech was elicited with role plays, riddles, imaginary TV interviews and personal questions, designed to resemble the planned chatbot conversations. Tasks were piloted with a grade 4 class first.
  • WEPR is a new metric that scores only the words annotated as errors. It is reported alongside standard metrics such as WER across 7 pre-trained systems and the fine-tuned model.
  • The released model is chall_wav2vec2_xlsr_300m (ChaLL-300M), and code is on GitHub. The audio and transcripts are restricted to protect the minors and are available only through a joint-project agreement.
  • Early fine-grained annotation shows verbal errors are the largest category, about 22% of all errors, with subject–verb agreement the most frequent within it. Code-switching into German remains unsolved, even for Whisper-Large.

Interesting ideas

  • I1. Evaluate faithfulness only at the positions that are known to be deviant, instead of averaging over all words, so that a strong overall score cannot hide systematic "correction" of the cases you care about.
  • I2. A powerful prior (here the ASR language model) can erase the very signal a downstream module needs, so the front end must be judged on whether it passes input through unchanged, not only on accuracy.
  • I3. Verbatim transcripts with inline error symbols give a two-layer resource: what was actually said, plus where it departs from the norm.
  • I4. Data was elicited with tasks deliberately shaped like the target deployment, and the pilot tasks that produced unusable data were dropped.
  • I5. Comparing many pre-trained systems on both a general metric and a targeted metric exposes trade-offs that a single number would hide.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Add an evaluation slice that scores the tokenizer, tagger and parser only on annotated nonstandard tokens, used as a negative control against silent normalization.
  • I2 (tokens): Lemmatization and morphology should keep the observed surface form, such as "do" where "is" was needed, and attach the expected form as a separate feature rather than overwriting it.
  • I3 (tree): Learner or noisy text could be parsed with error marks carried as token attributes, so UD trees reflect the real input while still flagging agreement violations for later explanation.
  • I4 (answer): Build test sets for school-course readers by mimicking the actual task formats pupils face, piloting them, and discarding items that produce no usable answers.
  • I5 (learn): When ranking witness parsers for silver data, report a general score and a targeted score, such as accuracy on rare constructions, side by side before trusting their agreement.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓ZHAW Zurich University of Applied SciencesWinterthur · Switzerland6 works · 181 citations🎓Zurich University of Teacher EducationZurich · Switzerland2 works · 5 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksSpeech & spoken language · 480 works

Cite

Cite as Janick Michot, Manuela Hürlimann, Jan Deriu, Luzia Sauer, Katsiaryna Mlynchyk and Mark Cieliebak. 2024. Error-preserving Automatic Speech Recognition of Young English Learners' Language. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.acl-long.348

@inproceedings{michot2024error,
  title = {{Error-preserving Automatic Speech Recognition of Young English Learners' Language}},
  author = {Janick Michot, Manuela Hürlimann, Jan Deriu, Luzia Sauer, Katsiaryna Mlynchyk and Mark Cieliebak},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.348},
  url = {https://doi.org/10.18653/v1/2024.acl-long.348}
}

References

34 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Beena Ahmed, Kirrie J. Ballard, Denis Burnham, Thar-makulasingam Sirojan, Hadi Mehmood, Dominique Estival, Elise Baker, Felicity Cox, Joanne Arciuli, Titia Benders, Katherine Demuth, Barbara Kelly, Chloé Diskin-Holdaway, Mostafa Shahin, Vidhyasaharan Sethu, Julien Epps, Chwee Beng Lee, and Eliathamby Ambikairajah. 2021. AusKidTalk: An Auditory-Visual Corpus of 3- to 12-Year-Old Australian Children’s Speech. In Proc. Interspeech 2021, pages 3680–3684. find
  2. R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber. 2020. Common voice: A massively-multilingual speech corpus. In Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4211–4215. ↗
  3. Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, et al. 2021. Xls-r: Self-supervised cross-lingual speech representation learning at scale. arXiv preprint arXiv:2111.09296. ↗
  4. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449–12460.
  5. Anton Batliner, Mats Blomberg, Shona D’Arcy, Daniel Elenius, Diego Giuliani, Matteo Gerosa, Christian Hacker, Martin Russell, Stefan Steidl, and Michael Wong. 2005. The PF_STAR children’s speech corpus. In Proc. Interspeech 2005, pages 2761–2764. find
  6. Claudia Baur, Andrew Caines, Cathy Chua, Johanna Gerlach, Mengjie Qian, Manny Rayner, Martin Russell, Helmer Strik, and Xizi Wei. 2018. Overview of the 2018 spoken call shared task. In Proceedings of Interspeech 2018, Interspeech, pages 2354–2358. ISCA. Interspeech 2018 ; Conference date: 02-09-2018 Through 06-09-2018. ↗
  7. Jan Deriu, Alvaro Rodrigo, Arantxa Otegi, Guillermo Echegoyen, Sophie Rosset, Eneko Agirre, and Mark Cieliebak. 2021. Survey on evaluation methods for dialogue systems. Artificial Intelligence Review, 54:755–810. ↗
  8. Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 2021. 8-bit optimizers via block-wise quantization. arXiv preprint arXiv:2110.02861. Use of AI Assistants Rod Ellis. 2021. Explicit and Implicit Oral Corrective Feedback. The Cambridge Handbook of Corrective Feedback in Second Language Learning and Teaching, pages 341–364. ChatGPT was used to support the creation of some figures. No AI assistants were used to write the text of this paper. ↗
  9. Maxine S Eskenazi. 1996. Kids: a database of children’s speech. The Journal of the Acoustical Society of America, 100(4_Supplement):2759–2759. find
  10. Matteo Gerosa, Diego Giuliani, and Shrikanth Narayanan. 2006. Acoustic analysis and automatic recognition of spontaneous children’s speech. In Ninth International Conference on Spoken Language Processing. find
  11. Roberto Gretter, Marco Matassoni, Stefano Bannò, and Falavigna Daniele. 2020. TLT-school: a corpus of non native children speech. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 378–385, Marseille, France. European Language Resources Association. ↗
  12. Jinjie Ni, Tom Young, Vlad Pandelea, Fuzhao Xue, and Erik Cambria. 2023. Recent advances in deep learning based dialogue systems: A systematic survey. Artificial intelligence review, 56(4):3055–3155. ↗
  13. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on, pages 5206–5210. IEEE. find
  14. Nancy Grimm, Michael Meyer, and Laurenz Volkmann. 2015. Teaching English. Narr Francke Attempto Verlag. find
  15. Simone E. Pfenninger and Johanna Lendl. 2017. Transitional woes: On the impact of l2 input continuity from primary to secondary school. Studies in Second Language Learning and Teaching, 7(3):443–469. ↗
  16. Jonatas Grosman. 2021. Fine-tuned XLSR-53 large model for speech recognition in English. https://huggingface.co/jonatasgrosman/ wav2vec2-large-xlsr-53-english. ↗
  17. Maja Popović. 2015. chrf: character n-gram f-score for automatic mt evaluation. In Proceedings of the tenth workshop on statistical machine translation, pages 392–395.
  18. Jonatas Grosman. 2022. Fine-tuned XLS-R 1B model for speech recognition in English. https://huggingface.co/jonatasgrosman/ wav2vec2-xls-r-1b-english. ↗
  19. A. Potamianos and S. Narayanan. 2003. Robust recognition of children’s speech. IEEE Transactions on Speech and Audio Processing, 11(6):603–616. ↗
  20. Andreas Hagen, Bryan Pellom, and Ronald Cole. 2003. Children’s speech recognition with application to interactive books and tutors. In 2003 IEEE Workshop on Automatic Speech Recognition and Understanding (IEEE Cat. No. 03EX721), pages 186–191. IEEE. find
  21. Tricia Hedge. 2001. Teaching and learning in the language classroom, volume 106. find
  22. Robert Kleinschroth and Pete Oldham. 2014. Sprechkompetenz-Training im Englischunterricht 7-8: Lebensnahe Sprechanlässe und vielfältige Aufgaben (7. und 8. Klasse). Auer Verlag. find
  23. Sungbok Lee, Alexandros Potamianos, and Shrikanth Narayanan. 1999. Acoustics of children’s speech: Developmental changes of temporal and spectral parameters. The Journal of the Acoustical Society of America, 105(3):1455–1468. ↗
  24. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. ↗
  25. Renee Lu, Mostafa Shahin, and Beena Ahmed. 2022. Improving children’s speech recognition by fine-tuning self-supervised adult speech representations. arXiv preprint arXiv:2211.07769. ↗
  26. Rao Ma, Mengjie Qian, Mark Gales, and Katherine M Knill. 2023. Adapting an ASR Foundation Model for Spoken Language Assessment. In Proc. 9th Workshop on Speech and Language Technology in Education (SLaTE), pages 104–108. find
  27. Mishaim Malik, Muhammad Kamran Malik, Khawar Mehmood, and Imran Makhdoom. 2021. Automatic speech recognition: a survey. Multimedia Tools and Applications, 80:9411–9457. ↗
  28. Sameer Pradhan, Ron Cole, and Wayne Ward. 2016. My science Tutor—Learning science with a conversational virtual tutor. In Proceedings of ACL-2016 System Demonstrations, pages 121–126, Berlin, Germany. Association for Computational Linguistics. find
  29. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. Robust Speech Recognition via Large-Scale Weak Supervision. ↗
  30. Lars Rumberg, Christopher Gebauer, Hanna Ehlert, Maren Wallbaum, Lena Bornholt, Jörn Ostermann, and Ulrike Lüdtke. 2022. kidsTALC: A Corpus of 3-to 11-year-old German Children’s Connected Natural Speech. In Proc. Interspeech 2022, pages 5160– 5164. find
  31. Prashanth Gurunath Shivakumar and Panayiotis Georgiou. 2020. Transfer learning from adult to children for speech recognition: Evaluation, analysis and recommendations. Computer Speech & Language, 63:101077. ↗
  32. Prashanth Gurunath Shivakumar and Shrikanth Narayanan. 2022. End-to-end neural systems for automatic children speech recognition: An empirical study. Computer Speech & Language, 72:101289. ↗
  33. Khaldoun Shobaki, John-Paul Hosom, and Ronald A. Cole. 2000. The OGI kids² speech corpus and recognizers. In Proc. 6th International Conference on Spoken Language Processing (ICSLP 2000), pages vol. 4, 258–261. find
  34. Wayne Ward, Ron Cole, and Sameer Pradhan. 2019. My science tutor and the myst corpus. Boulder Learn. Inc. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/michot-2024-error-preserving-automatic-speech-recognition/