Elamur Research⌕
☰
home / papers / 2024 / LearnerVoice: A Dataset of Non-Native English Learners' Spontaneous Speech

LearnerVoice: A Dataset of Non-Native English Learners' Spontaneous Speech

Paper · 2024 · Haechan Kim, Junho Myung, Seoyoung Kim et al.

Summary

Speech recognizers often fail on spontaneous speech from second-language (L2) learners. Such speech is full of hesitations, restarts and grammar mistakes, and ASR systems tend to drop these or quietly "fix" them. That is harmful when transcripts are used to grade speaking ability. The authors release LearnerVoice, about 50 hours of one-to-one tutoring audio from 58 Korean-L1 adult learners on the Ringle platform. It was transcribed by trained annotators who were told to keep every filler, self-repair and ungrammatical phrase. An automatic counting pipeline shows the corpus has clearly more of these "L2S features" than Switchboard or Librispeech. Fine-tuning whisper-small.en on it almost halves WER, beating both the vanilla model and a version tuned on a token-matched Switchboard subset. A manual error analysis shows that most vanilla-model errors stem from L2S features, and that fine-tuning removes these much more than other error types. The resource matters for fairer ASR and for automatic speaking assessment.

Key points

  • LearnerVoice has 50.04 hours from 239 lessons (20- or 40-minute sessions), 229,671 whitespace tokens and 32 kHz audio. Speakers are 58 Korean learners (38 female, 20 male) with estimated IELTS speaking bands of 4–9, averaging 5.95.
  • Construction: pyannote VAD segmentation, then transcription by native-Korean annotators. Annotators had a year or more of US residence or TOEFL > 100 and were trained on paired audio/transcript examples of L2S features.
  • L2S counts per unit, LearnerVoice vs Switchboard vs Librispeech:
    • filler words per token: .136 / .077 / .000
    • self-repairs per token: .127 / .118 / .040
    • grammar errors per c-unit: .477 / .200 / .280
    • All differences are significant (Mann-Whitney U).
  • WER on the LearnerVoice test set:
    • whisper large-v3: 19.18
    • vanilla small.en: 18.38
    • small.en tuned on Switchboard: 15.03
    • small.en tuned on LearnerVoice: 10.26 (a 44.2% reduction)
  • A 9-type error taxonomy attributes 54.2% of vanilla errors to L2S-related types (Filler word, Self-repair, Ungrammatical Expression). After fine-tuning, these drop by 48.1%, versus 19.2% for the other types.

Interesting ideas

  • I1. Training sets for the comparison were matched by transcript token count (about 185k each) rather than by audio hours, so speaking rate cannot confound the result.
  • I2. Disfluency is measured by a staged pipeline: strip fillers by lexicon, then find self-repairs as repeated lemma n-grams (n < 5), then count grammar errors by diffing against a GEC rewrite with ERRANT.
  • I3. Errors are not reported as one number but tagged by cause, and improvement is measured separately per cause, which reveals what fine-tuning actually fixed.
  • I4. The transcription policy deliberately preserves learner mistakes, because systems that auto-correct grammar erase exactly the signal needed for assessment.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when comparing treebanks or silver batches for gparse, equalize by token count so a gain reflects data kind, not data volume.
  • I2 (tree): the same cascade could pre-mark fillers as discourse and repeated spans as reparandum before parsing, so learner or spoken text gets correct UD structure.
  • I3 (learn): break LAS drops and repair-rule gains down by phenomenon (disfluency, agreement error, etc.), giving an explainable evaluation that a single score hides.
  • I4 (back): the text → graph → text compressor should be able to round-trip ungrammatical input faithfully, or explicitly flag each normalization it applies.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Haechan Kim, Junho Myung, Seoyoung Kim, Sungpah Lee, Dongyeop Kang and Juho Kim. 2024. LearnerVoice: A Dataset of Non-Native English Learners' Spontaneous Speech. arXiv:2407.04280.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.21437/interspeech.2024-2392 · arXiv 2407.04280

@misc{kim2024learnervoice,
  title = {{LearnerVoice: A Dataset of Non-Native English Learners' Spontaneous Speech}},
  author = {Haechan Kim, Junho Myung, Seoyoung Kim, Sungpah Lee, Dongyeop Kang and Juho Kim},
  year = {2024},
  doi = {10.21437/interspeech.2024-2392},
  eprint = {2407.04280},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.21437/interspeech.2024-2392}
}

References

26 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. Dufour, V. Jousse, Y. Estève, F. Béchet, and G. Linarès, “Spontaneous speech characterization and detection in large audio database,” SPECOM, St. Petersburg, 2009. find
  2. P. Boula de Mareüil, B. Habert, F. Bénard, M. Adda-Decker, C. Barras, G. Adda, and P. Paroubek, “A quantitative study of disfluencies in French broadcast interviews,” in Proc. Disfluency in Spontaneous Speech (DiSS 2005), 2005, pp. 27–32. find
  3. Y. Qiao, W. Zhou, E. Kerz, and R. Schlüter, “The Impact of ASR on the Automatic Analysis of Linguistic Complexity and Sophistication in Spontaneous L2 Speech,” in Proc. Interspeech 2021, 2021, pp. 4453–4457. ↗
  4. K.-n. Hassanali, S.-Y. Yoon, and L. Chen, “Automatic scoring of non-native children’s spoken language proficiency.” in SLaTE, 2015, pp. 13–18. find
  5. R. Gretter, M. Matassoni, K. Allgaier, S. Tchistiakova, and D. Falavigna, “Automatic assessment of spoken language proficiency of non-native children,” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 7435–7439. find
  6. A. Housen, F. Kuiken, and I. Vedder, “Complexity, accuracy and fluency,” Dimensions of L2 performance and proficiency: Complexity, accuracy and fluency in SLA, vol. 32, pp. 1–20, 2012. ↗
  7. X. Yan, H. R. Kim, and J. Y. Kim, “Complexity, accuracy and fluency (caf) features of speaking performances on aptis across different levels on the common european framework of reference for languages (cefr),” ARAGs Research Reports, AR-G/2018, vol. 1, 2018. find
  8. K. Knill, M. Gales, K. Kyriakopoulos, A. Malinin, A. Ragni, Y. Wang, and A. Caines, “Impact of ASR Performance on Free Speaking Language Assessment,” in Proc. Interspeech 2018, 2018, pp. 1641–1645. find
  9. J. J. Godfrey, E. C. Holliman, and J. McDaniel, “Switchboard: Telephone speech corpus for research and development,” in Acoustics, speech, and signal processing, ieee international conference on, vol. 1. IEEE Computer Society, 1992, pp. 517–520. find
  10. V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, pp. 5206–5210. find
  11. S. Goldwater, D. Jurafsky, and C. D. Manning, “Which words are hard to recognize? prosodic, lexical, and disfluency factors that increase speech recognition error rates,” Speech Communication, vol. 52, no. 3, pp. 181–200, 2010. ↗
  12. G. Zhu, Y. Yan, J.-P. Caceres, and Z. Duan, “Transcription free filler word detection with neural semi-crfs,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5. find
  13. A. Zafar, B. Mamlin, S. Perkins, A. M. Belsito, J. Overhage, and C. J. McDonald, “A simple error classification system for understanding sources of error in automatic speech recognition and human transcription,” International Journal of Medical Informatics, vol. 73, no. 9, pp. 719–730, 2004. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1386505604001200 ↗
  14. J. Spilker, M. Klarner, and G. Görz, “Processing self-corrections in a speech-to-speech system,” in Verbmobil: Foundations of speech-to-speech translation. Springer, 2000, pp. 131–140. ↗
  15. M.-S. Kim, “An error analysis of a learner corpus of written and spoken english produced by korean university students,” Unpublished doctoral dissertation). Korea University, South Korea, 2009. find
  16. H. Strik, J. van Doremalen, J. van de Loo, and C. Cucchiarini, “Improving ASR processing of ungrammatical utterances through grammatical error modeling,” in Proc. Speech and Language Technology in Education (SLaTE 2011), 2011, pp. 109–112. find
  17. R. Frieske and B. E. Shi, “Hallucinations in neural automatic speech recognition: Identifying errors and hallucinatory models,” arXiv preprint arXiv:2401.01572, 2024. ↗
  18. A. Rodrigues, R. Santos, J. Abreu, P. Beça, P. Almeida, and S. Fernandes, “Analyzing the performance of asr systems: The effects of noise, distance to the device, age and gender,” in Proceedings of the XX International Conference on Human Computer Interaction, 2019, pp. 1–8. find
  19. I. Vasilescu, D. Yahia, N. Snoeren, M. Adda-Decker, and L. Lamel, “Cross-lingual study of asr errors: on the role of the context in human perception of near-homophones,” in Twelfth Annual Conference of the International Speech Communication Association, 2011. find
  20. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in International Conference on Machine Learning. PMLR, 2023, pp. 28 492–28 518. ↗
  21. H. Bredin and A. Laurent, “End-to-end speaker segmentation for overlap-aware resegmentation,” in Proc. Interspeech 2021, Brno, Czech Republic, August 2021. find
  22. IELTS. Ielts test statistics. Last Accessed: 2024-03-11. [Online]. Available: https://ielts.org/researchers/our-research/test-statistics ↗
  23. S. L. Eisenberg and L.-Y. Guo, “Percent grammatical responses as a general outcome measure: Initial validity.” Language, speech, and hearing services in schools, vol. 49 1, pp. 98–107, 2018. [Online]. Available: https://api.semanticscholar.org/CorpusID: 36645664 ↗
  24. V. Raheja, D. Kumar, R. Koo, and D. Kang, “Coedit: Text editing by task-specific instruction tuning,” 2023. find
  25. C. Bryant, M. Felice, and T. Briscoe, “Automatic annotation and evaluation of error types for grammatical error correction,” in Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), R. Barzilay and M.-Y. Kan, Eds. Vancouver, Canada: Association for Computational Linguistics, Jul. 2017, pp. 793–805. [Online]. Available: https://aclanthology.org/P17-1074 ↗
  26. V. I. Levenshtein et al., “Binary codes capable of correcting deletions, insertions, and reversals,” in Soviet physics doklady, vol. 10, no. 8. Soviet Union, 1966, pp. 707–710. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kim-2024-learnervoice-dataset-non-native-english/