Elamur Research⌕
☰
home / papers / 2025 / Automatic Speech Recognition for Non-Native English: Accuracy and Disfluency Handling

Automatic Speech Recognition for Non-Native English: Accuracy and Disfluency Handling

Paper · 2025 · Michael McGuire

Summary

This study asks how well current, widely available speech recognition services cope with English spoken by non-native speakers, which matters for language-learning and language-testing tools. The author ran five state-of-the-art ASR systems, all with default settings except disfluency-retention options, on recordings from the L2-ARCTIC corpus. Their native languages fell into six groups: Korean, Arabic, Vietnamese, Hindi, Spanish and Chinese. Two kinds of speech were used: read single sentences and spontaneous picture-story narratives. Accuracy was measured with Match Error Rate (MER). The study also checked how faithfully each system kept disfluencies (fillers, repetitions, revisions) and how long each took to process audio.

On read speech, Whisper and AssemblyAI came close to human transcription quality. On spontaneous speech, RevAI was the most accurate. Systems differed sharply in how they treated disfluencies. Some, like default Whisper, quietly tidied the speech into fluent and even grammatically corrected text. The results help teachers and researchers choose a system for a given purpose, such as verbatim learner transcripts versus readable text.

Key points

  • Data: L2-ARCTIC corpus.
    • Read speech: 2,400 sentence recordings from 24 speakers.
    • Spontaneous speech: narratives from 22 speakers.
    • Six L1 groups.
  • Read speech: Whisper had a mean MER of 0.054 and AssemblyAI 0.056, roughly human-level.
  • Spontaneous speech: RevAI was best, with a mean MER of 0.063.
  • Prompting Whisper with an initial prompt of topic words plus filler tokens ("man woman suitcase city um uh") cut one narrative's MER against a verbatim reference from 0.238 to 0.033. Without the prompt, Whisper dropped fillers and repairs and fixed grammar, for example turning "there is the uh stars" into "there are stars".
  • Slower processing did not reliably buy better accuracy.
  • Accuracy varied by L1, so results may not carry over to other accents.

Interesting ideas

  • I1. Fluent output can be wrong output: a system that silently "improves" speech scores badly against a verbatim reference and erases exactly the learner errors a teacher needs, so any normalizing step should be measured for what it removes.
  • I2. A short priming prompt that mixes topic vocabulary with the desired style tokens (here fillers) switched a model from cleaned to verbatim transcription without any retraining.
  • I3. Scoring disfluencies by type (fillers, repetitions, revisions) separately shows strengths that a single error rate hides.
  • I4. Reporting results split by condition (read vs. spontaneous, per L1) showed the ranking of systems can flip between conditions.
  • I5. Measuring processing time next to accuracy showed that cost and quality are not tied together.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): when spoken or learner text arrives, keep fillers and self-repairs as UD discourse/reparandum attachments instead of letting a cleanup pass delete them, so the tree stays explainable against the real input.
  • I2 (learn): when querying LLM teacher tiers for silver annotations, seed the prompt with in-domain words and examples of the exact output format, so teachers stop "correcting" the source text.
  • I3 (tokens): evaluate the tokenizer and tagger on disfluent input per phenomenon (repeated tokens, "um/uh", restarts) to find which one breaks lemmas or POS tags.
  • I4 (answer): report reader accuracy separately for school tests, word problems and fables, rather than one pooled score, so a regression in one genre is not masked.
  • I5 (learn): log latency for the perceptron parser and gparse alongside LAS in every evaluation, to decide where the GPU parser is actually worth its cost.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 worksSignal Processing · 452 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksSpeech & spoken language · 480 works

Cite

Cite as Michael McGuire. 2025. Automatic Speech Recognition for Non-Native English: Accuracy and Disfluency Handling. arXiv:2503.06924.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2503.06924 · arXiv 2503.06924

@misc{mcguire2025automatic,
  title = {{Automatic Speech Recognition for Non-Native English: Accuracy and Disfluency Handling}},
  author = {Michael McGuire},
  year = {2025},
  doi = {10.48550/arxiv.2503.06924},
  eprint = {2503.06924},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2503.06924}
}

References

57 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adda-Decker, M., & Lamel, L. (2005). Do speech recognizers prefer female speakers? Interspeech 2005, 2205–2208. https://doi.org/10.21437/Interspeech.2005-699 ↗
  2. Amann, R., Li, Z., Bruno, B., & Niehues, J. (2024, September). Augmenting automatic speech recognition models with disfluency detection [arXiv:2409.10177 [cs]]. https://doi.org/10.48550/arXiv.2409.10177 Amodei, D., Anubhai, R., Battenberg, E., Case, C., Casper, J., Catanzaro, B., Chen, J., Chrzanowski, M., Coates, A., Diamos, G., Elsen, E., Engel, J., Fan, L., Fougner, C., Han, T., Hannun, A., Jun, B., LeGresley, P., Lin, L., . . . Zhu, Z. (2015, December). Deep Speech 2: End-to-end speech recognition in English and Mandarin [arXiv:1512.02595 [cs]]. https://doi.org/10.48550/arXiv.1512.02595 ↗
  3. Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate [arXiv:1409.0473 [cs]]. https://doi.org/10.48550/arXiv.1409.0473 ↗
  4. Ballier, N., Arnold, T., Méli, A., Thurston, T., & Yunès, J.-B. (2024). Whisper for L2 speech scoring. International Journal of Speech Technology, 27(4), 923–934. https://doi.org/10.1007/s10772-024-10141-5 ↗
  5. Bell, P., Fainberg, J., Klejch, O., Li, J., Renals, S., & Swietojanski, P. (2021). Adaptation algorithms for neural network-based speech recognition: An overview [arXiv:2008.06580 [eess]]. IEEE Open Journal of Signal Processing, 2, 33–66. https://doi.org/10.1109/OJSP.2020.3045349 ↗
  6. Browne, C. (2014). The New General Service List Version 1.01: Getting Better All the Time. Korea TESOL Journal, 11(1), 35–50. find
  7. Byrne, W., Beyerlein, P., Huerta, J., Khudanpur, S., Marthi, B., Morgan, J., Peterek, N., Picone, J., Vergyri, D., & Wang, T. (2000). Towards language independent acoustic modeling. 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.00CH37100), 2, II1029–II1032. https://doi.org/10.1109/ICASSP.2000.859138 Chan, W., Jaitly, N., Le, Q., & Vinyals, O. (2016). Listen, attend and spell: A neural network for large vocabulary conversational speech recognition [ISSN: 2379-190X]. 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 4960–4964. https://doi.org/10.1109/ICASSP.2016.7472621 ↗
  8. Chorowski, J., Bahdanau, D., Cho, K., & Bengio, Y. (2014, December). End-to-end continuous speech recognition using attention-based recurrent NN: First results [arXiv:1412.1602 [cs]]. https://doi.org/10.48550/arXiv.1412.1602 CMUSphinx open source speech recognition. (2025). CMUSphinx open source speech recognition [http://cmusphinx. github.io/]. CMUSphinx Open Source Speech Recognition. ↗
  9. Coniam, D. (1999). Voice recognition software accuracy with second language speakers of English. System, 27(1), 49–64. https://doi.org/10.1016/S0346-251X(98)00049-9 ↗
  10. Dahl, G. E., Dong Yu, Li Deng, & Acero, A. (2012). Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition. IEEE Transactions on Audio, Speech, and Language Processing, 20(1), 30–42. https://doi.org/10.1109/TASL.2011.2134090 ↗
  11. Davis, K. H., Biddulph, R., & Balashek, S. (1952). Automatic recognition of spoken digits. The Journal of the Acoustical Society of America, 24(6), 637–642. https://doi.org/10.1121/1.1906946 ↗
  12. Derwing, T. M., Munro, M. J., & Carbonaro, M. (2000). Does popular speech recognition software work with ESL speech? [Publisher: [Wiley, Teachers of English to Speakers of Other Languages, Inc. (TESOL)]]. TESOL Quarterly, 34(3), 592–603. https://doi.org/10.2307/3587748 ↗
  13. Derwing, T. M., Munro, M. J., Thomson, R. I., & Rossiter, M. J. (2009). The relationship between L1 fluency and L2 fluency development [Place: United Kingdom Publisher: Cambridge University Press]. Studies in Second Language Acquisition, 31(4), 533–557. https://doi.org/10.1017/S0272263109990015 ↗
  14. Do, C.-T., Imai, S., Doddipatla, R., & Hain, T. (2024, July). Improving accented speech recognition using data augmentation based on unsupervised text-to-speech synthesis [arXiv:2407.04047 [cs]]. https://doi.org/10. 48550/arXiv.2407.04047 Ehsani, F., & Knodt, E. (1998). Speech technology in computer-aided language. Language Learning & Technology, 2(1), 54–73. http://llt.msu.edu/vol2num1/article3/ ↗
  15. Fiscus, J. G., Fisher, W. M., Martin, A. F., Przybocki, M. A., & Pallett, D. S. (2000). 2000 NIST evaluation of conversational speech recognition over the telephone: English and Mandarin performance results. find
  16. Fox, J. (2023, September). Introducing Nova-2: The fastest, most accurate speech-to-text API. Retrieved February 4, 2025, from https://deepgram.com/learn/nova-2-speech-to-text-api ↗
  17. Goldwater, S., Jurafsky, D., & Manning, C. D. (2010). Which words are hard to recognize? Prosodic, lexical, and disfluency factors that increase speech recognition error rates. Speech Communication, 52(3), 181–200. https://doi.org/10.1016/j.specom.2009.10.001 ↗
  18. Graham, C., & Roll, N. (2024). Evaluating OpenAI’s Whisper ASR: Performance analysis across diverse accents and speaker traits. JASA Express Letters, 4(2), 025206. https://doi.org/10.1121/10.0024876 ↗
  19. Graves, A. (2012, November). Sequence transduction with recurrent neural networks [arXiv:1211.3711 [cs]]. https://doi.org/10.48550/arXiv.1211.3711 ↗
  20. Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006). Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. Proceedings of the 23rd international conference on Machine learning, 369–376. https://doi.org/10.1145/1143844.1143891 ↗
  21. Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., & Pang, R. (2020, May). Conformer: Convolution-augmented transformer for speech recognition [arXiv:2005.08100 [eess]]. https://doi.org/10.48550/arXiv.2005.08100 ↗
  22. Hinsvark, A., Delworth, N., Rio, M. D., McNamara, Q., Dong, J., Westerman, R., Huang, M., Palakapilly, J., Drexler, J., Pirkin, I., Bhandari, N., & Jette, M. (2021, June). Accented speech recognition: A survey [arXiv:2104.10747 [cs]]. https://doi.org/10.48550/arXiv.2104.10747 ↗
  23. Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-r., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., & Kingsbury, B. (2012). Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 29(6), 82–97. https://doi.org/10.1109/MSP.2012. 2205597 Hirai, A., & Kovalyova, A. (2024). Speech-to-text applications’ accuracy in English language learners’ speech transcription. Language Learning & Technology, 28(1), 1–22. https://hdl.handle.net/10125/73555 ↗
  24. Jaitly, N., Nguyen, P., Senior, A., & Vanhoucke, V. (2012). Application of pretrained deep neural networks to large vocabulary speech recognition. Interspeech 2012, 2578–2581. https://doi.org/10.21437/Interspeech.2012-10 ↗
  25. Jette, M. (2024, January). What makes Rev’s V2 best in class? Retrieved February 4, 2025, from https://www.rev.com/ blog/what-makes-revs-v2-best-in-class Kahng, J. (2022, January). Fluency. In The Routledge Handbook of Second Language Acquisition and Speaking (1st ed., pp. 188–200). Routledge. https://doi.org/10.4324/9781003022497-17 ↗
  26. Koenecke, A., Nam, A., Lake, E., Nudell, J., Quartey, M., Mengesha, Z., Toups, C., Rickford, J. R., Jurafsky, D., & Goel, S. (2020). Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, 117(14), 7684–7689. https://doi.org/10.1073/pnas.1915768117 ↗
  27. Kominek, J., & Black, A. W. (2004). The CMU ARCTIC speech databases, 223–224. https://www.isca-archive.org/ ssw_2004/kominek04b_ssw.pdf Lawrence, M. A. (2016, November). Ez: Easy analysis and visualization of factorial experiments. https://CRAN.R-project.org/package=ez ↗
  28. Lee, K.-F. (1988). On large-vocabulary speaker-independent continuous speech recognition. Speech Communication, 7(4), 375–379. https://doi.org/10.1016/0167-6393(88)90053-2 ↗
  29. Lickley, R. J. (2015, March). Fluency and disfluency. In M. A. Redford (Ed.), The Handbook of Speech Production (1st ed., pp. 445–474). Wiley. https://doi.org/10.1002/9781118584156.ch20 ↗
  30. Lin, H., Deng, L., Yu, D., Gong, Y.-f., Acero, A., & Lee, C.-H. (2009). A study on multilingual acoustic modeling for large vocabulary ASR. 2009 IEEE International Conference on Acoustics, Speech and Signal Processing, 4333–4336. https://doi.org/10.1109/ICASSP.2009.4960588 ↗
  31. McCrocklin, S. (2019). ASR-based dictation practice for second language pronunciation improvement. Journal of Second Language Pronunciation, 5(1), 98–118. https://doi.org/10.1075/jslp.16034.mcc ↗
  32. McCrocklin, S., Humaidan, A., & Edalatishams, I. (2019). ASR dictation program accuracy: Have current programs improved? In J. Levis, C. Nagle, & E. Todey (Eds.), Proceedings of the 10th Pronunciation in Second Language Learning and Teaching Conference. find
  33. McGuire, M., & Larson-Hall, J. (2025). Assessing Whisper automatic speech recognition and WER scoring for elicited imitation: Steps toward automation. Research Methods in Applied Linguistics, 4(1). https://doi.org/https://doi.org/10.1016/j.rmal.2025.100197 Morris, A. C., Maier, V., & Green, P. (2004). From WER and RIL to MER and WIL: Improved evaluation measures for connected speech recognition. Interspeech 2004, 2765–2768. https://doi.org/10.21437/Interspeech.2004-668 ↗
  34. Neri, A., Cucchiarini, C., & Strik, W. (2003). Automatic speech recognition for second language learning: How and why it actually works, 1157–1160. http://www.internationalphoneticassociation.org/icphs/icphs2003 ↗
  35. Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015). Librispeech: An ASR corpus based on public domain audio books. 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 5206–5210. https://doi.org/10.1109/ICASSP.2015.7178964 ↗
  36. Przybocki & Martin. (2001). 2000 NIST speaker recognition evaluation [Artwork Size: 2735785 KB Pages: 2735785 KB]. https://doi.org/10.35111/EX24-J205 ↗
  37. Rabiner, L. R. (1989). A tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2), 257–286. https://doi.org/10.1109/5.18626 ↗
  38. Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2022). Robust speech recognition via large-scale weak supervision. In Proceedings of Machine Learning Research (pp. 28492–28518). PMLR. ↗
  39. Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre-training. Retrieved February 9, 2025, from https://cdn.openai.com/research-covers/language-unsupervised/ language_understanding_paper.pdf Rev Press. (2022, April). Rev improves accuracy by 30% with new ASR model. Retrieved February 6, 2025, from https://www.rev.com/blog/rev-improves-accuracy-by-over-25-with-launch-of-new-v2-asr-model ↗
  40. Robinson, T., & Fallside, F. (1991). A recurrent error propagating network speech recognition system. Computer Speech and Language, 1991(5), 259–274. find
  41. Robinson, T., Hochberg, M., & Renals, S. (1996). The use of recurrent neural networks in continuous speech recognition [Series Title: The Kluwer International Series in Engineering and Computer Science]. In C.-H. Lee, F. K. Soong, & K. K. Paliwal (Eds.), Automatic Speech and Speaker Recognition (pp. 233–258, Vol. 355). Springer US. https://doi.org/10.1007/978-1-4613-1367-0_10 ↗
  42. Saeki, M., Takatsu, H., Kurata, F., Suzuki, S., Eguchi, M., Matsuura, R., Takizawa, K., Yoshikawa, S., & Matsuyama, Y. (2024, September). InteLLA: Intelligent language learning assistant for assessing language proficiency through interviews and roleplays. In T. Kawahara, V. Demberg, S. Ultes, K. Inoue, S. Mehri, D. Howcroft, & K. Komatani (Eds.), Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue (pp. 385–399). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.sigdial-1.34 ↗
  43. Saito, K. (2012). Effects of form-focused instruction and corrective feedback on L2 pronunciation development of /r/ by Japanese learners of English. Language Learning, 62(2), 595–633. https://doi.org/10.1111/j.1467-9922.2011.00639.x ↗
  44. Sakai, T., & Doshita, S. (1961). Phonetic typewriter. The Journal of the Acoustical Society of America, 33(11_Supplement), 1664. https://doi.org/10.1121/1.1936652 ↗
  45. Shadiev, R., & Liu, J. (2023). Review of research on applications of speech recognition technology to assist language learning. ReCALL, 35(1), 74–88. https://doi.org/10.1017/S095834402200012X ↗
  46. Steadman, L., & Williams, W. (2024, October). Ursa 2: Elevating speech recognition across 50+ languages. Retrieved February 4, 2025, from https://www.speechmatics.com/company/articles-and-news/ursa-2-elevating-speech-recognition-across-52-languages ↗
  47. Stolcke, A., & Droppo, J. (2017). Comparing human and machine errors in conversational speech transcription [arXiv:1708.08615 [cs]]. Interspeech 2017, 137–141. https://doi.org/10.21437/Interspeech.2017-1544 ↗
  48. Suzuki, J. (1961). Recognition of Japanese vowels [Publisher: Tokyo: Radio Research Laboratory]. Journal of the Radio Research Laboratory, 8(37). find
  49. Team, R. C. (2024). R: A language and environment for statistical computing. https://www.R-project.org/ ↗
  50. Team, R. (2024). RStudio: Integrated development environment for R. https://posit.co/products/open-source/rstudio/ ↗
  51. Thi-Nhu Ngo, T., Hao-Jan Chen, H., & Kuo-Wei Lai, K. (2024). The effectiveness of automatic speech recognition in ESL/EFL pronunciation: A meta-analysis. ReCALL, 36(1), 4–21. https://doi.org/10.1017/S0958344023000113 "Universal-2: Comprehensive speech-to-text for solving last-mile challenges". (2024, October). Universal-2: Comprehensive speech-to-text for solving last-mile challenges. AssemblyAI. Retrieved February 4, 2025, from https://www.assemblyai.com/research/universal-2 ↗
  52. Vaessen, N. (2018). Jiwer. https://github.com/jitsi/jiwer ↗
  53. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention is all you need [arXiv:1706.03762 [cs]]. 31st Conference on Neural Information Processing Systems, 261–272. https://doi.org/10.48550/arXiv.1706.03762 ↗
  54. Vergyri, D., Lamel, L., & Gauvain, J.-L. (2010). Automatic speech recognition of multiple accented English data, 1652–1655. https://doi.org/10.21437/Interspeech.2010-477 ↗
  55. Weinberger, S. (2015). Speech accent archive. http://accent.gmu.edu ↗
  56. Wills, S., Bai, Y., Tejedor-Garcia, C., Cucchiarini, C., & Strik, H. (2023). Automatic speech recognition of non-native child speech for language learning applications [arXiv:2306.16710 [cs]]. OASIcs, Volume 113, SLATE 2023, 113, 7:1–7:8. https://doi.org/10.4230/OASIcs.SLATE.2023.7 ↗
  57. Xiong, W., Droppo, J., Huang, X., Seide, F., Seltzer, M., Stolcke, A., Yu, D., & Zweig, G. (2017, February). Achieving human parity in conversational speech recognition [arXiv:1610.05256 [cs]]. https://doi.org/10.48550/arXiv. 1610.05256 Zhao, G., Sonsaat, S., Silpachai, A., Lucic, I., Chukharev-Hudilainen, E., Levis, J., & Gutierrez-Osuna, R. (2018). L2-ARCTIC: A non-native English speech corpus. Interspeech 2018, 2783–2787. https://doi.org/10.21437/ Interspeech.2018-1110 Mean MER and SD by L1, Gender, Speaker, and ASR System AssemblyAI Deepgram RevAI Speechmatics Whisper L1 Gender Speaker ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mcguire-2025-automatic-speech-recognition-non-native/