Elamur Research⌕
☰
home / papers / 2023 / Investigating the Sensitivity of Automatic Speech Recognition Systems to Phonetic Variation in L2 Englishes

Investigating the Sensitivity of Automatic Speech Recognition Systems to Phonetic Variation in L2 Englishes

Paper · 2023 · Emma O'Neill and Julie Carson-Berndsen

Summary

The paper asks why ASR systems trained mostly on "standard" English perform worse on non-native speech, and whether these failures can be traced to specific phonetic realisations. The authors ran a pretrained wav2vec 2.0 model, which has no word-level language model, on the phonemically annotated subset of the L2-Arctic corpus. They converted both the read prompts and the ASR transcripts into ARPAbet phoneme strings using the CMU Pronouncing Dictionary, with a grapheme-to-phoneme tool for non-words. The two sequences were aligned with an edit distance whose costs come from a phoneme similarity matrix. The alignments yield per-speaker confusion matrices of substitutions, insertions and deletions. When these matrices are treated as vectors and clustered, speakers who share an L1 group together, so the model's errors are systematic rather than random. The detected substitutions broadly agree with human annotators, although the rates can differ. The practical payoff is twofold: data collection for fine-tuning can target specific weaknesses, and annotators correcting ASR output can be told which errors to expect.

Key points

  • The data is the annotated subset of the L2-Arctic corpus: 24 speakers, with four (two male, two female) for each of six L1s (Arabic, Hindi, Korean, Mandarin, Spanish, Vietnamese). This comes to 3599 utterances and about 219.81 minutes of read Arctic prompts.
  • The ASR is wav2vec 2.0, fine-tuned on 100 hours of Librispeech. Because it has no word-level LM, it can output non-words that reflect what was actually pronounced, such as "simbol" for "simple".
  • The prompt pronunciations from the CMU dictionary are explicitly framed as what the ASR expects, not as correct pronunciations.
  • For an L1 Hindi speaker, the most frequent confusion was expected /T/ being recognised as /D/. The authors attribute this to retroflex or unaspirated word-initial realisations of /T/.
  • Agreement with human annotators is only partial in rate: one /TH/ substitution appears in 7.5% of cases in the ASR output but 13.1% in human labels. The authors therefore recommend ASR as an aid to annotation, not a replacement.

Interesting ideas

  • I1. Plain Levenshtein alignment can produce linguistically odd pairings, so the authors weight each edit by how plausible it is, making cheap operations like /HH/ deletion win ties. This gives alignments that are meaningful for analysis, not just minimal.
  • I2. The gold reference is treated as the system's expectation rather than ground truth, which keeps the analysis honest about variation.
  • I3. Each speaker's confusion matrix is flattened into a vector and clustered, showing that error profiles carry structure that matches speaker groups.
  • I4. The system's error patterns are checked against human judgements to decide whether the tool can annotate on its own or only assist.
  • I5. Systematic errors suggest a small, targeted data collection that elicits exactly the problematic productions, and the gains should then carry over to similar speakers.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): When aligning our tokenizer and lemmatizer output against treebank tokens, costs could reflect likely mismatches (clitic splits, hyphen joins). Error reports would then name the real cause rather than an arbitrary edit path.
  • I2 (tree): When converting between UD dialects or comparing gparse with the perceptron parser, differences from a target treebank could be logged as convention divergences. They would then not automatically count as parse errors.
  • I3 (learn): Per-document deprel or POS confusion vectors from our witness parsers could be clustered by genre or source. This would show which text types share systematic failures before we trust their silver data.
  • I4 (learn): Before promoting silver trees or LLM-teacher labels, we could compare their confusion profile with a small gold sample. Where the rates diverge, the output would stay assist-only.
  • I5 (learn): Construction types that recur in our confusion matrices could drive generation of targeted training sentences through self-play or the LLM teacher. This would be cheaper than broad re-annotation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University College DublinDublin · Ireland15 works · 1,643 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksPsychology · 1,649 worksExperimental and Cognitive Psychology · 930 worksSignal Processing · 493 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,379 worksSpeech & spoken language · 519 works

Cite

Cite as Emma O'Neill and Julie Carson-Berndsen. 2023. Investigating the Sensitivity of Automatic Speech Recognition Systems to Phonetic Variation in L2 Englishes. arXiv:2305.07389.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.07389 · arXiv 2305.07389

@misc{oneill2023investigating,
  title = {{Investigating the Sensitivity of Automatic Speech Recognition Systems to Phonetic Variation in L2 Englishes}},
  author = {Emma O'Neill and Julie Carson-Berndsen},
  year = {2023},
  doi = {10.48550/arxiv.2305.07389},
  eprint = {2305.07389},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.07389}
}

References

24 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Baevski, Alexei, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems 33:12449–12460.
  2. Broselow, Ellen, Su-I Chen, and Chilin Wang. 1998. The emergence of the unmarked in second language phonology. Studies in second language acquisition 20:261–280. find
  3. CMUSphinx. 2016. Grapheme-to-phoneme tool based on sequence-to-sequence learning. find
  4. Huang, Meichan, and Lucy Pickering. 2014. Revisiting the pronunciation of English by speakers from Mainland China. In Pronunciation in Second Language Learning and Teaching Conference (ISSN 2380-9566), 206. find
  5. Jimerson, Robbie, and Emily Prud’hommeaux. 2018. Asr for documenting acutely under-resourced indigenous languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). find
  6. Khanal, Subash, Michael T Johnson, Mohammad Soleymanpour, and Narjes Bozorg. 2021. Mispronunciation detection and diagnosis for Mandarin accented English speech. In 2021 International Conference on Speech Technology and Human-Computer Dialogue (SpeD), 62–67. IEEE. find
  7. Koenecke, Allison, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups, John R Rickford, Dan Jurafsky, and Sharad Goel. 2020. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences 117:7684–7689. find
  8. Kominek, John, and Alan W Black. 2004. The CMU Arctic speech databases. In Fifth ISCA workshop on speech synthesis. find
  9. Levenshtein, Vladimir I. 1966. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, volume 10.8, 707–710. Soviet Union. find
  10. Van der Maaten, Laurens, and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research 9.11. find
  11. MacQueen, J. 1967. Classification and analysis of multivariate observations. In 5th Berkeley Symp. Math. Statist. Probability, 281–297. University of California Los Angeles LA USA. find
  12. Markl, Nina. 2022. (commercial) automatic speech recognition as a tool in sociolinguistic research. University of Pennsylvania Working Papers in Linguistics 28:11. find
  13. Michaud, Alexis, Oliver Adams, Trevor Cohn, Graham Neubig, and Séverine Guillaume. 2018. Integrating automatic transcription into the language documentation workflow: Experiments with na data and the persephone toolkit. Language Documentation & Conservation 12:393–429. find
  14. Nogita, Akitsugu, and Yanan Fan. 2012. Not vowel epenthesis: Mandarin and Japanese ESL learners’ production of English consonant clusters. Working Papers of the Linguistics Circle 22:1–26. find
  15. Omachi, Motoi, Yuya Fujita, Shinji Watanabe, and Matthew Wiesner. 2021. End-to-end asr to jointly predict transcriptions and linguistic annotations. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1861–1871. find
  16. O’Neill, Emma, and Julie Carson-Berndsen. 2019. The effect of phoneme distribution on perceptual similarity in English. INTERSPEECH 2019 1941–1945. find
  17. O’Neill, Emma, Robert Young, Elsa Thiaville, Muireann MacCarthy, Julie Carson-Berndsen, and Anthony Ventresque. 2020. S-capade: Spelling correction aimed at particularly deviant errors. In International Conference on Statistical Language and Speech Processing, 85–96. Springer. find
  18. Panayotov, Vassil, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: An ASR corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 5206–5210. IEEE. find
  19. Reddy, Sravana, and James N Stanford. 2015. Toward completely automated vowel extraction: Introducing darla. Linguistics Vanguard 1:15–28. find
  20. Siqi, Li, and Andrew Sewell. 2012. Phonological features of China English. Asian Englishes 15:80–101. find
  21. Tatman, Rachael, and Conner Kasten. 2017. Effects of Talker Dialect, Gender & Race on Accuracy of Bing Speech and YouTube Automatic Captions. In Interspeech, 934–938. find
  22. Wagner, Robert A, and Michael J Fischer. 1974. The string-to-string correction problem. JACM 21:168–173. find
  23. Weide, Robert L. 1998. The CMU pronouncing dictionary. find
  24. Zhao, Guanlong, Sinem Sonsaat, Alif Silpachai, Ivana Lucic, Evgeny Chukharev-Hudilainen, John Levis, and Ricardo Gutierrez-Osuna. 2018. L2-ARCTIC: A non-native English speech corpus. In Interspeech, 2783– 2787. ADAPT Research Centre School of Computer Science University College Dublin Dublin, Ireland D04 V1W8 emma.l.oneill@ucdconnect.ie julie.berndsen@ucd.ie find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/o-neill-2023-investigating-sensitivity-automatic-speech-recognition/