Elamur Research⌕
☰
home / papers / 2022 / Language-specific Effects on Automatic Speech Recognition Errors for World Englishes

Language-specific Effects on Automatic Speech Recognition Errors for World Englishes

Paper · 2022 · June Choe, Yiran Chen, May Pik Yu Chan et al.

Summary

The paper asks why commercial speech recognition serves some English speakers worse than others. Earlier work mostly reported a single Word Error Rate (WER), and that number cannot show which sounds fail or why. The authors tested Otter, a captioning engine used inside Zoom, on 1,227 readings of the same passage from the Speech Accent Archive. The recordings cover 22 English varieties: half officially supported by Otter, half not. They first aligned the transcripts to the reference text at the word level. They then mapped the word-level mistakes to phones with the CMU Pronouncing Dictionary, which yields a Phone Error Rate (PER) and a list of individual sound substitutions. These substitutions were grouped into categories drawn from phonology (consonant clusters, stop voicing, regions of the vowel space). Each category was compared with typological properties of the speakers' first languages. The error types turned out to be predictable from the speaker's native sound system. This points ASR adaptation toward specific phonological targets rather than broad demographic labels.

Key points

  • Data come from the Speech Accent Archive (Weinberger, 2015), where every speaker reads the 77-word "Please Call Stella" passage, about 30 seconds per recording. The sample has 1,227 speakers from 11 Otter-supported and 11 unsupported varieties.
  • The recordings were joined into one 9.5-hour file and uploaded in chunks of up to 4 hours.
    • About 10% of Otter's speaker splits were wrong.
    • Two annotators fixed these with 99.8% agreement, and a third settled their disagreements.
  • The PER pipeline found 2,382 consonant substitutions. The analysis focused on the two most frequent kinds:
    • Cluster errors ("ask" heard as "asked").
    • Voicing errors ("bag" heard as "back").
  • Each L1 was coded for whether it allows consonant clusters in onset and in coda, giving four types (Onset-Coda, Onset-only, Coda-only, Neither).
    • Speakers whose L1 forbids clusters got more Cluster errors.
    • Speakers whose L1 contrasts aspiration rather than voicing got more stop-voicing errors.
  • Vowel errors followed the L1 vowel inventory:
    • Smaller inventories produced errors in predictable zones of the vowel space.
    • Inventories similar to English produced fewer errors.
    • Native-variety patterns also showed up, such as errors around high vowels in Canadian English, linked to Canadian raising.

Interesting ideas

  • I1. The authors push a coarse word-level score down to finer units with a pronunciation lexicon, so that deleting a whole word costs several phones while a one-suffix slip costs just one. This makes error severity proportional instead of all-or-nothing.
  • I2. Error categories are not mined blindly; they are fixed in advance from linguistic theory and then checked against coded properties of the input's source. This turns error analysis into hypothesis testing that yields actionable targets.
  • I3. Using one fixed passage for every speaker keeps content constant, so any difference in errors can be traced to the variety rather than to vocabulary or topic.
  • I4. The sample deliberately balances supported and unsupported varieties, which helps separate the effect of training exposure from the effect of linguistic structure.
  • I5. Looking at substitution patterns rather than a single rate shows where a system is confused, much like a confusion map over a feature space (here the vowel chart).

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Lemmatizer and morphology errors could be scored the same way, so a wrong inflection ("ask" vs "asked") counts less than a wrong lemma, giving a more honest picture of crate en quality.
  • I2 (learn): Evaluation sets could tag each text with source features (e.g., the writer's L1 or the English variety) and test whether specific parser or tagger error classes correlate with them, instead of reporting only global LAS.
  • I3 (tree): A small parallel set of the same story written in several English varieties (spelling, idiom, L2-style syntax) would isolate dialect effects on gparse and the perceptron parser.
  • I4 (learn): Silver-data evaluations could pair in-distribution and out-of-distribution sources as a built-in control, so improvements from self-induced repair rules are not just familiarity with the training genre.
  • I5 (tree): A confusion matrix over UD relations, broken down by text source, would show which dependency labels get swapped (e.g., obl vs obj) and steer targeted fixes in UD dialect conversion.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 worksSignal Processing · 452 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksSpeech & spoken language · 480 works

Cite

Cite as June Choe, Yiran Chen, May Pik Yu Chan, Aini Li, Xin Gao and Nicole Holliday. 2022. Language-specific Effects on Automatic Speech Recognition Errors for World Englishes. Proceedings of the 29th International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{choe2022language,
  title = {{Language-specific Effects on Automatic Speech Recognition Errors for World Englishes}},
  author = {June Choe, Yiran Chen, May Pik Yu Chan, Aini Li, Xin Gao and Nicole Holliday},
  booktitle = {Proceedings of the 29th International Conference on Computational Linguistics},
  year = {2022},
  doi = {10.48448/wv0k-hr42},
  url = {https://doi.org/10.48448/wv0k-hr42}
}

References

49 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martine Adda-Decker and Lori Lamel. 2005. Do speech recognizers prefer female speakers? In Ninth European Conference on Speech Communication and Technology. find
  2. Alëna Aksënova, Daan van Esch, James Flynn, and Pavel Golik. 2021. How might we create better benchmarks for speech recognition? In Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future, Online. Association for Computational Linguistics. find
  3. International Phonetic Association, International Phonetic Association Staff, et al. 1999. Handbook of the International Phonetic Association: A guide to the use of the International Phonetic Alphabet. Cambridge University Press. find
  4. Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2015. Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1):1– 48. ↗
  5. Catherine T. Best et al. 1994. The emergence of native-language phonological influences in infants: A perceptual assimilation model. The development of speech perception: The transition from speech sounds to spoken words, 167(224):233–277. find
  6. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of "bias" in nlp. arXiv preprint. arXiv:2005.14050. ↗
  7. Paul Boersma. 2006. Praat: doing phonetics by computer. http://www.praat. org/. ↗
  8. Marcely Zanon Boito, Laurent Besacier, Natalia Tomashenko, and Yannick Estève. 2022. A study of gender impact in self-supervised models for speech-to-text systems. arXiv preprint arXiv:2204.01397. ↗
  9. Jack K. Chambers. 1973. Canadian raising. Canadian Journal of Linguistics/Revue canadienne de linguistique, 18(2):113–135. find
  10. May Pik Yu Chan, June Choe, Aini Li, Yiran Chen, Xin Gao, and Nicole Holliday. 2022. Training and typological bias in asr performance for world englishes. In Proceedings of the 23rd Conference of the International Speech Communication Association. find
  11. Stephen Pit Corder. 1983. A role for the mother tongue. Language transfer in language learning, 1:85–97. find
  12. Hans-Wilhelm Dechert and Manfred Raupach. 1989. Transfer in language production. Praeger. find
  13. Jürg Fleischer and Stephan Schmid. 2006. Zurich german. Journal of the International Phonetic Association, 36(2):243–253. find
  14. Gillian Elizabeth Scott Gallagher. 2010. The perceptual basis of long-distance laryngeal restrictions. Ph.D. thesis, Massachusetts Institute of Technology. find
  15. Michael Magnus Thyne Henderson. 1972. Dari (Kabul Persian) Phonology. The University of Wisconsin-Madison. find
  16. Dirk Hovy and Shrimai Prabhumoye. 2021. Five sources of bias in natural language processing. Language and Linguistics Compass, 15(8):e12432. ↗
  17. James P. Kirby. 2011. Vietnamese (Hanoi Vietnamese). Journal of the International Phonetic Association, 41(3):381–392. ↗
  18. Allison Koenecke, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups, John R Rickford, Dan Jurafsky, and Sharad Goel. 2020. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, 117(14):7684–7689. ↗
  19. Martin Kramer. 2009. The phonology of Italian. OUP Oxford. ↗
  20. Laurence Labrune. 2012. The phonology of Japanese. Oxford University Press. ↗
  21. Allen Lai. 2021. Supported languages. Otter.ai Help Center. find
  22. Dave Lee. 2021. The next big tech battle: Amazon’s bet on healthcare begins to take shape. Financial Times. find
  23. Xinjian Li, Siddharth Dalmia, Juncheng Li, Matthew Lee, Patrick Littell, Jiali Yao, Antonios Anastasopoulos, David R. Mortensen, Graham Neubig, Alan W Black, and Florian Metze. 2020. Universal phone recognition with a multilingual allophone system. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8249–8253. ↗
  24. Rosina Lippi-Green. 2011. English with accents: Language, ideology, and discrimination in the United States. Routledge, New York. find
  25. Shahrzad Mahootian and Lewis Gebhardt. 1997. Persian (descriptive grammars). London: Routledge. doi, 10:9780203192887. find
  26. Nina Markl and Stephen Joseph McNulty. 2022. Language technology practitioners as language managers: arbitrating data bias and predictive bias in asr. arXiv preprint arXiv:2202.12603. ↗
  27. Joshua L. Martin and Kevin Tang. 2020. Understanding racial disparities in automatic speech recognition: The case of habitual "be". In INTERSPEECH, pages 626–630. find
  28. Rachael Tatman. 2017. Gender and dialect bias in youtube’s automatic captions. In Proceedings of the first ACL workshop on ethics in natural language processing, pages 53–59.
  29. Iqbal Mazhar and Mazhar Ranjha. 2012. Urdu syllable: Templates and constraints. In Proceeding of the Conference on Language and technology 2012. find
  30. Rachael Tatman. 2020. Sociolinguistic variation and automatic speech recognition: Challenges and approaches. In 2020 Annual Meeting. AAAS. find
  31. Josh Meyer, Lindy Rauchenstein, Joshua D. Eisenberg, and Nicholas Howell. 2020. Artie bias corpus: An open dataset for detecting demographic bias in speech applications. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 6462–6468.
  32. Rachael Tatman and Conner Kasten. 2017. Effects of talker dialect, gender & race on accuracy of bing speech and youtube automatic captions. In Interspeech, pages 934–938. find
  33. Simone Mikuteit and Henning Reetz. 2007. Caught in the ACT: The timing of aspiration and voicing in East Bengali. Language and speech, 50(2):247–277. ↗
  34. Maryam Najafian and Martin Russell. 2020. Automatic accent identification as an analytical tool for accent robust automatic speech recognition. Speech Communication, 122:44–55. ↗
  35. Manjari Ohala. 1983. Aspects of Hindi phonology. Motilal Banarsidass Publishers Pvt. Limited. find
  36. James Otto and David Kahle. 2022. ggdensity: Interpretable Bivariate Density Visualization with ’ggplot2’. Https://jamesotto852.github.io/ggdensity/, https://github.com/jamesotto852/ggdensity/. ↗
  37. Olga Petrova, Rosemary Plapp, Catherine Ringen, and Szilárd Szentgyörgyi. 2006. Voice and aspiration: Evidence from Russian, Hungarian, German, Swedish, and Turkish. Walter de Gruyter. ↗
  38. Robin Thelwall and M. Akram Sa’Adeddin. 1990. Arabic. Journal of the International Phonetic Association, 20(2):37–39. find
  39. Lenka Tranová. 2016. The voicing contrast in Vietnamese English. Univerzita Karlova, Filozofická fakulta. find
  40. Alicia Beckford Wassink, Cady Gansen, and Isabel Bartholomew. 2022. Uneven success: automatic speech recognition and ethnicity-related dialects. Speech Communication, 140:50–70. ↗
  41. Steven Weinberger. 2015. Speech accent archive. george mason university. http://accent.gmu.edu. ↗
  42. Barbara Wheatley and Joseph Picone. 1991. Voice Across America: Toward robust speaker-independent speech recognition for telecommunications applications. Digital Signal Processing, 1(2):45–63. ↗
  43. Jiahong Yuan, Mark Liberman, et al. 2008. Speaker identification on the scotus corpus. Journal of the Acoustical Society of America, 123(5):3878. find
  44. Jean-Paul G. Potet. 1995. Tagalog monosyllabic roots. Oceanic linguistics, pages 345–374. ↗
  45. Manish Raghavan, Solon Barocas, Jon Kleinberg, and Karen Levy. 2020. Mitigating bias in algorithmic hiring: Evaluating claims and practices. In Proceedings of the 2020 conference on fairness, accountability, and transparency, pages 469–481. ↗
  46. Majdi Sawalha and Mohammad Abushariah. 2013. The effects of speakers’ gender, age, and region on overall performance of Arabic automatic speech recognition systems using the phonetically rich and balanced Modern Standard Arabic speech corpus. In Proceedings of the 2nd Workshop of Arabic Corpus Linguistics WACL-2. Leeds. find
  47. SIL International. 2022. Ethnologue: English. find
  48. Shruti Sircar and Sonali Nag. 2013. 19 akshara–syllable mappings in bengali: a language-specific skill for reading. South and Southeast Asian psycholinguistics, page 202. find
  49. Craig D. Soderberg and Kenneth S. Olson. 2008. Indonesian. Journal of the International Phonetic Association, 38(2):209–213. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/choe-2022-language-specific-effects-automatic-speech/