Elamur Research⌕
☰
home / papers / 2022 / Global Performance Disparities Between English-Language Accents in Automatic Speech Recognition

Global Performance Disparities Between English-Language Accents in Automatic Speech Recognition

Paper · 2022 · Alex DiChristofano, Henry Shuster, Shefali Chandra et al.

Summary

The paper asks whether commercial English speech recognition serves speakers worse depending on their background, and whether that pattern tracks geopolitics rather than only individual traits. The authors audit several widely used English ASR services on The Speech Accent Archive. The archive holds recordings of over 2,700 speakers born in 171 countries with 212 first languages. Every participant reads the same elicitation passage, so vocabulary and syntax are held fixed and the measured differences come mainly from pronunciation. The authors model transcription error against speaker covariates, including whether English is the first language and the age at which the speaker began using English. Even with these controls, every service tested makes significantly more errors for speakers born outside NATO countries, meaning countries less aligned with US geopolitical power. The authors place this finding in the history of language standardization as a tool of colonial and national dominance. They argue that the gap will not close by itself through more data, because feedback loops and design choices tend to keep disfavored speakers underrepresented.

Key points

  • Data: The Speech Accent Archive, with 2,713 speakers, 212 first languages and 171 birth countries. All speakers read one fixed text, which isolates pronunciation from word choice.
  • Hypotheses tested:
    • Speakers born in the US with English as a first language get lower error.
    • A later age of onset of English raises error.
    • Alignment of the birth country with the US predicts performance beyond the other covariates.
  • Main finding: after controlling for first-language-related covariates, birth outside a NATO country is linked to significantly worse recognition. This holds for all audited services.
  • The work is explicitly an audit, not a mitigation method. It builds on earlier audits that found gaps by race, British accent stigma, tone-language L1, Indian vs. US accent, and Scottish vs. Californian/New Zealand speakers.
  • The authors list three reasons the gap may persist:
    • Disfavored users opt out of products and data collection.
    • Optimizing average loss can sacrifice small groups even when they are represented in proportion.
    • Ground-truth labels tend to be noisier for those groups.

Interesting ideas

  • I1. Using a single fixed elicitation text for every speaker removes lexical and syntactic confounds, so error differences can be pinned on one factor (pronunciation).
  • I2. Instead of reporting raw per-group error, the authors fit a model with covariates and test whether a group variable still matters once confounds are accounted for.
  • I3. Grouping speakers by a coarse external attribute (geopolitical bloc) exposes structure that per-country breakdowns are too sparse to show.
  • I4. Collected data and labels for disfavored groups are scarcer and noisier, so retraining can quietly reinforce the original bias.
  • I5. Minimizing average loss can trade away accuracy on small subpopulations even when they are represented in proportion.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Build evaluation sets where the same content is rewritten in different varieties or registers, so parser and reader drops can be attributed to surface form rather than content.
  • I2 (learn): When an error slice looks worse (e.g., non-US spelling, learner English), regress errors on sentence length, OOV rate and construction type before concluding that the variety itself is the cause.
  • I3 (learn): Attach provenance metadata (source region, genre, author L1) to treebank and test sentences, and report LAS per aggregated slice to catch systematic weaknesses.
  • I4 (learn): Silver data accepted only when witness parsers agree will underrepresent the varieties they all parse badly, so track the share of each variety among rejected sentences and route those to LLM teacher tiers.
  • I5 (tree): When training the perceptron parser or gparse, monitor the worst-slice LAS alongside the mean, and consider per-slice reweighting so rare UD dialect constructions are not sacrificed.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Washington University in St. LouisSt Louis · United States12 works · 1,028 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksSpeech & spoken language · 480 works

Cite

Cite as Alex DiChristofano, Henry Shuster, Shefali Chandra and Neal Patwari. 2022. Global Performance Disparities Between English-Language Accents in Automatic Speech Recognition. arXiv:2208.01157.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2208.01157 · arXiv 2208.01157

@misc{dichristofano2022global,
  title = {{Global Performance Disparities Between English-Language Accents in Automatic Speech Recognition}},
  author = {Alex DiChristofano, Henry Shuster, Shefali Chandra and Neal Patwari},
  year = {2022},
  doi = {10.48550/arxiv.2208.01157},
  eprint = {2208.01157},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2208.01157}
}

References

52 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alëna Aksënova, Zhehuai Chen, Chung-Cheng Chiu, Daan van Esch, Pavel Golik, Wei Han, Levi King, Bhuvana Ramabhadran, Andrew Rosenberg, Suzan Schwartz, et al. 2022. Accented speech recognition: Benchmarking, pre-training, and diverse data. arXiv preprint arXiv:2205.08014 (2022). ↗
  2. May Pik Yu Chan, June Choe, Aini Li, Yiran Chen, Xin Gao, and Nicole Holliday. 2022. Training and typological bias in ASR performance for world Englishes. In Interspeech 2022. ISCA, 1273–1277. https://doi.org/10.21437/Interspeech.2022-10869 ↗
  3. Shefali Chandra. 2012. The Sexual Life of English: Languages of Caste and Desire in Colonial India. Duke University Press, Durham, NC. find
  4. Iain Connell and Robert Florence. 2015. Burnistoun - Voice Recognition Lift. https://youtu.be/TqAu-DDlINs ↗
  5. Emily Denton. 2019. Critical Perspectives on Computer Vision. https://slideslive.com/38923500/critical-perspectives-on-computer-vision ↗
  6. Pranav Dheram, Murugesan Ramakrishnan, Anirudh Raju, I-Fan Chen, Brian King, Katherine Powell, Melissa Saboowala, Karan Shetty, and Andreas Stolcke. 2022. Toward Fairness in Speech Recognition: Discovery and mitigation of performance disparities. arXiv preprint arXiv:2207.11345 (2022). ↗
  7. Djellel Difallah, Elena Filatova, and Panos Ipeirotis. 2018. Demographics and Dynamics of Mechanical Turk Workers. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining (Marina Del Rey, CA, USA) (WSDM ’18). Association for Computing Machinery, New York, NY, USA, 135–143. https://doi.org/10.1145/3159652.3159661 ↗
  8. Jörg Dollmann, Irena Kogan, and Markus Weißmann. 2019. Speaking Accent-Free in L2 Beyond the Critical Period: The Compensatory Role of Individual Abilities and Opportunity Structures. Applied Linguistics 41, 5 (07 2019), 787–809. https://doi.org/10.1093/applin/amz029 arXiv:https://academic.oup.com/applij/article-pdf/41/5/787/33866356/amz029.pdf ↗
  9. Nina Sun Eidsheim. 2023. Rewriting Algorithms for Just Recognition. Vol. 3. Univ. of California Press, 134. find
  10. Danielle Ensign, Sorelle A. Friedler, Scott Neville, Carlos Scheidegger, and Suresh Venkatasubramanian. 2018. Runaway Feedback Loops in Predictive Policing. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research, Vol. 81), Sorelle A. Friedler and Christo Wilson (Eds.). PMLR, 160–171. https://proceedings.mlr.press/v81/ensign18a.html ↗
  11. James Emil Flege, Murray J. Munro, and Ian R. A. MacKay. 1995. Factors affecting strength of perceived foreign accent in a second language. The Journal of the Acoustical Society of America 97, 5 (1995), 3125–3134. https://doi.org/10.1121/1.413041 arXiv:https://doi.org/10.1121/1.413041 ↗
  12. K.L. Gauba. 1974. Friends and Foes: An Autobiography. Indian Book Company, New Delhi, Delhi. find
  13. Sharon Goldwater, Dan Jurafsky, and Christopher D. Manning. 2008. Which Words Are Hard to Recognize? Prosodic, Lexical, and Disfluency Factors that Increase ASR Error Rates. In Proceedings of ACL-08: HLT. Association for Computational Linguistics, Columbus, Ohio, 380–388. https://aclanthology.org/P08-1044 ↗
  14. Foster R Goss, Suzanne V Blackley, Carlos A Ortega, Leigh T Kowalski, Adam B Landman, Chen-Tan Lin, Marie Meteer, Samantha Bakes, Stephen C Gradwohl, David W Bates, and Li Zhou. 2019. A clinician survey of using speech recognition for clinical documentation in the electronic health record. Intl. Journal of Medical Informatics 130 (2019), 103938. ↗
  15. Harald Hammarström, Robert Forkel, Martin Haspelmath, and Sebastian Bank. 2021. glottolog/glottolog: Glottolog database 4.5. https://doi.org/10.5281/zenodo.5772642 ↗
  16. Drew Harwell. 2018. The accent gap: How Amazon’s and Google’s smart speakers leave certain voices behind. Washington Post (19 July 2018). https://www.washingtonpost.com/graphics/2018/business/alexa-does-not-understand-your-accent/ ↗
  17. Mar Hicks. 2021. When did the fire start? In Your computer is on fire, Thomas S Mullaney, Benjamin Peters, Mar Hicks, and Kavita Philip (Eds.). MIT Press, Chapter 2. ↗
  18. Arthur Hinsvark, Natalie Delworth, Miguel Del Rio, Quinten McNamara, Joshua Dong, Ryan Westerman, Michelle Huang, Joseph Palakapilly, Jennifer Drexler, Ilya Pirkin, et al. 2021. Accented speech recognition: A survey. arXiv preprint arXiv:2104.10747 (2021). ↗
  19. Marek Hlavac. 2018. stargazer: Well-Formatted Regression and Summary Statistics Tables. Central European Labour Studies Institute (CELSI), Bratislava, Slovakia. https://CRAN.R-project.org/package=stargazer R package version 5.2.2. ↗
  20. Sara Hooker. 2021. Moving beyond “algorithmic bias is a data problem”. Patterns 2, 4 (2021), 100241. find
  21. Wiebke Toussaint Hutiri and Aaron Yi Ding. 2022. Bias in Automated Speaker Recognition. In 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT 2022) (Seoul, Republic of Korea) (FAccT ’22). 230–247. https://doi.org/10.1145/3531146.3533089 ↗
  22. Eun Seo Jo and Timnit Gebru. 2020. Lessons from archives: Strategies for collecting sociocultural data in machine learning. In Conference on Fairness, Accountability, and Transparency (FAccT 2020). 306–316. ↗
  23. Braj B. Kachru. 1986. The power and politics of English. World Englishes 5, 2-3 (1986), 121–140. ↗
  24. Allison Koenecke, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups, John R. Rickford, Dan Jurafsky, and Sharad Goel. 2020. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences 117, 14 (2020), 7684–7689. https://doi.org/10.1073/pnas.1915768117 arXiv:https://www.pnas.org/content/117/14/7684.full.pdf ↗
  25. Halcyon M. Lawrence. 2021. Siri disciplines. In Your Computer Is On Fire, Thomas S. Mullaney, Benjamin Peters, Mar Hicks, and Kavita Philip (Eds.). MIT Press, 179–198. https://mitpress.mit.edu/books/your-computer-fire ↗
  26. Rosina Lippi-Green. 2012. English with an accent: Language, ideology, and discrimination in the United States (2 ed.). Routledge. find
  27. Zhe Liu, Irina-Elena Veliche, and Fuchun Peng. 2022. Model-Based Approach for Measuring the Fairness in ASR. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2022). 6532–6536. ↗
  28. Nina Markl. 2022. Language variation and algorithmic bias: understanding algorithmic bias in British English automatic speech recognition. In ACM Conference on Fairness, Accountability, and Transparency (FAccT 2022). 521–534. find
  29. Mari J. Matsuda. 1991. Voices of America: Accent, Antidiscrimination Law, and a Jurisprudence for the Last Reconstruction. The Yale Law Journal 100, 5 (1991), 1329—-1407. https://doi.org/10.2307/796694 ↗
  30. Bonnie McElhinny and Monica Heller. 2020. The Linguistic Intimacy of Five Continents. , 129–152 pages. https://doi.org/10.1093/oxfordhb/9780190845995.013.8 ↗
  31. Josh Meyer, Lindy Rauchenstein, Joshua D. Eisenberg, and Nicholas Howell. 2020. Artie Bias Corpus: An Open Dataset for Detecting Demographic Bias in Speech Applications. In Proceedings of the 12th Language Resources and Evaluation Conference. European Language Resources Association, Marseille, France, 6462–6468. https://aclanthology.org/2020.lrec-1.796 ↗
  32. Matt Mitchell. 2018. If Alexa was Southern. https://www.youtube.com/watch?v=q3j6708kzEY ↗
  33. Shakir Mohamed, Marie-Therese Png, and William Isaac. 2020. Decolonial AI: Decolonial Theory as Sociotechnical Foresight in Artificial Intelligence. Philosophy & Technology 33, 4 (July 2020), 659–684. https://doi.org/10.1007/s13347-020-00405-8 ↗
  34. Andrew Morris. 2002. An information theoretic measure of sequence recognition performance. Technical Report. EPFL. http://infoscience.epfl.ch/record/82766 ↗
  35. Andrew C. Morris, Viktoria Maier, and Phil D. Green. 2004. From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition. In Interspeech. https://doi.org/10.21437/Interspeech.2004-668 ↗
  36. Alene Moyer. 2007. Do Language Attitudes Determine Accent? A Study of Bilinguals in the USA. Journal of Multilingual and Multicultural Development 28, 6 (2007), 502–518. https://doi.org/10.2167/jmmd514.0 arXiv:https://doi.org/10.2167/jmmd514.0 ↗
  37. Dariusz Mrozek, Szymon Kwaśnicki, Vaidy Sunderam, Bożena Małysiak-Mrozek, Krzysztof Tokarz, and Stanisław Kozielski. 2021. Comparison of Speech Recognition and Natural Language Understanding Frameworks for Detection of Dangers with Smart Wearables. In International Conference on Computational Science. Springer, 471–484. find
  38. Veena Naregal. 2001. Language Politics, Elites and the Public Sphere: Western India under Colonialism. Permanent Black, New Delhi, Delhi. find
  39. Julia Nee, Genevieve Macfarlane Smith, Alicia Sheares, and Ishita Rustagi. 2022. Linguistic justice as a framework for designing, developing, and managing natural language processing tools. Big Data & Society 9, 1 (2022), 20539517221090930. ↗
  40. Nanjala Nyabola. 2018. Digital Democracy, Analogue Politics: How the Internet Era is Transforming Politics in Kenya. Zed Books, London, England. find
  41. Gareth Price. 2014. English for all? Neoliberalism, globalization, and language policy in Taiwan. Language in Society 43, 5 (2014), 567–589. find
  42. Rashida Richardson, Jason M. Schultz, and Kate Crawford. 2019. Dirty Data, Bad Predictions: How Civil Rights Violations Impact Police Data, Predictive Policing Systems, and Justice. New York University Law Review Online 94 (2019), 15. find
  43. Joel Ross, Lilly Irani, M. Six Silberman, Andrew Zaldivar, and Bill Tomlinson. 2010. Who Are the Crowdworkers? Shifting Demographics in Mechanical Turk. In CHI ’10 Extended Abstracts on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI EA ’10). Association for Computing Machinery, New York, NY, USA, 2863–2872. https://doi.org/10.1145/1753846.1753873 ↗
  44. Rachael Tatman. 2017. Gender and Dialect Bias in YouTube’s Automatic Captions. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing. Association for Computational Linguistics, Valencia, Spain, 53–59. https://doi.org/10.18653/v1/W17-1606 ↗
  45. Rachael Tatman and Conner Kasten. 2017. Effects of Talker Dialect, Gender & Race on Accuracy of Bing Speech and YouTube Automatic Captions. In Proc. Interspeech 2017. 934–938. https://doi.org/10.21437/Interspeech.2017-1746 ↗
  46. Mithra Venkatesan, Lini Mohan, Nisith Manika, and Aishwarya Mathapati. 2021. Voice-Controlled Intelligent Wheelchair for Quadriplegics. In Computer Communication, Networking and IoT. Springer, 41–51. ↗
  47. Eugene Weber. 1976. Peasants into Frenchmen: The Modernization of Rural France, 1870-1914. Stanford University Press, Palo Alto, CA. find
  48. Steven Weinberger. 2015. Speech Accent Archive. http://accent.gmu.edu George Mason University. ↗
  49. Meredith Whittaker, Meryl Alper, Cynthia L Bennett, Sara Hendren, Liz Kaziunas, Mara Mills, Meredith Ringel Morris, Joy Rankin, Emily Rogers, Marcel Salas, and Sarah Myers West. 2019. Disability, Bias, and AI. Technical Report. AI Now Institute. find
  50. Li Zhou, Suzanne V Blackley, Leigh Kowalski, Raymond Doan, Warren W Acker, Adam B Landman, Evgeni Kontrient, David Mack, Marie Meteer, David W Bates, et al. 2018. Analysis of errors in dictated clinical documents assisted by speech recognition software and professional transcriptionists. JAMA Network Open 1, 3 (2018). ↗
  51. James Zou and Londa Schiebinger. 2018. AI can be sexist and racist — it’s time to make it fair. Nature 559, 7714 (Jul 2018), 324–326. https://doi.org/10.1038/d41586-018-05707-8 ↗
  52. Juan Zuluaga-Gomez, Amrutha Prasad, Iuliia Nigmatulina, Seyyed Saeed Sarfjoo, Petr Motlicek, Matthias Kleinert, Hartmut Helmke, Oliver Ohneiser, and Qingran Zhan. 2023. How does pre-trained Wav2Vec 2.0 perform on domain-shifted ASR? An extensive benchmark on air traffic control communications. In 2022 IEEE Spoken Language Technology Workshop (SLT). 205–212. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dichristofano-2022-global-performance-disparities-between-english/