Elamur Research⌕
☰
home / papers / 2023 / Accents in Speech Recognition through the Lens of a World Englishes Evaluation Set

Accents in Speech Recognition through the Lens of a World Englishes Evaluation Set

Paper · 2023 · Miguel Del Río, Corey Miller, Ján Profant et al.

Summary

Speech recognizers are less accurate on accented English, and the field lacks a shared benchmark for comparing fixes. The authors, mostly from Rev.com, first review how accent labels are usually assigned: self-description, expert annotation, geographic inference and automatic labeling. They argue that existing accent sets such as Common Voice, the Artie Bias dataset and Casual Conversations are often read speech, single-speaker or unevenly distributed. In response they release Earnings-22, a free corpus of English earnings calls from companies around the world, grouped into seven regional accent clusters. The recordings are spontaneous, involve several speakers and come from a domain where customers really order transcripts. Comparing commercial ASR systems on this corpus shows that word error rate varies with the speakers' region. The authors then test two remedies. Custom vocabulary (user-supplied term lists) helps across varieties. A multilingual English–Spanish ("Spanglish") model helps mainly where the accent is shaped by the speakers' native language. The work matters because it gives a realistic, reusable yardstick for accent bias.

Key points

  • Earnings-22 contains 125 files and about 119 hours of earnings-call audio, covering 7 regional accent groupings, and is publicly available.
  • In 20,000 hours of Rev.com audio, more than 90% of 3-minute segments had multiple speakers. The authors use this to argue that single-speaker test sets are unrealistic.
  • Common Voice self-reported accents are messy: Reid & Williams normalized 14,822 distinct English accent entries down to 164.
  • For "Colombia" in the Spanish/Portuguese partition, F1 rose from 0.64 (English model) to 0.78 (English model + CV or Spanglish model alone) and 0.81 (Spanglish model + CV).
  • For "EBITDA", the English model reached F1 0.82 on the inner-circle partition but only 0.35 on the Spanish/Portuguese partition. Custom vocabulary raised these to 0.90 and 0.47. The Spanglish model did not beat the English one on the accented partition.

Interesting ideas

  • I1. Build the benchmark to match real use (spontaneous, multi-speaker, in-domain) rather than convenient read speech, so that scores predict field performance.
  • I2. Pair the item that should benefit (the Spanish-origin "Colombia") with a control item expected not to benefit (the acronym "EBITDA"), so a gain can be attributed to the mechanism rather than to general improvement.
  • I3. Score a single confusable pair (Colombia vs. Columbia) with false positives and correct accepts, instead of relying only on corpus-wide error rates, which hide targeted failures.
  • I4. Let users inject a short domain term list at inference time; this boosts rare names and jargon without retraining.
  • I5. Split results by language variety, using labels that come cheaply from metadata such as company headquarters, while remembering that location does not guarantee how someone speaks.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Test readers on texts written like the material users actually bring, not only on clean school-style passages, so that reported accuracy carries over.
  • I2 (learn): When a new rule or teacher tier is credited with a gain, also measure a matched control set where it should not help, as a built-in negative control.
  • I3 (world): Track entity-resolution confusions for specific near-identical names (country vs. university) as their own small probes in the knowledge graph.
  • I4 (tokens): Accept a per-document lexicon of names and terms, so the tokenizer and tagger treat unknown proper nouns and acronyms correctly instead of guessing.
  • I5 (tree): Report parser scores separately for different World Englishes text sources, so a single average does not hide weak spots in non-inner-circle varieties.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Northwestern UniversityEvanston · United States41 works · 2,194 citations

Also on the byline: Walgreens.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 worksSignal Processing · 452 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,263 worksEvaluation & benchmarks · 1,220 worksSpeech & spoken language · 480 works

Cite

Cite as Miguel Del Río, Corey Miller, Ján Profant, Jennifer Drexler-Fox, Quinn Mcnamara, Nishchal Bhandari, Natalie Delworth, Ilya Pirkin, Migüel Jetté, Shipra Chandra, Peter Ha and Ryan Westerman. 2023. Accents in Speech Recognition through the Lens of a World Englishes Evaluation Set. Research in Language 21.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18778/1731-7533.21.3.02

@inproceedings{rio2023accents,
  title = {{Accents in Speech Recognition through the Lens of a World Englishes Evaluation Set}},
  author = {Miguel {Del Río}, Corey Miller, Ján Profant, Jennifer Drexler-Fox, Quinn Mcnamara, Nishchal Bhandari, Natalie Delworth, Ilya Pirkin, Migüel Jetté, Shipra Chandra, Peter Ha and Ryan Westerman},
  booktitle = {Research in Language 21},
  year = {2023},
  doi = {10.18778/1731-7533.21.3.02},
  url = {https://doi.org/10.18778/1731-7533.21.3.02}
}

References

40 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ardila, R., Branson, M., Davis, K., Henretty, M., Kohler, M., Meyer, J., Morais, R., Saunders, L., Tyers, F. M., & Weber, G.. (2020). Common Voice: A massively-multilingual speech corpus. Proceedings of the 12th Conference on Language Resources and Evaluation, pp. 4218-4222. ↗
  2. Arons, B. (1992). A Review of the Cocktail Party Effect. AVIOS. find
  3. Baese-Berk, M. M., McLaughlin, D. J. & McGowan, K. B. (2020). Perception of non-native speech. Language and Linguistics Compass, pp. 1-20. https://doi.org/10.1111/lnc3.12375 ↗
  4. Chang, X., Qian, Y., Yu, K. & Watanabe, S. (2019). End-To-End Monaural Multi-Speaker ASR System Without Pretraining. Proceedings of ICASSP. https://doi.org/10.1109/ICASSP.2019.8682822 ↗
  5. Chiswick, B. R. and Miller P. W. (2005). Linguistic distance: A quantitative measure of the distance between English and other languages. Journal of Multilingual and Multicultural Development, vol. 26, no. 1, pp. 1–11. https://doi.org/10.1080/14790710508668395 ↗
  6. Del Río, M., Delworth, N., Westerman, R., Huang, M., Bhandari, N., Palakapilly, J., McNamara, Q., Dong, J., Zelasko, Z., and Jetté, M. (2021). “Earnings-21: A Practical Benchmark for ASR in the Wild,” in Proc. Interspeech 2021, pp. 3465–3469. https://doi.org/10.21437/Interspeech.2021-1915 ↗
  7. Drexler-Fox, J. & Delworth, N. (2022). Improving contextual recognition of rare words with an alternate spelling prediction model. Proceedings of Interspeech. find
  8. Gabler, P., Geiger, B. C., Schuppler, B. & Kern, R. (2023). Reconsidering Read and Spontaneous Speech: Causal Perspectives on the Generation of Training Data for Automatic Speech Recognition. Information, 14, 137. https://doi.org/10.3390/info14020137 ↗
  9. Gandhi, S., Von Platen, P., & Rush, A. M. (2022). ESB: A Benchmark for Multi-Domain End-to-End Speech Recognition. arXiv preprint arXiv:2210.13352. ↗
  10. Goldwater, S., Jurafsky, D., and Manning, C. D. (2010). “Which words are hard to recognize? prosodic, lexical, and disfluency factors that increase speech recognition error rates,” Speech Communication, vol. 52, no. 3, pp. 181–200. https://doi.org/10.1016/j.specom.2009.10.001 ↗
  11. Good, P. I. (2004). Permutation, Parametric, and Bootstrap Tests of Hypotheses. Springer Series in Statistics. Springer-Verlag. find
  12. Hazirbas, C., Bitton, J., Dolhansky, B., Pan, J., Gordo, A. & Ferrer, C. C. (2021). Towards measuring fairness in AI: the Casual Conversations dataset. ArXiv. find
  13. Hazirbas, C., Bang, Y., Yu, T., Assar, P., Porgali, B., Albiero, V., Hermanek, S., Pan, J., McReynolds, E., Bogen, M., Fung, P. & Ferrer, C. C. (2022). Casual Conversations v2: Designing a large consent-driven dataset to measure algorithmic bias and robustness. https://doi.org/10.1109/TBIOM.2021.3132237 ↗
  14. Hinsvark, A. J., Delworth, N., Del Río, M., McNamara, Q., Dong, J., Westerman, R., Huang, M., Palakapilly, J., Drexler, J., Pirkin, I., Bhandari, N. & Jetté, M. (2021). Accented Speech Recognition: A Survey. ArXiv. ↗
  15. Holmes, J. (2013). An introduction to sociolinguistics. Routledge. https://doi.org/10.4324/9781315833057 ↗
  16. Incera, S., Shah, A. P., McLennan, C. T. & Wetzel, M. T. (2017). Sentence context influences the subjective perception of foreign accents. Acta Psychologica 172, pp. 71-76. ↗
  17. Jones, T. (2015). Toward a description of African American Vernacular English dialect regions using “Black Twitter”. American Speech, Vol. 90, No. 4. https://doi.org/10.1215/00031283-3442117 ↗
  18. Kachru, B. (1992). The Other Tongue: English across cultures. University of Illinois Press. find
  19. Kang, Y. M. & Zhou, Y. (2020). Fast and robust unsupervised contextual biasing for speech recognition. ArXiv. ↗
  20. Koenecke, A., Nam, A., Lake, E., Nudell, J., Quartey, M., Mengesha, Z , Toups, C., Rickford, J. R., Jurafsky, D. & Goel, S. (2020). Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, vol. 117, no. 14, pp. 7684–7689. https://doi.org/10.1073/pnas.1915768117 ↗
  21. Kosmala, L., and Crible, L. (2021). The dual status of filled pauses: Evidence from genre, proficiency and co-occurrence. ↗
  22. Language and Speech, May 2021. [Online]. Available: https://halshs.archives-ouvertes.fr/halshs-03225622 https://doi.org/10.1177/00238309211010862 ↗
  23. Levi, S. V., Winters, S. J. & Pisoni, D. B. (2007). Speaker-independent factors affecting the perception of foreign accent in a second language. Journal of the Acoustic Society of America, 121(4), pp. 2327-2338. https://doi.org/10.1121/1.2537345 ↗
  24. Lippi-Green, R. (2012). English with an Accent: Language, Ideology and Discrimination in the United States. Routledge. https://doi.org/10.4324/9780203348802 ↗
  25. Meyer, J., Rauchenstein, L., Eisenberg, J. D. & Howell, N. (2020). Artie bias corpus: An open dataset for detecting demographic bias in speech applications. Proceedings of the 12th Language Resources and Evaluation Conference, pp. 6462–6468.
  26. Miller, C., Tzoukermann E., Doyon J., and Mallard, E., (2021). Corpus creation and evaluation for speech-to-text and speech translation. Proceedings of Machine Translation Summit XVIII: Users and Providers Track, pp. 44–53. find
  27. O’Neill, P. K., Lavrukhin, V., Majumdar, S., Noroozi, V., Zhang, Y., Kuchaiev, O., Balam, J., Dovzhenko, Y., Freyberg, K., Shulman, M. D., Ginsburg, B., Watanabe, S., and Kucsko, G. (2021). “SPGISpeech: 5,000 Hours of Transcribed Financial Audio for Fully Formatted End-to-End Speech Recognition,” in Proc. Interspeech, pp. 1434–1438. ↗
  28. Palanica, A., Thommandram, A., Lee, A., Li, M. & Fossat, Y. (2019). Do you understand the words that are comin outta my mouth? Voice assistant comprehension of medication names. NPJ Digital Medicine, vol. 55, pp. 1-6. https://doi.org/10.1038/s41746-019-0133-x ↗
  29. Pharies, D. A. (2007). A Brief History of the Spanish Language. University Of Chicago Press. https://doi.org/10.1038/s41746-019-0133-x ↗
  30. Porgali, B., Albiero, V., Ryda, J., Ferrer, C. C. & Hazirbas, C. (2023). The Casual Conversations v2 Dataset. ArXiv. ↗
  31. Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2022). Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.04356. ↗
  32. Ralli, A. (2020). Greek in Contact with Romance. In M. Loporcaro & F. Gardani (eds.) The Oxford Encyclopedia of Romance Linguistics. Oxford. https://doi.org/10.1093/acrefore/9780199384655.013.422 ↗
  33. Reid, K. & Williams, E. T. (2023). Common Voice and accent choice: Data contributors self-describe their spoken accents in diverse ways. EasyChair. https://doi.org/10.1145/3617694.3623258 ↗
  34. Trinh, V. A., Gharemani, P., King, B., Droppo, J., Stolcke, A. & Maas, R. (2022). Reducing geographic disparities in automatic speech recognition via elastic weight consolidation. Proceedings of Interspeech. https://doi.org/10.21437/Interspeech.2022-11063 van Rooy, B. (2020). English in Africa. In D. Schreier, M. Hundt & E. W. Schneider (eds.), The Cambridge Handbook of World Englishes, pp. 210-235. Cambridge University Press. ↗
  35. Wagner, E., Liao, Y.-F. & Wagner, S. (2021). Authenticated Spoken Texts for L2 Listening Tests. Language Assessment Quarterly 18:3, pp. 205-227. https://doi.org/10.1080/15434303.2020.1860057 ↗
  36. Wells, J. C. (1982). Accents of English: Volume 3: Beyond the British Isles. Cambridge University Press. https://doi.org/10.1017/CBO9780511611766 Wrembel, M., Gut, U., Kopečková, R. & Balas, A. Cross-linguistic interactions in third language acquisition: Evidence from multi-feature analysis of speech perception. (2020). Languages 5:52, pp. 1-21. https://doi.org/10.3390/languages5040052 ↗
  37. Yang, X., Audhkhasi, K., Rosenberg, A., Thomas, S., Ramabhadran, B., and Hasegawa-Johnson, M. (2018). “Joint modeling of accents and acoustics for multi-accent speech recognition,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
  38. IEEE, 2018, pp. 1–5. https://doi.org/10.1109/ICASSP.2018.8462557 ↗
  39. Xiong, W., Droppo, J., Huang, X., Seide, F., Seltzer, M., Stolcke, A., Yu, D. & Zweig, G. (2016). Achieving human parity in conversational speech recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing. https://doi.org/10.1109/TASLP.2017.2756440 ↗
  40. Zhou, L., Li, J., Sun, E. & Liu, S. (2022). A Configurable Multilingual Model is all you need to recognize all languages. Proceedings of ICASSP. https://doi.org/10.1109/ICASSP43922.2022.9747905 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rio-2023-accents-speech-recognition-through-lens/