Elamur Research⌕
☰
home / papers / 2023 / The Edinburgh International Accents of English Corpus: Towards the Democratization of English ASR

The Edinburgh International Accents of English Corpus: Towards the Democratization of English ASR

Paper · 2023 · Ramon Sanabria, Nikolay Bogoychev, Nina Markl et al.

Summary

English ASR systems post very low error rates on standard benchmarks, but those test sets mostly contain US or UK speakers, often reading text aloud or talking over narrowband phones. They say little about how well the systems serve the global English-speaking population. The authors address this gap with EdAcc, a freely licensed (CC-BY-SA) evaluation corpus of about 40 hours of relaxed Zoom conversations between pairs of people who know each other. The participants include both native and non-native speakers drawn from many different varieties of English. Each participant filled in a detailed questionnaire on their languages, English-learning history, places of residence and self-described accent. Each also read a shared control passage. Professional transcribers produced the transcripts, which were then normalized for scoring. The authors tested wav2vec 2.0 variants, Whisper and an anonymized commercial engine. Even the strongest model, Whisper trained on 680k hours, averages 19.7% WER, against 2.7% on clean US read speech. Errors rise noticeably for Indian, Jamaican and Nigerian English. EdAcc therefore works as a diagnostic tool for exposing accent bias in speech recognition.

Key points

  • Size and format: EdAcc has almost 40 hours of dyadic video-call conversations, each lasting 20–60 minutes. The speakers report more than 40 accents and 51 distinct first languages.
  • Splits: There are only dev and test splits, with 31 and 30 conversations respectively. No speaker or conversation appears in both. The splits were made with Kaldi's random subset script.
  • Speaker metadata: The questionnaire asked about L1s acquired before age 5, L2s, and when the speaker started learning English. It also covered:
    • which language they use for work, with friends and with family;
    • places lived for more than three years;
    • their relationship with the conversation partner;
    • their self-described accent;
    • age, gender, ethnicity and education.
  • Recording and transcription:
    • Both speakers share one audio channel, as in real deployments.
    • Every speaker also read the "Stella" passage from the Speech Accent Archive.
    • Transcripts mark overlaps, noise, laughter and hesitations.
    • Sounds and tokens with several valid spellings (e.g. numbers) were located by Kaldi forced alignment and excluded from scoring.
  • Results: Whisper performs best at 19.7% average WER, far from its benchmark numbers. All systems degrade on Indian, Jamaican and Nigerian English speakers.

Interesting ideas

  • I1. Collecting a rich linguistic profile per speaker lets errors be broken down by L1, accent and language history rather than by a vague "accented" label. This turns one global score into a map of who the system fails.
  • I2. The authors locate spans that a system could legitimately render in several forms, such as numbers or laughter, and exclude them from scoring. This keeps the metric from punishing harmless notational choices.
  • I3. Every speaker reads the same short passage next to their free conversation. This gives a controlled slice where only the speaker varies, alongside a naturalistic one.
  • I4. Sociolinguistic elicitation tricks reduce self-monitoring and produce speech closer to real use: talking with friends, engaging prompts, and self-recording. Read speech, by contrast, can hide the very features being tested.
  • I5. Reporting a strong model's error on the clean benchmark next to its error on a diverse set exposes how optimistic standard numbers are.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Tag every evaluation text with its source variety, genre and author background, so parser and reader failures can be grouped by origin rather than averaged away.
  • I2 (tokens): When scoring tokenizer and lemma output against gold, mark spans with several valid analyses (numerals, contractions, hyphenated forms) and skip or soft-score them, so errors reflect real mistakes.
  • I3 (tree): Keep a fixed control text rendered in several varieties or paraphrase styles and parse each version, so dialect-driven differences in the UD trees can be isolated from content differences.
  • I4 (answer): Build test sets for fable and story QA from naturally written texts, not only edited textbook prose, so the readers are checked on language as people actually write it.
  • I5 (learn): Always publish Mova's scores on its home treebank together with an out-of-domain or other-variety set, as a built-in check on overfitting to one kind of English.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom359 works · 77,352 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksSpeech & spoken language · 480 works

Cite

Cite as Ramon Sanabria, Nikolay Bogoychev, Nina Markl, Andrea Carmantini, Ondrej Klejch and Peter Bell. 2023. The Edinburgh International Accents of English Corpus: Towards the Democratization of English ASR. arXiv:2303.18110.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2303.18110 · arXiv 2303.18110

@misc{sanabria2023edinburgh,
  title = {{The Edinburgh International Accents of English Corpus: Towards the Democratization of English ASR}},
  author = {Ramon Sanabria, Nikolay Bogoychev, Nina Markl, Andrea Carmantini, Ondrej Klejch and Peter Bell},
  year = {2023},
  doi = {10.48550/arxiv.2303.18110},
  eprint = {2303.18110},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.18110}
}

References

34 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. L. Campbell, “Ethnologue: Languages of the world,” 2008. find
  2. L. Bauer, An Introduction to International Varieties of English. Edinburgh University Press, 2003. find
  3. A. Koenecke, A. Nam, E. Lake, J. Nudell, M. Quartey, Z. Mengesha, C. Toups, J. R. Rickford, D. Jurafsky, and S. Goel, “Racial disparities in automated speech recognition,” Proceedings of the National Academy of Sciences, vol. 117, 2020. ↗
  4. S. L. Blodgett, S. Barocas, H. Daumé III, and H. Wallach, “Language (technology) is power: A critical survey of “bias” in NLP,” in ACL, 2020. — Language (Technology) is Power: A Critical Survey of “Bias” in NLP
  5. N. Markl, “Language variation and algorithmic bias: Understanding algorithmic bias in british english automatic speech recognition,” in Conference on Fairness, Accountability, and Transparency, 2022. John Wiley & Sons, find
  6. G. Van Herk, What is sociolinguistics. Inc, 2018. find
  7. J. J. Godfrey, E. C. Holliman, and J. McDaniel, “Switchboard: Telephone speech corpus for research and development,” in ICASSP, 1992. find
  8. Z. Tüske, G. Saon, and B. Kingsbury, “On the limit of english conversational speech recognition,” in Interspeech, 2021.
  9. A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” NeurIPS, 2020.
  10. V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in ICASSP, 2015. find
  11. W.-N. Hsu, A. Sriram, A. Baevski, T. Likhomanenko, Q. Xu, V. Pratap, J. Kahn, A. Lee, R. Collobert, G. Synnaeve, and M. Auli, “Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training,” in Interspeech, 2021. find
  12. G.-T. Lin, C.-J. Hsu, D.-R. Liu, H.-Y. Lee, and Y. Tsao, “Analyzing the robustness of unsupervised speech recognition,” in ICASSP, 2022. ↗
  13. E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell, “On the dangers of stochastic parrots: Can language models be too big,” in Conference on Fairness, Accountability, and Transparency (ACM), 2021. find
  14. N. Markl, “Mind the data gap(s): Investigating power in speech and language datasets,” Second Workshop on Language Technology for Equality, Diversity and Inclusion (ACL), 2022. find
  15. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” OpenAI, Tech. Rep., 2022. ↗
  16. J. S. Garofolo, L. F. Lamel, W. M. Fisher, J. G. Fiscus, and D. S. Pallett, “Darpa timit acoustic-phonetic continous speech corpus cd-rom,” NASA STI, 1993. find
  17. D. B. Paul and J. Baker, “The design for the wall street journal-based csr corpus,” in Speech and Natural Language: Proceedings of a Workshop Held at Harriman, 1992. find
  18. J. Kahn, M. Rivière, W. Zheng, E. Kharitonov, Q. Xu, P.-E. Mazaré, J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen et al., “Libri-light: A benchmark for asr with limited or no supervision,” in ICASSP, 2020. find
  19. I. Demirsahin, O. Kjartansson, A. Gutkin, and C. Rivera, “Open-source multi-speaker corpora of the english accents in the british isles,” in LREC, 2020.
  20. G. Zhao, S. Sonsaat, A. Silpachai, I. Lucic, E. Chukharev-Hudilainen, J. Levis, and R. Gutierrez-Osuna, “L2-arctic: A non-native english speech corpus.” in Interspeech, 2018. find
  21. R. Ardila, M. Branson, K. Davis, M. Kohler, J. Meyer, M. Henretty, R. Morais, L. Saunders, F. Tyers, and G. Weber, “Common voice: A massively-multilingual speech corpus,” in EACL, 2019. ↗
  22. X. Shi, F. Yu, Y. Lu, Y. Liang, Q. Feng, D. Wang, Y. Qian, and L. Xie, “The accented english speech recognition challenge 2020: open datasets, tracks, baselines, results and methods,” in ICASSP, 2021. ↗
  23. J. L. Martin and K. Tang, “Understanding racial disparities in automatic speech recognition: The case of habitual “be”,” in Proc. Interspeech 2020, 2020, pp. 626–630. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2020-2893 ↗
  24. J. Meyer, L. Rauchenstein, J. D. Eisenberg, and N. Howell, “Artie bias corpus: An open dataset for detecting demographic bias in speech applications,” in Proceedings of the 12th language resources and evaluation conference, 2020.
  25. N. Schilling, Sociolinguistic Fieldwork, ser. Key Topics in Sociolinguistics. Cambridge University Press, 2013. find
  26. H. Giles, N. Coupland, and J. Coupland, Accommodation theory: Communication, context, and consequence. Cambridge University Press, 1991. find
  27. S. Weinberger, “The speech accent archive,” online, 2015. [Online]. Available: http://accent.gmu.edu ↗
  28. D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, J. Silovsky, G. Stemmer, and K. Vesely, “The kaldi speech recognition toolkit,” in ASRU, Dec. 2011. find
  29. K. Heafield, “KenLM: Faster and smaller language model queries,” in SMT, 2011.
  30. Z.-H. Tan, N. Dehak et al., “rvad: An unsupervised segment-based robust voice activity detection method,” Computer speech & language, 2020. ↗
  31. I. McCowan, J. Carletta, W. Kraaij, S. Ashby, S. Bourban, M. Flynn, M. Guillemot, T. Hain, J. Kadlec, V. Karaiskos et al., “The ami meeting corpus,” in International Conference on Methods and Techniques in Behavioral Research, 2005. find
  32. P. Bell, M. J. Gales, T. Hain, J. Kilgour, P. Lanchantin, X. Liu, A. McParland, S. Renals, O. Saz, M. Wester et al., “The mgb challenge: Evaluating multi-genre broadcast media recognition,” in ASRU, 2015. find
  33. M. Ravanelli, T. Parcollet, P. Plantinga, A. Rouhe, S. Cornell, L. Lugosch, C. Subakan, N. Dawalatabad, A. Heba, J. Zhong, J.-C. Chou, S.-L. Yeh, S.-W. Fu, C.-F. Liao, E. Rastorgueva, F. Grondin, W. Aris, H. Na, Y. Gao, R. D. Mori, and Y. Bengio, “Speechbrain: A general-purpose speech toolkit,” 2021. ↗
  34. N. Dawalatabad, M. Ravanelli, F. Grondin, J. Thienpondt, B. Desplanques, and H. Na, “ECAPA-TDNN Embeddings for Speaker Diarization,” in Interspeech, 2021. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sanabria-2023-edinburgh-international-accents-english-corpus/