Elamur Research⌕
☰
home / papers / 2025 / Automatic Speech Recognition Biases in Newcastle English: an Error Analysis

Automatic Speech Recognition Biases in Newcastle English: an Error Analysis

Paper · 2025 · Dana Serditova, Kevin Tang and Jochen Steffens

Summary

Commercial speech recognisers are known to be less accurate for some racial, gender and age groups, but regional dialect bias has received less attention. This study examines how ASR handles Newcastle (Tyneside) English, an accent that earlier work identified as one of the hardest in the UK for these systems. The authors pre-tested four recognisers on DECTE and chose Rev AI as the strongest. They then ran a two-step analysis. First, two sociophoneticians listened to a subsample and hand-coded every error that dialect could explain, using a two-level taxonomy of error types. Second, scripts aligned ASR output with reference transcripts across the whole corpus to find misrecognitions of the local pronouns "yous" and "wor". Mixed-effects regression tested age and gender while controlling for recording noise. Recognition errors tracked specific dialect features closely, while speaker age and gender mattered less. The paper argues that training data needs more dialect variety and that linguistically informed error analysis is needed to diagnose bias.

Key points

  • Data: DECTE, a diachronic collection of electronically stored Tyneside English recordings, containing 72 hours of spontaneous speech from 160 speakers in 99 files, all transcribed in full by humans.
  • System selection used 10% of the data. Google Cloud Speech-to-Text and Deepgram scored roughly 40–60% WER and were dropped. Rev AI then beat CrisperWhisper by about 10 WER points on most recordings, and Rev AI (English UK) averaged 31.95% WER over the full corpus.
  • The manual stage covered 16 files and 32 speakers and yielded 1,076 dialect-attributable errors. These fell into five types: phonological, lexical, standardisation, morphosyntactic and spelling.
  • Pronoun errors made up about one third of morphosyntactic errors, which motivated the corpus-wide case study of "yous" and "wor".
  • Male speakers accounted for more errors than female speakers (53.7% vs 46.3%, χ² = 11.6, p < 0.001). Noise was estimated with WADA-SNR and entered as a covariate in glmer models that included a random effect for speaker.

Interesting ideas

  • I1. The authors define a separate "standardisation" error class for cases where the system silently swaps a dialect form for a standard one (e.g., "me life" → "my life"), keeping these apart from plain misrecognitions; this separates normalisation bias from noise.
  • I2. They pair a small, finely coded manual pass with a scripted, corpus-wide extraction aimed only at the most frequent problem category, which keeps expensive annotation small while still producing statistics on the full data.
  • I3. They put recording noise (SNR) into the model as a covariate and treat speaker as a random effect, so demographic effects are not confused with recording quality or individual habits.
  • I4. The paper notes that without tagging dialect features, legitimate regional forms get counted as errors, which distorts evaluation.
  • I5. Regional pronouns carry different person/number meanings: "yous" is 2nd plural, "wor" means "our", and object "us" can mean "me". Mistranscribing them changes who is being referred to.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): The en crate could map forms like "yous", "wor", "div", "gan" and possessive "me" to standard lemmas while keeping a dialect-variant feature, instead of treating them as typos or overwriting them.
  • I2 (learn): Hand-audit a sample of parser or frame failures, group them into a taxonomy, then write targeted extractors for the biggest bucket across all silver data.
  • I3 (learn): When comparing accuracy across text genres or sources, control for nuisance factors such as OCR noise or sentence length, and group by document, so that apparent bias is not an artefact.
  • I4 (tree): During UD dialect conversion, mark non-standard constructions such as unmarked plurals ("six month") and "they've broke it" so the parser is not penalised against a treebank that assumes standard morphology.
  • I5 (frames): Coreference and role assignment could use a small dialect pronoun lexicon so that object "us" can resolve to the singular speaker and "yous" to a group of addressees.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksPsychology · 1,649 worksExperimental and Cognitive Psychology · 930 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,379 worksSpeech & spoken language · 519 works

Cite

Cite as Dana Serditova, Kevin Tang and Jochen Steffens. 2025. Automatic Speech Recognition Biases in Newcastle English: an Error Analysis. arXiv:2506.16558 · Proc. Interspeech 2025 (2025) 3204-3208.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.21437/interspeech.2025-1973 · arXiv 2506.16558

@misc{serditova2025automatic,
  title = {{Automatic Speech Recognition Biases in Newcastle English: an Error Analysis}},
  author = {Dana Serditova, Kevin Tang and Jochen Steffens},
  year = {2025},
  doi = {10.21437/interspeech.2025-1973},
  eprint = {2506.16558},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.21437/interspeech.2025-1973}
}

References

35 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Koenecke, A. Nam, E. Lake, J. Nudell, M. Quartey, Z. Mengesha, C. Toups, J. R. Rickford, D. Jurafsky, and S. Goel, “Racial disparities in Automated Speech Recognition,” Proceedings of the National Academy of Sciences, vol. 117, no. 14, pp. 7684–7689, 2020. ↗
  2. S. Feng, B. M. Halpern, O. Kudina, and O. Scharenborg, “Towards inclusive Automatic Speech Recognition,” Computer Speech & Language, vol. 84, p. 101567, 2024. ↗
  3. C. Liu, M. Picheny, L. Sarı, P. Chitkara, A. Xiao, X. Zhang, M. Chou, A. Alvarado, C. Hazirbas, and Y. Saraf, “Towards measuring fairness in speech recognition: Casual conversations dataset transcriptions,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 6162–6166. ↗
  4. G. Zellou, M. Cohn, and B. Ferenc Segedin, “Age-and gender-related differences in speech alignment toward humans and voice-AI,” Frontiers in Communication, vol. 5, p. 600361, 2021. ↗
  5. A. B. Wassink, C. Gansen, and I. Bartholomew, “Uneven success: Automatic Speech Recognition and ethnicity-related dialects,” Speech Communication, vol. 140, pp. 50–70, 2022. ↗
  6. N. Markl, “Language variation and algorithmic bias: understanding algorithmic bias in British English Automatic Speech Recognition,” in Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, 2022, pp. 521–534. find
  7. L.-F. Lai, J. G. van Hell, and J. Lipski, “Dialect bias in Automatic Speech Recognition: Analysis of Appalachian English,” American Speech: A Quarterly of Linguistic Usage, pp. 1–17, 2024. ↗
  8. A. Mearns, “Tyneside,” in Researching Northern English. Benjamins Publishing Company, 2015, pp. 161–182. find
  9. A. Hughes, P. Trudgill, and D. Watt, English Accents and Dialects: An introduction to social and regional varieties of English in the British Isles. Routledge, 2013. find
  10. E. W. Schneider, K. Burridge, B. Kortmann, R. Mesthrie, and C. Upton, A Handbook of Varieties of English: A multimedia reference tool two volumes plus CD-ROM. De Gruyter Mouton, 2004. find
  11. M. K. Ngueajio and G. Washington, “Hey ASR system! Why aren’t you more inclusive? Automatic Speech Recognition systems’ bias and proposed bias mitigation techniques. A literature review,” in International Conference on Human-Computer Interaction. Springer, 2022, pp. 421–440. find
  12. R. Tatman and C. Kasten, “Effects of talker dialect, gender & race on accuracy of Bing speech and YouTube automatic captions,” in Interspeech, 2017, pp. 934–938. find
  13. J. L. Martin and K. Tang, “Understanding racial disparities in Automatic Speech Recognition: The case of habitual ”be”.” in Interspeech, 2020, pp. 626–630. find
  14. R. Sanabria, N. Bogoychev, N. Markl, A. Carmantini, O. Klejch, and P. Bell, “The Edinburgh international accents of English corpus: Towards the democratization of English ASR,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5. 5 https://credit.niso.org/ ↗
  15. C. Montgomery, “The effect of proximity in perceptual dialectology,” Journal of Sociolinguistics, vol. 16, no. 5, pp. 638–668, 2012. ↗
  16. J. Beal, “English dialects in the North of England: Phonology,” A Handbook of Varieties of English, vol. 1, pp. 113–133, 2004. find
  17. G. Docherty and P. Foulkes, “Sociophonetic variation in ‘glottals’ in Newcastle English,” in Proceedings of the 14th International Congress of Phonetic Sciences. University of California, Berkeley, 1999, pp. 1037–1040. find
  18. J. Grama, J. Mechler, L. Bauernfeind, M. E. Eiswirth, and I. Buchstaller, “Post-educator relaxation in the U-shaped curve: Evidence from a panel study of Tyneside (ing),” Language Variation and Change, vol. 35, no. 3, pp. 325–350, 2023. ↗
  19. D. Watt and L. Milroy, “Patterns of variation and change in three Newcastle vowels: Is this dialect levelling?” in Urban Voices. Routledge, 2014, pp. 25–46. find
  20. J. Beal, “The grammar of Tyneside and Northumbrian English,” in Real English. Routledge, 1993, pp. 187–213. find
  21. M. Pearce, “Folk accounts of dialect differences in Tyne and Wear,” Dialectologia et Geolinguistica, vol. 20, no. 1, pp. 5–25, 2012. ↗
  22. A.-M. Moelders, “Navigating the vernacular across the lifespan: A panel study of the phonetic realisation of the first-person singular possessive,” English Language & Linguistics, pp. 1–27, 2025. ↗
  23. K. P. Corrigan, I. Buchstaller, A. Mearns, and H. Moisl, “The Diachronic Electronic Corpus of Tyneside English,” Online at https://research.ncl.ac.uk/decte/index.htm (Accessed April 2, 2024), 2012. ↗
  24. M. Zusag, L. Wagner, and B. Thallinger, “Crisperwhisper: Accurate timestamps on verbatim speech transcriptions,” in Interspeech 2024, 2024, pp. 1265–1269. ↗
  25. C. Kim and R. M. Stern, “Robust signal-to-noise ratio estimation based on waveform amplitude distribution analysis,” in Ninth Annual Conference of the International Speech Communication Association, 2008. find
  26. A. Kuznetsova, P. B. Brockhoff, and R. H. B. Christensen, “lmerTest package: Tests in linear mixed effects models,” Journal of Statistical Software, vol. 82, no. 13, pp. 1–26, 2017. ↗
  27. D. Villarreal, L. Clark, J. Hay, and K. Watson, “From categories to gradience: Auto-coding sociophonetic variation with random forests,” Laboratory Phonology: Journal of the Association for Laboratory Phonology, vol. 11, no. 1, p. 6, 2020. ↗
  28. T. Kendall, C. Vaughn, C. Farrington, K. Gunter, J. McLean, C. Tacata, and S. Arnson, “Considering performance in the automated and manual coding of sociolinguistic variables: Lessons from variable (ing),” Frontiers in Artificial Intelligence, vol. 4, 2021. ↗
  29. P. Trudgill, “Sex, covert prestige and linguistic change in the urban British English of Norwich,” Language in Society, vol. 1, no. 2, pp. 179–195, 1972. ↗
  30. W. Labov, “The social motivation of a sound change,” Word, vol. 19, no. 3, pp. 273–309, 1963. ↗
  31. L. Milroy, Language and Social Networks. Blackwell, 1980. find
  32. P. Nichols, “Linguistic options and choices for Black women in the rural South,” Language, Gender and Society, pp. 54–68, 1983. find
  33. L. Bauernfeind, C. Ahrens, J. Grama, R. Skarnitzl, and J. Volı́n, “Change across the lifespan in goat: Evidence from a panel study of Tyneside English,” in Proceedings of the 20th International Congress of Phonetic Science. Guarant International, 2023, pp. 2064–2068. find
  34. J. Grama, M. E. Eiswirth, I. Buchstaller, R. Skarnitzl, and J. Volı́n, “Tracking creak from early to late adulthood: A panel study from the North East of England,” in Proceedings of the 20th International Congress of Phonetic Science. Guarant International, 2023, pp. 2049–2053. find
  35. J. Grieve, S. Bartl, M. Fuoli, J. Grafmiller, W. Huang, A. Jawerbaum, A. Murakami, M. Perlman, D. Roemling, and B. Winter, “The sociolinguistic foundations of language modeling,” Frontiers in Artificial Intelligence, vol. 7, 2025. John ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/serditova-2025-automatic-speech-recognition-biases-newcastle/