Elamur Research⌕
☰
home / papers / 2024 / Modeling Gender and Dialect Bias in Automatic Speech Recognition

Modeling Gender and Dialect Bias in Automatic Speech Recognition

Paper · 2024 · Camille Harris, Chijioke Mgbahurike, Neha Kumar et al.

Summary

The paper asks whether automatic speech recognition (ASR) errors fall unevenly across speakers when gender and English dialect are considered together, rather than one at a time. The authors drew audio from the Spotify Podcast Dataset, using seed keyword lists to locate episodes likely to feature women, men, Black, or Latino speakers. They then built a corpus of about 13 hours covering four US varieties: Standard American English (SAE), AAVE, Chicano English, and Spanglish. Transcription followed a "dialect-centered" process: annotators who speak each variety wrote the reference text, so that dialect spellings, grammar, and vocabulary are kept rather than standardized. Several state-of-the-art ASR models, including Whisper, wav2vec 2.0, and HuBERT, were scored by word error rate in a zero-shot setting and again after fine-tuning. Every model did better on SAE than on the minority dialects. The gender pattern reversed by dialect: women were recognized better within the minority dialects, but men were recognized better within SAE. This points to men who speak minority dialects as the group most exposed to transcription errors. Fine-tuning raised overall accuracy.

Key points

  • The new resource is about 13 hours of podcast audio, sampled from the Spotify Podcast Dataset released by Clifton et al. in 2020.
    • It is labeled for dialect (SAE, AAVE, Chicano English, Spanglish), binary gender, and other metadata.
    • Metadata was assigned per episode, not per audio snippet.
  • Across all evaluated models, SAE had lower WER than the three minority dialects.
    • The worst minority-dialect results appeared when several minority-dialect speakers talked in the same recording.
  • Whisper tended to swap dialect-specific words for their SAE equivalents, while wav2vec and HuBERT were more likely to keep them.
    • The authors attribute part of this to dialect terms missing from Whisper's vocabulary.
  • Code-switching is a possible confound.
    • Code-switching was flagged in 87% of AAVE samples with women and in none with men.
    • It was flagged in all Chicano English/Spanglish samples with women and in none with men.
    • Two checks suggest the gender gap is not just a code-switching effect: AAVE women without code-switching still had much lower WER than AAVE men, and the same advantage for women held on CORAAL.
  • Access to the source dataset was restricted after Spotify stopped maintaining it in December 2023.

Interesting ideas

  • I1. Dialect-centered annotation means reference transcripts are written by people who speak the variety, so the gold standard does not quietly "correct" dialect forms into the standard. Otherwise, faithful output gets scored as errors.
  • I2. Breaking error rates down by crossed groups (dialect × gender) exposed a reversal that the single-axis averages concealed.
  • I3. Whisper's habit of replacing dialect words with standard ones is a quiet normalization error: the output reads fluently but changes what the speaker actually said.
  • I4. The authors tested a suspected confound twice: they re-checked the effect on the subset without the confound (no code-switching) and on an outside corpus (CORAAL).
  • I5. Seed keyword lists over episode descriptions offered a cheap way to sample under-represented speaker groups from a large unlabeled pool.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): When building or converting UD treebanks for non-standard English, have dialect-competent reviewers check the gold trees so that constructions like copula deletion or negative concord are annotated as they occur, not repaired.
  • I2 (learn): Evaluation reports could slice parser and QA accuracy by combinations of text properties (register × dialect features × genre), so that compounding failures become visible rather than averaged away.
  • I3 (tokens): The tagger and lemmatizer should keep dialect forms (e.g., "ain't", habitual "be") as distinct, explainable analyses and should not map them silently onto standard lemmas that change their meaning.
  • I4 (learn): This works as a negative-control template: whenever a metric gap appears, re-measure it on a confound-free subset and on an independent dataset before attributing it to the suspected cause.
  • I5 (learn): Similar seed-term filters could pull dialect-rich or code-switched English text out of large corpora to build targeted silver-data and stress-test sets for the parsers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States506 works · 330,052 citations🎓Georgia Institute of TechnologyAtlanta · United States152 works · 27,763 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSafety Research · 626 works

🧭 Topics 1Research subjects and languages the work is about.Speech & spoken language · 519 works

Cite

Cite as Camille Harris, Chijioke Mgbahurike, Neha Kumar and Diyi Yang. 2024. Modeling Gender and Dialect Bias in Automatic Speech Recognition. Findings of the Association for Computational Linguistics: EMNLP 2024.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.findings-emnlp.890

@inproceedings{harris2024modeling,
  title = {{Modeling Gender and Dialect Bias in Automatic Speech Recognition}},
  author = {Camille Harris, Chijioke Mgbahurike, Neha Kumar and Diyi Yang},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2024},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.890},
  url = {https://doi.org/10.18653/v1/2024.findings-emnlp.890}
}

References

44 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alfredo Ardila. 2005. Spanglish: an anglicized spanish dialect. Hispanic Journal of Behavioral Sciences, 27(1):60–81. ↗
  2. Giuseppe Attanasio, Beatrice Savoldi, Dennis Fucci, and Dirk Hovy. 2024. Multilingual speech models for automatic speech recognition exhibit gender performance gaps. arXiv preprint arXiv:2402.17954. ↗
  3. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33:12449–12460. of fairness. In 2020 IEEE 36th International Conference on Data Engineering (ICDE), pages 1918–1921. IEEE.
  4. Marcely Zanon Boito, Laurent Besacier, Natalia Tomashenko, and Yannick Estève. 2022. A study of gender impact in self-supervised models for speech-to-text systems. arXiv preprint arXiv:2204.01397. ↗
  5. Marcio Fuckner, Sophie Horsman, Pascal Wiggers, and Iskaj Janssen. 2023. Uncovering bias in asr systems: Evaluating wav2vec2 and whisper for dutch speakers. In 2023 International Conference on Speech Technology and Human-Computer Dialogue (SpeD), pages 146–151. find
  6. Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency, pages 77–91. PMLR. find
  7. Reem Gody and David Harwath. 2023. Unsupervised fine-tuning data selection for asr using self-supervised speech models. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. find
  8. John Butzberger, Hy Murveit, Elizabeth Shriberg, and Patti Price. 1992. Spontaneous speech effects in large vocabulary speech recognition applications. In Speech and Natural Language: Proceedings of a Workshop Held at Harriman, New York, February 23-26, 1992. find
  9. Eugenia Casielles-Suárez. 2017. Spanglish: The hybrid voice of latinos in the united states. Atlantis, pages 147–168. ↗
  10. Xuankai Chang, Wangyou Zhang, Yanmin Qian, Jonathan Le Roux, and Shinji Watanabe. 2020. End-to-end multi-speaker speech recognition with transformer. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6134–6138. IEEE. ↗
  11. Ann Clifton, Sravana Reddy, Yongze Yu, Aasish Pappu, Rezvaneh Rezapour, Hamed Bonab, Maria Eskevich, Gareth Jones, Jussi Karlgren, Ben Carterette, and Rosie Jones. 2020. 100,000 podcasts: A spoken English document corpus. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5903–5917, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  12. Jay L. Cunningham. 2023. Collaboratively mitigating racial disparities in automated speech recognition and language technologies with african american english speakers: Community-collaborative and equity-centered approaches toward designing inclusive natural language systems. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, CHI EA ’23, New York, NY, USA. Association for Computing Machinery. find
  13. Dorottya Demszky, Devyani Sharma, Jonathan H Clark, Vinodkumar Prabhakaran, and Jacob Eisenstein. 2020. Learning to recognize dialect features. arXiv preprint arXiv:2010.12707. ↗
  14. Charlie Farringon and Tyler Kendall. 2024. Home. https://oraal.github.io. Accessed: 2024-10-4. ↗
  15. James R Foulds, Rashidul Islam, Kamrun Naher Keya, and Shimei Pan. 2020. An intersectional definition Christina N Harrington, Radhika Garg, Amanda Woodward, and Dimitri Williams. 2022. “it’s kind of like code-switching”: Black older adults’ experiences with a voice assistant for health information seeking. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pages 1–15. find
  16. Camille Harris, Matan Halevy, Ayanna Howard, Amy Bruckman, and Diyi Yang. 2022. Exploring the role of grammar and word choice in bias toward african american english (aae) in hate speech classification. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 789–798. find
  17. Camille Harris, Amber Gayle Johnson, Sadie Palmer, Diyi Yang, and Amy Bruckman. 2023. "honestly, i think tiktok has a vendetta against black creators": Understanding black content creator experiences on tiktok. Proc. ACM Hum.-Comput. Interact., 7(CSCW2). find
  18. Arthur Hinsvark, Natalie Delworth, Miguel Del Rio, Quinten McNamara, Joshua Dong, Ryan Westerman, Michelle Huang, Joseph Palakapilly, Jennifer Drexler, Ilya Pirkin, et al. 2021. Accented speech recognition: A survey. arXiv preprint arXiv:2104.10747. ↗
  19. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460. ↗
  20. Allison Koenecke, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups, John R Rickford, Dan Jurafsky, and Sharad Goel. 2020. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, 117(14):7684–7689. ↗
  21. Youjin Kong. 2022. Are “intersectionally fair” ai algorithms really fair to women of color? a philosophical analysis. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’22, page 485–494, New York, NY, USA. Association for Computing Machinery. find
  22. Bernd Kortmann and Kerstin Lunkenheimer. 2012. The Mouton world atlas of variation in English. de Gruyter. find
  23. Chenda Li, Yao Qian, Zhuo Chen, Naoyuki Kanda, Dongmei Wang, Takuya Yoshioka, Yanmin Qian, and Michael Zeng. 2023. Adapting multi-lingual asr models for handling multiple talkers. arXiv preprint arXiv:2305.18747. ↗
  24. Chunxi Liu, Michael Picheny, Leda Sarı, Pooja Chitkara, Alex Xiao, Xiaohui Zhang, Mark Chou, Andres Alvarado, Caner Hazirbas, and Yatharth Saraf. 2022. Towards measuring fairness in speech recognition: Casual conversations dataset transcriptions. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6162–6166. IEEE. ↗
  25. John Myhill. 1995. The use of features of present-day aave in the ex-slave recordings. American Speech, 70(2):115–147. ↗
  26. Mikel K Ngueajio and Gloria Washington. 2022. Hey asr system! why aren’t you more inclusive? automatic speech recognition systems’ bias and proposed bias mitigation techniques. a literature review. In International Conference on Human-Computer Interaction, pages 421–440. Springer. find
  27. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pages 28492–28518. PMLR. ↗
  28. Gabriela T Richard and Yasmin B Kafai. 2016. Blind spots in youth diy programming: Examining diversity in creators, content, and comments within the scratch online community. In Proceedings of the 2016 CHI conference on Human Factors in Computing Systems, pages 1473–1485. find
  29. Morgane Riviere, Jade Copet, and Gabriel Synnaeve. 2021. Asr4real: An extended benchmark for speech models. arXiv preprint arXiv:2110.08583. ↗
  30. Jennifer Roth-Gordon, Jessica Harris, and Stephanie Zamora. 2020. Producing white comfort through “corporate cool”: Linguistic appropriation, social media, and@ brandssayingbae. International Journal of the Sociology of Language, 2020(265):107–128. find
  31. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 1668–1678.
  32. Morgan Klaus Scheuerman, Kandrea Wade, Caitlin Lustig, and Jed R Brubaker. 2020. How we’ve taught algorithms to see identity: Constructing race and gender in image databases for facial analysis. Proceedings of the ACM on Human-computer Interaction, 4(CSCW1):1–35. ↗
  33. William Seymour, Xiao Zhan, Mark Coté, and Jose Such. 2023. A systematic review of ethical concerns with voice assistants. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’23, page 131–145, New York, NY, USA. Association for Computing Machinery. ↗
  34. Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating gender bias in natural language processing: Literature review. arXiv preprint arXiv:1906.08976. ↗
  35. Rachael Tatman. 2017. Gender and dialect bias in youtube’s automatic captions. In Proceedings of the first ACL workshop on ethics in natural language processing, pages 53–59.
  36. Rachael Tatman and Conner Kasten. 2017. Effects of talker dialect, gender & race on accuracy of bing speech and youtube automatic captions. In Interspeech, pages 934–938. find
  37. Mike Thelwall. 2018. Gender bias in sentiment analysis. Online Information Review. ↗
  38. Kassym-Jomart Tokayev. 2023. Ethical implications of large language models a multidimensional exploration of societal, economic, and technical concerns. International Journal of Social Analytics, 8(9):17– 33. find
  39. Eva Vanmassenhove, Christian Hardmeier, and Andy Way. 2019. Getting gender right in neural machine translation. arXiv preprint arXiv:1909.05088. ↗
  40. Alexandra Voit, Jasmin Niess, Caroline Eckerth, Maike Ernst, Henrike Weingärtner, and Paweł W. Woźniak. 2020. ‘it’s not a romantic relationship’: Stories of adoption and abandonment of smart speakers at home. In Proceedings of the 19th International Conference on Mobile and Ubiquitous Multimedia, MUM ’20, page 71–82, New York, NY, USA. Association for Computing Machinery. find
  41. Angelina Wang, Vikram V Ramaswamy, and Olga Russakovsky. 2022. Towards intersectionality in machine learning: Including more identities, handling underrepresentation, and performing evaluation. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’22, page 336–349, New York, NY, USA. Association for Computing Machinery. find
  42. Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. 2021. Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. arXiv preprint arXiv:2101.00390. ↗
  43. Alicia Beckford Wassink, Cady Gansen, and Isabel Bartholomew. 2022. Uneven success: automatic speech recognition and ethnicity-related dialects. Speech Communication, 140:50–70. ↗
  44. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. 2019. Gender bias in contextualized word embeddings. arXiv preprint arXiv:1904.03310. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/harris-2024-modeling-gender-dialect-bias-automatic/