Elamur Research⌕
☰
home / papers / 2017 / Gender and Dialect Bias in YouTube's Automatic Captions

Gender and Dialect Bias in YouTube's Automatic Captions

Summary

This paper asks whether recent gains in automatic speech recognition have reached all speakers equally. It tests YouTube's automatic captions, which run on Google's ASR, on native English speakers grouped by gender and dialect. The data come from "accent tag" videos, a popular internet genre in which people state their regional background and then read a fixed list of dialect-sensitive words. The author hand-checked the captions for 80 speakers, computed a word error rate (WER) for each, and modelled the results with linear mixed-effects regression.

Both gender and dialect clearly affected accuracy:

  • Women's speech was transcribed worse than men's.
  • Scottish speakers fared worst of the five regions.
  • The size of the gender gap depended on the region.

A follow-up test shifted the pitch of a single female voice and found no change in accuracy. This suggests pitch alone may not explain the gender gap, though the test hit a ceiling.

The author links these gaps to poorly stratified training corpora. The paper argues that sociolinguistically stratified evaluation is an ethical necessity, especially as speech tools enter areas like hiring.

Key points

  • Sample design: 80 speakers in total, with 8 men and 8 women drawn from each of five regions: Scotland, New Zealand, California, Georgia, and New England (represented by speakers from Maine and New Hampshire). Only the word-list part of each accent tag video was used.
  • Statistical model: WER was modelled with linear mixed-effects regression. Speaker and upload year were random effects, covering recording and speaker quality as well as ASR improvements over five years.
    • Including both gender and dialect as fixed effects gave the best fit by AIC.
    • Adding a gender × dialect interaction improved the fit further.
    • Women had higher WER than men (t(78) = -3.5, p < 0.01).
  • Ruling out alternative explanations for Scotland: The poor Scottish result is not explained by population or wealth. New Zealand has about 80% of Scotland's population and nearly the same GDP per capita, yet it had the second-lowest WER.
  • Pitch experiment: One recording was shifted ±60 Hz in 20 Hz steps using Praat, giving seven versions. These produced only 9 errors across 434 tokens (WER .002) and no significant pitch effect.
  • Corpus imbalances cited:
    • About 69% of TIMIT speech comes from male speakers.
    • Switchboard undersamples women, Southerners and speakers without college education.
    • The Numbers corpus and the AMI corpus lack gender and dialect metadata.
    • ATAROS is cited as a corpus designed with better stratification.
    • Data and code are released on GitHub (rctatman/youtubeDialectAccuracy).

Interesting ideas

  • I1. Use self-labelled public content as a demographic test set. Accent tag videos come with speakers' own descriptions of their background and a shared reading list. This kind of content can serve as a free, controlled benchmark, which is useful wherever labelled speaker metadata is scarce, for example Ukrainian regional speech.
  • I2. Model per-speaker error with random effects. Treating speaker and collection time as random effects separates group-level bias from recording quality and system drift. It is a more honest approach than comparing raw averages between groups.
  • I3. Choose a deliberately hard, language-model-free task to expose acoustic bias. Isolated words that are known dialect markers remove help from context. This turns the evaluation into a stress test of pronunciation robustness.
  • I4. Manipulate one variable on a single voice to test a hypothesis. Shifting the pitch of one recording isolates a candidate cause of bias, here fundamental frequency, without needing a new corpus. The same trick could probe other acoustic factors, such as intensity.
  • I5. Audit and document corpus stratification. At minimum, record which groups are underrepresented when building speech or text corpora. This lets downstream users anticipate where a trained model is likely to fail.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

81 papers of our library cite it (of the 140884 whose bibliographies we read); 12 of them have a page here, 69 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w17-1606

@inproceedings{tatman2017gender,
  title = {{Gender and Dialect Bias in YouTube's Automatic Captions}},
  author = {Tatman, Rachael},
  booktitle = {Proceedings of the First ACL Workshop on Ethics in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/w17-1606},
  url = {https://doi.org/10.18653/v1/w17-1606}
}

References

41 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. This does not help to address existing imbalances in training data, however. One way of do-S. Ali, K. Siddiqui, N. Safdar, K. Juluru, W. Kim, and E. Siegel. 2007. Affect of gender on speech Waleed H. Abdulla and Nikola K. Kasabov. 2001. Improving speech recognition performance through gender separat… find
  2. Ifeoma Ajunwa, Sorelle Friedler, Carlos Scheidegger, and Suresh Venkatasubramanian. 2016. Hiring by algorithm: predicting and preventing disparate impact. Available at SSRN. find
  3. John J. Godfrey, Edward C. Holliman, and Jane Mc-Daniel. 1992. Switchboard: Telephone speech corpus for research and development. In Acoustics, Speech, and Signal Processing, 1992. ICASSP-92., 1992 IEEE International Conference on, volume 1, pages 517–520. IEEE. recognition accuracy. In American Jou… find
  4. Su Lin Blodgett, Lisa Green, and Brendan O’Connor. 2016. Demographic Dialectal Variation in Social Media: A Case Study of African-American English. arXiv preprint arXiv:1608.08868. find our card
  5. Sharon Goldwater, Dan Jurafsky, and Christopher D. Manning. 2010. Which words are hard to recognize? prosodic, lexical, and disfluency factors that increase speech recognition error rates. Speech Communication, 52(3):181–200. find
  6. Benjamin Bock and Lior Shamir. 2015. Assessing the efficacy of benchmarks for automatic speech accent recognition. In Proceedings of the 8th International Conference on Mobile Multimedia Communications, pages 133–136. ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications E… find
  7. Ken Harrenstien. 2009. Automatic captions in YouTube. The Official Google Blog, 11. find
  8. Jennifer Hay, Margaret Maclagan, and Elizabeth Gordon. 2008. New Zealand English. Edinburgh University Press. find
  9. Paul Boersma et al. 2002. Praat, a system for doing phonetics by computer. Glot international, 5(9/10):341–345. find
  10. Julia Hirschberg, Diane Litman, and Marc Swerts. 2004. Prosodic and other cues to speech recognition failures. Speech Communication, 43(1):155–175. find
  11. Donald J. Bogue, Douglas L. Anderton, and Richard E. Barrett. 2010. The population of the United States. Simon and Schuster. find
  12. Dirk Hovy and Shannon L. Spruit. 2016. The social impact of natural language processing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, volume 2, pages 591–598. find our card
  13. Frederic Gomes Cassidy et al. 1985. Dictionary of American Regional English. Belknap Press of Harvard University Press. find
  14. Constance M. Clarke and Merrill F. Garrett. 2004. Rapid adaptation to foreign-accented English. The Journal of the Acoustical Society of America, 116(6):3647–3658. find
  15. Yochai Konig and Nelson Morgan. 1992. Gdnn: A gender-dependent neural network for continuous speech recognition. In Neural Networks, 1992. IJCNN., International Joint Conference on, volume 2, pages 332–337. IEEE. find
  16. Ronald A. Cole, Mike Noel, Terri Lander, and Terry Durham. 1995. New telephone speech corpora at CSLU. In Eurospeech. Citeseer. find
  17. Hank Liao, Erik McDermott, and Andrew Senior. 2013. Large scale deep neural network acoustic modeling with semi-supervised training data for YouTube video transcription. In Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on, pages 368–373. IEEE. find
  18. Ghania Droua-Hamdani, Sid-Ahmed Selouani, and Malika Boudraa. 2012. Speaker-independent asr for modern standard arabic: effect of regional accents. International Journal of Speech Technology, 15(4):487–493. find
  19. Michael Lockrey. 2015. YouTube automatic craptions score an incredible 95% accuracy rate! medium.com, July. [Online; posted 25-July-2015]. find
  20. Penelope Eckert. 1989. The whole woman: Sex and gender differences in variation. Language variation and change, 1(03):245–267. find
  21. Iain McCowan, Jean Carletta, W. Kraaij, S. Ashby, S. Bourban, M. Flynn, M. Guillemot, T. Hain, J. Kadlec, V. Karaiskos, et al. 2005. The AMI meeting corpus. In Proceedings of the 5th International Conference on Methods and Techniques in Behavioral Research, volume 88. find
  22. Valerie Freeman, Julian Chan, Gina-Anne Levow, Richard Wright, Mari Ostendorf, Victoria Zayats, Yi Luan, Heather Morrison, Lauren Fox, Maria Antoniak, et al. 2014. ATAROS Technical Report 1: Corpus collection and initial task validation. U. Washington Linguistic Phonetics Lab. find
  23. James Milroy and Lesley Milroy. 2014. Real English: the grammar of English dialects in the British Isles. Routledge. find
  24. John S. Garofolo, Lori F. Lamel, William M. Fisher, Jonathan G. Fiscus, David S. Pallett, Nancy L. Dahlgren, and Victor Zue. 1993. TIMIT acoustic-phonetic continuous speech corpus. Linguistic data consortium, Philadelphia, 33. find
  25. Lennox Morrison. 2017. Speech analysis could now land you a promotion. BBC News, Jan. find
  26. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an ASR corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206–5210. IEEE. find
  27. Marylou Pausewang Gelfer and Victoria A. Mikos. 2005. The relative contributions of speaking fundamental frequency and formant frequencies to gender identification based on isolated vowels. Journal of Voice, 19(4):544–554. find
  28. Erwan Pépiot. 2014. Male and female speech: a study of mean f0, f0 range, phonation type and speech rate in Parisian French and American English speakers. In Speech Prosody 7, pages 305–309. find
  29. Luke Plunkett. 2010. Report: Kinect Doesn’t Speak Spanish (It Speaks Mexican). September. find
  30. M. Sawalha and M. Abu Shariah. 2013. The effects of speakers’ gender, age, and region on overall performance of Arabic automatic speech recognition systems using the phonetically rich and balanced Modern Standard Arabic speech corpus. In Proceedings of the 2nd Workshop of Arabic Corpus Linguistics W… find
  31. Fei Sha and Lawrence K. Saul. 2007. Large margin hidden Markov models for automatic speech recognition. Advances in neural information processing systems, 19:1249. find
  32. Aarti Shahani. 2015. Now algorithms are deciding whom to hire, based on voice. All Tech Considered: Tech, culture and connection, March. find
  33. Meghan Sumner and Arthur G. Samuel. 2009. The effect of experience on the perception and representation of dialect variants. Journal of Memory and Language, 60(4):487–501. find
  34. Antonio Torralba and Alexei A. Efros. 2011. Unbiased look at dataset bias. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pages 1521–1528. IEEE. find
  35. Peter Trudgill. 1972. Sex, covert prestige and linguistic change in the urban British English of Norwich. Language in society, 1(02):179–195. find
  36. Bert Vaux and Scott Golder. 2003. The Harvard dialect survey. Cambridge, MA: Harvard University Linguistics Department. find
  37. Barbara Wheatley and Joseph Picone. 1991. Voice Across America: Toward robust speaker-independent speech recognition for telecommunications applications. Digital Signal Processing, 1(2):45–63. find
  38. Ke Wu and Donald G. Childers. 1991. Gender recognition from speech. part i: Coarse analysis. The journal of the Acoustical society of America, 90(4):1828–1840. find
  39. W. Xiong, J. Droppo, X. Huang, F. Seide, M. Seltzer, A. Stolcke, D. Yu, and G. Zweig. 2016. find
  40. The Microsoft 2016 Conversational Speech Recognition System. arXiv preprint arXiv:1609.03528. — [arXiv:1609.03528](https://arxiv.org/abs/1609.03528) find
  41. Guoli Ye, Chaojun Liu, and Yifan Gong. 2016. Geolocation dependent deep neural network acoustic model for speech recognition. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5870–5874. IEEE. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tatman-2017-gender-dialect-bias-youtube-s/