Elamur Research⌕
☰
home / papers / 2024 / Self-supervised Speech Representations Still Struggle with African American Vernacular English

Self-supervised Speech Representations Still Struggle with African American Vernacular English

Paper · 2024 · Kalvin Chang, Yi-Hui Chou, Jiatong Shi et al.

Summary

The paper asks whether self-supervised speech models, which have helped low-resource languages, can narrow the long-known ASR accuracy gap between African American Vernacular English (AAVE) and Mainstream American English (MAE). The authors froze four SSL encoders (wav2vec 2.0, HuBERT, WavLM, XLS-R) and used each as the feature front-end of a Transformer encoder-decoder ASR system, following the ML-SUPERB setup in ESPnet. A model with FBANK features served as the baseline. All systems were trained only on LibriSpeech train clean 100, a corpus where AAVE grammar is rare. They were then tested zero-shot on CORAAL (AAVE interviews) and on the Nationwide Speech Project (NSP, MAE). Propensity score matching was used to balance confounders between the two test sets. The SSL front-ends did not remove the disparity. Error rates also grew as utterances carried more phonological and morphosyntactic AAVE features. The takeaway is that pre-training on out-of-domain audio is not enough by itself to fix dialect bias, which has real costs in hiring, healthcare and everyday voice interfaces.

Key points

  • Four frozen SSL models were tested, with 317 million SSL parameters kept fixed during training. They fed a 2-layer Transformer encoder and a 6-layer decoder trained with CTC on subword (sentencepiece unigram) tokens. Decoding used beam search with beam 60 and CTC weight 0.3.
  • Test data:
    • CORAAL: the 3-city subset (Princeville, Rochester, D.C.) borrowed from earlier work.
    • NSP: 60 young white speakers, 10 from each of six dialect regions, recorded in a sound-attenuated booth.
    • Both sets were downsampled to 16 kHz, and NSP was segmented with the Montreal Forced Aligner.
  • Propensity score matching reduced CORAAL from 19.39 h to 4.02 h, against 3.33 h of NSP. This lowered the average CORAAL speaker age from 43.41 to 25.38 (NSP: 19.18). Audio quality was tracked with DNS-MOS and UT-MOS.
  • One text normalization was applied to both corpora so it matches LibriSpeech conventions. It keeps apostrophes, fillers and dysfluencies, expands abbreviations, and rewrites reduced forms such as "sposta".
  • Prior work cited for context: commercial ASR reached WER 0.35 for Black speakers versus 0.19 for white speakers. Google's USM reached 11.2 WER on CORAAL, which shows the gap can be closed with large, diverse data.

Interesting ideas

  • I1. Matching test sets on confounders such as age and recording quality before comparing them separates a dialect effect from mere covariate differences.
  • I2. Error analysis plots error rate against how many dialect features an item contains, turning a vague bias claim into a dose-response curve.
  • I3. Checking how often key dialect features occur in the training corpus explains why a model trained there fails on them.
  • I4. Applying one shared normalization to both comparison sets, while deliberately keeping fillers and dysfluencies, keeps the comparison fair without erasing the speech itself.
  • I5. Freezing the pre-trained encoder isolates the question of what its representations already capture, separate from what fine-tuning adds.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Before reporting a parser gap between, say, fable text and news, match the samples on sentence length and source quality so the measured gap reflects the variety and not the noise.
  • I2 (tree): Count AAVE morphosyntactic markers (null copula, invariant be, perfective done, ain't) per sentence and track LAS as that count rises, to locate exactly which constructions break the parser.
  • I3 (learn): Audit the treebanks and silver data for the frequencies of non-mainstream constructions, and flag zero-coverage phenomena before they become silent parser errors.
  • I4 (tokens): Map reduced forms like "sposta" to canonical lemmas in crate en while keeping the original surface form, so the tree and frames stay explainable on vernacular input.
  • I5 (learn): When testing gparse on a new English variety, compare frozen-encoder and fine-tuned runs to tell representation gaps apart from training-data gaps.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 worksSignal Processing · 452 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksSpeech & spoken language · 480 works

Cite

Cite as Kalvin Chang, Yi-Hui Chou, Jiatong Shi, Hsuan-Ming Chen, Nicole Holliday, Odette Scharenborg and David R. Mortensen. 2024. Self-supervised Speech Representations Still Struggle with African American Vernacular English. arXiv:2408.14262.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2408.14262 · arXiv 2408.14262

@misc{chang2024self,
  title = {{Self-supervised Speech Representations Still Struggle with African American Vernacular English}},
  author = {Kalvin Chang, Yi-Hui Chou, Jiatong Shi, Hsuan-Ming Chen, Nicole Holliday, Odette Scharenborg and David R. Mortensen},
  year = {2024},
  doi = {10.48550/arxiv.2408.14262},
  eprint = {2408.14262},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2408.14262}
}

References

45 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. [1] L. J. Green, African American English: a linguistic introduction. Cambridge University Press, 2002. find
  2. [2] W. Wolfram, “Sociolinguistic premises and the nature of nonstandard dialects,” Communication Education, vol. 19, no. 3, pp. 177– 184, 1970. ↗
  3. [3] J. Baugh, “Linguistic profiling,” in Black linguistics. Routledge, 2005, pp. 167–180. find
  4. [4] K. E. Wright, “Housing policy and linguistic profiling: An audit study of three american dialects,” Language, 2023. ↗
  5. [5] Z. Mengesha, C. Heldreth et al., ““I Don’t Think These Devices are Very Culturally Sensitive.”—Impact of Automated Speech Recognition Errors on African Americans,” Frontiers in Artificial Intelligence, vol. 4, p. 169, 2021. find
  6. [6] J. L. Martin and K. E. Wright, “Bias in automatic speech recognition: The case of African American Language,” Applied Linguistics, vol. 44, no. 4, pp. 613–630, 2023. ↗
  7. [7] K. Wenzel, N. Devireddy et al., “Can voice assistants be microag-gressors? cross-race psychological responses to failures of automatic speech recognition,” in Proceedings of CHI 2023, 2023, pp. 1–14. find
  8. [8] K. Crawford, “The trouble with bias,” 2017, conference on Neural Information Processing Systems. find
  9. [9] A. Koenecke, A. Nam et al., “Racial disparities in automated speech recognition,” PNAS, vol. 117, no. 14, pp. 7684–7689, 2020. ↗
  10. [10] T. Kendall and C. Farrington, “The Corpus of Regional African American Language,” 2018. find
  11. [11] J. Cunningham, S. L. Blodgett et al., “Understanding the impacts of language technologies’ performance disparities on African American language speakers,” in Findings of ACL 2024, 2024, pp. 12 826–12 833. find
  12. [12] C. N. Harrington, R. Garg et al., ““It’s Kind of Like Code-Switching”: Black older adults’ experiences with a voice assistant for health information seeking,” in Proc. CHI 2022, ser. CHI ’22, New York, NY, USA, 2022. find
  13. [13] A. Mohamed, H.-y. Lee et al., “Self-supervised speech representation learning: A review,” IEEE Journal of Selected Topics in Signal Processing, 2022. ↗
  14. [22] J. L. Martin, “Automatic speech recognition systems, spoken corpora, and african american language: An examination of linguistic bias and morphosyntactic features,” Ph.D. dissertation, University of Florida, 2022. find
  15. [23] L.-F. Lai and N. Holliday, “Exploring sources of racial bias in automatic speech recognition through the lens of rhythmic variation,” in Proc. INTERSPEECH, vol. 2023, 2023, pp. 1284–1288. find
  16. [24] A. B. Wassink, C. Gansen, and I. Bartholomew, “Uneven success: automatic speech recognition and ethnicity-related dialects,” Speech Communication, vol. 140, pp. 50–70, 2022. ↗
  17. [25] M. Lehr, K. Gorman, and I. Shafran, “Discriminative pronunciation modeling for dialectal speech recognition,” in Proc. INTER-SPEECH, 2014. find
  18. [26] R. Dorn, “Dialect-specific models for automatic speech recognition of African American Vernacular English,” in Proceedings of the Student Research Workshop Associated with RANLP 2019. Varna, Bulgaria: INCOMA Ltd., Sep. 2019, pp. 16–20.
  19. [27] A. Johnson, K. Everson et al., “Automatic Dialect Density Estimation for African American English,” in Proc. INTERSPEECH, 2022, pp. 1283–1287.
  20. [28] A. Johnson, V. M. Shetty et al., “Leveraging multiple sources in automatic african american english dialect detection for adults and children,” in Proc. ICASSP. IEEE, 2023, pp. 1–5. find
  21. [29] A. Johnson, N. B. Shankar et al., “An exploratory study on dialect density estimation for children and adult’s African American English,” The Journal of the Acoustical Society of America, vol. 155, no. 4, pp. 2836–2848, 2024. find
  22. [30] L. Sarı, M. Hasegawa-Johnson, and C. D. Yoo, “Counterfactually fair automatic speech recognition,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3515–3525, 2021. find
  23. [31] H. Gao, X. Wang et al., “Seamless equal accuracy ratio for inclusive ctc speech recognition,” Speech Communication, vol. 136, pp. 76–83, 2022. ↗
  24. [32] Y. Meng, Y.-H. Chou et al., “Don’t speak too fast: The impact of data bias on self-supervised speech models,” in Proc. ICASSP, 2022, pp. 3258–3262. find
  25. [33] A. Aksënova, Z. Chen et al., “Accented speech recognition: Benchmarking, pre-training, and diverse data,” arXiv preprint arXiv:2205.08014, 2022. ↗
  26. [14] R. Coto-Solano, S. A. Nicholas et al., “Development of automatic speech recognition for the documentation of Cook Islands Māori,” in Proc. LREC. Marseille, France: European Language Resources Association, June 2022, pp. 3872–3882. find
  27. [34] S. Garg, Z. Huo et al., “Improving speech recognition for African American English with audio classification,” in Proc. ICASSP, 2024, pp. 12 356–12 360.
  28. [15] K. Choi and E. J. Yeo, “Opening the black box of wav2vec feature encoder,” arXiv preprint arXiv:2210.15386, 2022. ↗
  29. [35] Y. Zhang, W. Han et al., “Google USM: Scaling automatic speech recognition beyond 100 languages,” arXiv preprint arXiv:2303.01037, 2023. ↗
  30. [16] D. Wells, H. Tang, and K. Richmond, “Phonetic analysis of self-supervised representations of english speech,” in Proc. INTER-SPEECH. ISCA, 2022, pp. 3583–3587. find
  31. [17] P. Cormac English, J. D. Kelleher, and J. Carson-Berndsen, “Domain-informed probing of wav2vec 2.0 embeddings for phonetic features,” in Proc. SIGMORPHON 2019, Seattle, Washington, Jul. 2022, pp. 83–91. find
  32. [36] C. G. Clopper and D. B. Pisoni, “The Nationwide Speech Project: A new corpus of American English dialects,” Speech Communication, vol. 48, no. 6, pp. 633–644, 2006. ↗
  33. [37] W. Labov, S. Ash, and C. Boberg, The atlas of North American English: Phonetics, phonology and sound change. Mouton de Gruyter, 2006. find
  34. [18] K. Martin, J. Gauthier et al., “Probing Self-supervised Speech Models for Phonetic and Phonemic Information: A Case Study in Aspiration,” in Proc. INTERSPEECH, 2023, pp. 251–255. find
  35. [38] M. McAuliffe, M. Socolof et al., “Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi,” in Proc. INTER-SPEECH, 2017. find
  36. [19] M. Bartelds, W. de Vries et al., “Neural representations for modeling variation in speech,” Journal of Phonetics, vol. 92, p. 101137, 2022. ↗
  37. [39] C. K. Reddy, V. Gopal, and R. Cutler, “DNSMOS: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors,” in Proc. ICASSP. IEEE, 2021, pp. 6493–6497. find
  38. [20] M. Bartelds and M. Wieling, “Quantifying language variation acoustically with few resources,” in Proc. NAACL, Seattle, United States, Jul. 2022, pp. 3735–3741. find
  39. [40] T. Saeki, D. Xin et al., “UTMOS: Utokyo-sarulab system for voicemos challenge 2022,” arXiv preprint arXiv:2204.02152, 2022. ↗
  40. [21] R. Tatman and C. Kasten, “Effects of talker dialect, gender & race on accuracy of bing speech and youtube automatic captions.” in Proc. INTERSPEECH, 2017, pp. 934–938. find
  41. [41] T. Kudo, “Subword regularization: Improving neural network translation models with multiple subword candidates,” arXiv preprint arXiv:1804.10959, 2018. ↗
  42. [42] J. Choe, Y. Chen et al., “Language-specific effects on automatic speech recognition errors for world englishes,” in Proc. COLING, Gyeongju, Republic of Korea, Oct. 2022, pp. 7177–7186.
  43. [43] A. Kline and Y. Luo, “Psmpy: a package for retrospective cohort matching in python,” in 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). find
  44. [44] C. Farrington, “Incomplete neutralization in african american english: The case of final consonant voicing,” Language Variation and Change, vol. 30, no. 3, pp. 361–383, 2018. ↗
  45. [45] S. Feng, B. Halpern et al., “Towards inclusive automatic speech recognition,” Computer Speech and Language, vol. 84, 2023. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chang-2024-self-supervised-speech-representations-still/