Elamur Research⌕
☰
home / papers / 2026 / Layer-wise Probing of wav2vec 2.0 and Whisper for Consonant Cluster Reduction in African American English

Layer-wise Probing of wav2vec 2.0 and Whisper for Consonant Cluster Reduction in African American English

Paper · 2026 · Hamid Mojarad and Kevin Tang

Summary

Speech recognizers make more errors for speakers of African American English (AAE), and consonant cluster reduction (CCR) is one documented contributor. In CCR, a word-final cluster loses its last stop, so "test" can be pronounced as "tes". The authors ask how two speech encoders represent this process internally. They compare wav2vec2-base (self-supervised) and Whisper-small (supervised). They build a probing set from CORAAL by forced-aligning the audio with Montreal Forced Aligner and a CMU dictionary that was extended with reduced pronunciation variants, so the aligner itself labels each token as canonical or reduced. Speaker-independent probes are then trained layer by layer on frozen representations for two tasks:

  • Reduction detection: telling reduced tokens from canonical ones.
  • Restoration: recovering which stop was underlyingly present in reduced tokens.

Both models separate reduced from canonical forms with high accuracy. Reduced tokens still carry traces of the missing stop. This suggests the models treat CCR as gradient, structured variation rather than plain deletion. The finding matters for diagnosing dialect bias in ASR beyond WER alone.

Key points

  • Data: the CORAAL subcorpora DCA, DCB and DTA. These cover 156 speakers (80 men, 76 women) from Washington, DC and Detroit, spanning four age brackets and three socioeconomic tiers.
  • Alignment pipeline:
    • MFA 2.2.17 with the CMU Pronouncing Dictionary.
    • A G2P model trained on CMU supplied pronunciations for missing words, which were then checked by hand.
    • Reduced variants were added for CCR-prone words.
    • A custom acoustic model was trained on the full audio.
  • Candidate tokens: alignment produced over 85,000 tokens across 48 two-consonant cluster types, and 12 frequent clusters were kept.
  • Filtering:
    • /l/- and /r/-initial clusters were removed because L-vocalization and R-deletion would confound the CCR signal.
    • Bimorphemic past-tense forms were removed, leaving seven monomorphemic clusters: /ft/, /nd/, /nt/, /st/, /sk/, /pt/, /mp/.
  • Final set: frequent words were capped at 400 tokens each (200 reduced, 200 canonical), giving 6,760 tokens (3,409 canonical, 3,351 reduced). Data and code are released on OSF.

Interesting ideas

  • I1. Writing dialect variants into the pronunciation lexicon as alternatives lets the aligner choose the variant, which turns a labeling problem into automatic annotation at corpus scale.
  • I2. A second "restoration" probe asks whether the reduced surface form still encodes what was removed, which separates "noticed a difference" from "knows the underlying form".
  • I3. Excluding categories where a second process overlaps (here L-vocalization and R-deletion) and restricting to monomorphemic cases keeps the measured effect attributable to a single phenomenon.
  • I4. Capping tokens per dominant word type, with balanced classes, stops a classifier from scoring well by memorizing a few frequent words like "just".
  • I5. Speaker-independent evaluation keeps the same individuals out of both training and test, preventing identity leakage from inflating results.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Add reduced or dialectal written forms (e.g. "gonna", dropped-g spellings) as explicit alternative entries tied to the canonical lemma, so the tagger's choice is recorded and explainable.
  • I2 (frames): Test whether elided or implicit arguments can be recovered from the frame context, as a check that the system restores the missing role rather than ignoring it.
  • I3 (learn): Build evaluation slices that isolate one construction and exclude confounded cases, so a score change points to one specific rule.
  • I4 (learn): Cap per-lemma counts in silver training data and test sets, so that learned repair rules reflect structure rather than high-frequency lexical items.
  • I5 (learn): Split evaluation by source document or author, so that agreement between the witness parsers is not inflated by shared style.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Heinrich Heine University DüsseldorfDüsseldorf · Germany33 works · 1,487 citations🎓University of FloridaGainesville · United States33 works · 364 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 worksSignal Processing · 452 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksInterpretability · 130 works

Cite

Cite as Hamid Mojarad and Kevin Tang. 2026. Layer-wise Probing of wav2vec 2.0 and Whisper for Consonant Cluster Reduction in African American English. arXiv:2606.23948.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2606.23948 · arXiv 2606.23948

@misc{mojarad2026layer,
  title = {{Layer-wise Probing of wav2vec 2.0 and Whisper for Consonant Cluster Reduction in African American English}},
  author = {Hamid Mojarad and Kevin Tang},
  year = {2026},
  doi = {10.48550/arxiv.2606.23948},
  eprint = {2606.23948},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2606.23948}
}

References

37 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. [13] E. A Shams, I. Gessinger, and J. Carson-Berndsen, “Uncovering syllable constituents in the self-attention-based speech representations of whisper,” in Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, Y. Belinkov, N. Kim, J. Jumelet, H. Mohebbi, A. Mueller, and H. Chen, Eds. Miami, Florida, US: Association for Computational Linguistics, Nov. 2024, pp. 238–247. [Online]. Available: https://doi.org/10.18653/v1/2024.blackboxnlp-1.16 ↗
  2. [1] Z. Mengesha, C. Heldreth, M. Lahav, J. Sublewski, and E. Tuennerman, “I don’t think these devices are very culturally sensitive.-impact of automated speech recognition errors on African Americans,” Frontiers in Artificial Intelligence, vol. Volume 4 - 2021, 2021. [Online]. Available: https://doi.org/10. 3389/frai.2021.725911 ↗
  3. [2] A. Koenecke, A. Nam, E. Lake, J. Nudell, M. Quartey, Z. Mengesha, C. Toups, J. R. Rickford, D. Jurafsky, and S. Goel, “Racial disparities in automated speech recognition,” Proceedings of the National Academy of Sciences, vol. 117, no. 14, pp. 7684–7689, 2020. [Online]. Available: https://doi.org/10.1073/pnas.1915768117 ↗
  4. [3] J. L. Martin and K. Tang, “Understanding Racial Disparities in Automatic Speech Recognition: The Case of Habitual “be”,” in Interspeech 2020, 2020, pp. 626–630. [Online]. Available: https://doi.org/10.21437/Interspeech.2020-2893 ↗
  5. [4] J. L. Martin and K. E. Wright, “Bias in automatic speech recognition: The case of African American Language,” Applied Linguistics, vol. 44, no. 4, pp. 613–630, 12 2022. [Online]. Available: https://doi.org/10.1093/applin/amac066 ↗
  6. [5] A. B. Wassink, C. Gansen, and I. Bartholomew, “Uneven success: automatic speech recognition and ethnicity-related dialects,” Speech Communication, vol. 140, pp. 50–70, 2022. [Online]. Available: https://doi.org/10.1016/j.specom.2022.03.009 ↗
  7. [6] I. Parra, “Interpretable sparse features for probing self-supervised speech models,” in The 14th International Joint Conference on Natural Language Processing and The 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, S. T.y.s.s, S. Shimizu, and Y. Gong, Eds. Mumbai, India: Association for Computational Linguistics, Dec. 2025, pp. 1–9. [Online]. Available: https://aclanthology.org/2025.ijcnlp-srw.1/ ↗
  8. [7] A. Pasad, J.-C. Chou, and K. Livescu, “Layer-wise analysis of a self-supervised speech representation model,” in 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2021, pp. 914–921. [Online]. Available: https://doi.org/10.1109/ASRU51503.2021.9688093 ↗
  9. [8] A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: a framework for self-supervised learning of speech representations,” in Proceedings of the 34th International Conference on Neural Information Processing Systems, ser. NIPS ’20. Red Hook, NY, USA: Curran Associates Inc., 2020. [Online]. Available: https://dl.acm.org/doi/abs/10.5555/3495724. 3496768 ↗
  10. [9] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in Proceedings of the 40th International Conference on Machine Learning, ser. ICML’23. JMLR.org, 2023. [Online]. Available: https://dl.acm.org/doi/10.5555/3618408.3619590 ↗
  11. [10] A. Pasad, B. Shi, and K. Livescu, “Comparative layer-wise analysis of self-supervised speech models,” in ICASSP 2023 - [14] I. Gessinger, E. A. Shams, and J. Carson-Berndsen, “Under the hood: phonemic restoration in transformer-based automatic speech recognition,” Computer Speech & Language, vol. 96, p. 101893, 2026. [Online]. Available: https://doi.org/10.1016/j.csl. 2025.101893 ↗
  12. [15] W. Labov, Sociolinguistic Patterns. Pennsylvania Press, 1972. find
  13. Philadelphia: University of [16] D. Schreier, Consonant Change in English Worldwide: Synchrony Meets Diachrony. New York: Palgrave Macmillan, 2005. find
  14. [17] W. Wolfram, Dialect in Society. John Wiley & Sons, Ltd, 2017, ch. 7, pp. 107–126. [Online]. Available: https://doi.org/10.1002/9781405166256.ch7 ↗
  15. [18] G. R. Guy, “Explanation in variable phonology: An exponential model of morphological constraints,” Language Variation and Change, vol. 3, no. 1, pp. 1–22, 1991. [Online]. Available: https://doi.org/10.1017/S0954394500000429 ↗
  16. [19] E. R. Thomas and G. Bailey, “Segmental phonology of African American English,” in The Oxford Handbook of African American Language. Oxford University Press, 07 2015. [Online]. Available: https://doi.org/10.1093/oxfordhb/ 9780199795390.013.13 ↗
  17. [20] H. Mojarad and K. Tang, “Automatic Speech Recognition of African American English: Lexical and Contextual Effects,” in Interspeech 2025, 2025, pp. 3883–3887. [Online]. Available: https://doi.org/10.21437/Interspeech.2025-1511 ↗
  18. [21] L. Kim and C. Gendrot, “Using wav2vec 2.0 for phonetic classification tasks: methodological aspects,” in Interspeech 2024, 2024, pp. 1530–1534. [Online]. Available: https://doi.org/10.21437/Interspeech.2024-1155 ↗
  19. [22] A. Batra, B. Kar, and P. K. Das, “Exploring Whisper embeddings for stutter detection: A layer-wise study,” in 2025 33rd European Signal Processing Conference (EUSIPCO), 2025, pp. 61–65. [Online]. Available: https://doi.org/10.23919/EUSIPCO63237. 2025.11226086 ↗
  20. [23] Z. Yue, D. Kayande, Z. Cvetkovic, and E. Loweimi, “Probing Whisper for dysarthric speech in detection and assessment,” 2025. [Online]. Available: https://doi.org/10.48550/arXiv.2510.04219 ↗
  21. [24] T. Kendall and C. Farrington, “The Corpus of Regional African American Language,” 2023, publisher: The Online Resources for African American Language Project. [Online]. Available: https://oraal.uoregon.edu/coraal ↗
  22. [25] T. Kendall, C. Vaughn, C. Farrington, K. Gunter, J. McLean, C. Tacata, and S. Arnson, “Considering performance in the automated and manual coding of sociolinguistic variables: Lessons from variable (ING),” Frontiers in Artificial Intelligence, vol. 4, 2021. [Online]. Available: https://doi.org/10.3389/frai. 2021.648543 ↗
  23. [26] M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger, “Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi,” in Proc. Interspeech 2017, 2017, pp. 498–502. [Online]. Available: https://doi.org/10.21437/ Interspeech.2017-1386 ↗
  24. [27] H. Mohebbi, G. Chrupała, W. Zuidema, and A. Alishahi, “Homophone disambiguation reveals patterns of context mixing in speech transformers,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali, Eds. Singapore: Association for Computational Linguistics, Dec. 2023, pp. 8249–8260. [Online]. Available: https://doi.org/10.18653/v1/2023.emnlp-main.513 ↗
  25. [28] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and Édouard Duchesnay, “Scikit-learn: Machine learning in python,” Journal of Machine Learning Research, vol. 12, no. 85, pp. 2825–2830, 2011. [Online]. Available: http://jmlr.org/papers/v12/pedregosa11a.html ↗
  26. [29] R. Bayley and D. Villarreal, Coronal Stop Deletion in a Rural South Texas Community, ser. Studies in English Language. Cambridge University Press, 2019, p. 198–214. [Online]. Available: https://doi.org/10.1017/9781316162316.008 ↗
  27. [30] F. Grosjean, “Spoken word recognition processes and the gating paradigm,” Perception & Psychophysics, vol. 28, no. 4, pp. 267–283, 1980. [Online]. Available: https://doi.org/10.3758/ BF03204386 ↗
  28. [31] O. Agaoglu, Z. Yue, and Y. Zhang, “How does OpenAI’s Whisper interpret dysarthric speech?” TU Delft, Tech. Rep., 2024, student thesis/project report. [Online]. Available: https://resolver.tudelft.nl/uuid:47837feb-1b2e-4bba-9fad-f92d84024abb [32] A. de la Fuente and D. Jurafsky, “A layer-wise analysis of Mandarin and English suprasegmentals in SSL speech models,” in Interspeech 2024, 2024, pp. 1290–1294. [Online]. Available: https://doi.org/10.21437/Interspeech.2024-2341 ↗
  29. [33] H. Mitterer, O. Scharenborg, and J. M. McQueen, “Phonological abstraction without phonemes in speech perception,” Cognition, vol. 129, no. 2, pp. 356–361, 2013. [Online]. Available: https://doi.org/10.1016/j.cognition.2013.07.011 ↗
  30. [34] W. Wolfram and E. R. Thomas, The development of African American English. Blackwell Publishers, 2002. [Online]. Available: https://doi.org/10.1002/9780470690178 ↗
  31. [35] W. Labov, “The child as linguistic historian,” Language Variation and Change, vol. 1, no. 1, p. 85–97, 1989. [Online]. Available: https://doi.org/10.1017/S0954394500000120 ↗
  32. [36] W. Wolfram, “Reexamining the development of African American English: Evidence from isolated communities,” Language, vol. 79, pp. 282–316, 06 2003. [Online]. Available: https://doi.org/10.1353/lan.2003.0144 ↗
  33. [37] E. R. Thomas, “Phonological and phonetic characteristics of African American Vernacular English,” Language and Linguistics Compass, vol. 1, no. 5, pp. 450–475, 2007. [Online]. Available: https://doi.org/10.1111/j.1749-818X.2007.00029.x ↗
  34. [38] L. J. Green, African American English: A Linguistic Introduction. Cambridge University Press, 2002. find
  35. [39] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451–3460, 2021. [Online]. Available: https://doi.org/10.1109/TASLP.2021.3122291 ↗
  36. [40] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y. Qian, Y. Qian, J. Wu, M. Zeng, X. Yu, and F. Wei, “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022. [Online]. Available: https://doi.org/10.1109/JSTSP.2022.3188113 ↗
  37. [41] V. Pratap, A. Tjandra, B. Shi, P. Tomasello, A. Babu, S. Kundu, A. Elkahky, Z. Ni, A. Vyas, M. Fazel-Zarandi, A. Baevski, Y. Adi, X. Zhang, W.-N. Hsu, A. Conneau, and M. Auli, “Scaling speech technology to 1,000+ languages,” Journal of Machine Learning Research, vol. 25, no. 97, pp. 1–52, 2024. [Online]. Available: http://jmlr.org/papers/v25/23-1318.html ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mojarad-2026-layer-wise-probing-wav2vec-2/