Elamur Research⌕
☰
home / papers / 2023 / Improving Speech Recognition for African American English With Audio Classification

Improving Speech Recognition for African American English With Audio Classification

Paper · 2023 · Shefali Garg, Zhouyuan Huo, Khe Chai Sim et al.

Summary

Commercial speech recognizers make noticeably more errors on African American English (AAE) than on Mainstream American English (MAE). Fixing this with targeted training data is difficult because the available AAE recordings, mostly from CORAAL, are long-form interviews, while the production system handles short voice-search queries. The authors build an AAE-vs-other audio classifier on top of a pre-trained 600M-parameter conformer foundation model. They train it on modest amounts of CORAAL, YouTube and Common Voice US audio, cut into 5–20 s segments so it resembles short-form speech. Combined with coarse regional metadata, the classifier picks likely-AAE utterances out of a large pool of untranscribed voice-search queries. A teacher model transcribes these utterances, and the selected data is used to fine-tune a 120M-parameter streaming HAT recognizer. The fine-tuned model narrows the AAE–MAE word error rate gap by about 38.5% and does not hurt MAE accuracy. The paper shows that a classifier trained out of domain can serve as a data-mining tool for under-served language varieties when matched training data is lacking.

Key points

  • Classifier training data: roughly 20 h of CORAAL, 22.61 h of AAE YouTube and 18.03 h of MAE YouTube, with Common Voice US as negatives. The model is a BEST-RQ-pretrained conformer encoder plus a 2-layer Swish MLP head.
  • Design choices:
    • average pooling of encoder outputs;
    • a two-layer head, which beat one or three layers;
    • fine-tuning only the middle encoder layers (11–12), which beat tuning the bottom, top or all layers.
  • Classifier on short-form data: utterances scoring ≥0.7 are labelled AAE and those scoring <0.4 are labelled MAE. This gives 89% precision and 94.7% recall on human-labelled voice-search test data.
  • Mined data: 2.8M utterances from the Southern region set and 5.6M from the Top70 metro-area set, together about 1.8% of the unsupervised short-form pool. Before mining, southern-state data was only 5% of the training set.
  • Results:
    • AAE WER fell 7.7% relative.
    • The disparity reduction is reported as 38.5% (abstract) or 38.9% (intro).
    • The matched n-gram gap shrank 48.7%.
    • Simply adding more data, without classifier-based selection, did not improve AAE WER.

Interesting ideas

  • I1. A classifier trained on a small, out-of-domain labelled sample can be turned into a filter that mines in-domain examples from a huge unlabelled pool, with a cheap metadata prior (region) narrowing the search first.
  • I2. The two decision thresholds leave a gap between them (≥0.7 for the positive class, <0.4 for the negative), so ambiguous middle cases are never used as labels.
  • I3. Comparing error rates only on n-grams shared by both groups' references separates real modelling problems from mere vocabulary differences between test sets.
  • I4. With little adaptation data, updating just a couple of middle layers of a large pretrained encoder worked better than full fine-tuning, which risks overfitting.
  • I5. Success is measured as a relative shrinkage of the gap between two groups, and the majority group's quality is checked to make sure it did not drop.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): train a light text classifier on small dialect or informal-register samples and use it to pull matching sentences from raw corpora as candidates for silver parsing.
  • I2 (learn): when several witness parsers agree only partly, keep clear-agreement and clear-disagreement bands and drop the uncertain middle from silver training data.
  • I3 (tree): score the parser on standard versus dialect sentences restricted to matched lexical contexts, so that gaps reflect dialect syntax rather than unseen words.
  • I4 (tree): adapt gparse to a small dialect or genre treebank by unfreezing only a few middle transformer layers.
  • I5 (learn): track per-variety LAS gaps as a headline metric and treat standard-English accuracy as a control that must not regress.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States430 works · 424,363 citations🎓Stanford UniversityStanford · United States434 works · 314,419 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksSpeech & spoken language · 480 works

Cite

Cite as Shefali Garg, Zhouyuan Huo, Khe Chai Sim, Suzan Schwartz, Mason Chua, Alena Aksenova, Tsendsuren Munkhdalai, Levi King, Darryl Wright, Zion Mengesha, Dongseong Hwang, Tara Sainath, Françoise Beaufays and Pedro Moreno Mengibar. 2023. Improving Speech Recognition for African American English With Audio Classification. arXiv:2309.09996.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2309.09996 · arXiv 2309.09996

@misc{garg2023improving,
  title = {{Improving Speech Recognition for African American English With Audio Classification}},
  author = {Shefali Garg, Zhouyuan Huo, Khe Chai Sim, Suzan Schwartz, Mason Chua, Alena Aksenova, Tsendsuren Munkhdalai, Levi King, Darryl Wright, Zion Mengesha, Dongseong Hwang, Tara Sainath, Franccoise Beaufays and Pedro Moreno Mengibar},
  year = {2023},
  doi = {10.48550/arxiv.2309.09996},
  eprint = {2309.09996},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2309.09996}
}

References

20 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nina Markl and Stephen Joseph McNulty, “Language technology practitioners as language managers: Arbitrating data bias and predictive bias in asr,” in Proceedings of the Thirteenth Language Resources and Evaluation Conference. 2022, LREC Proceedings, pp. 6328–6339, European Language Resources Association (ELRA). find
  2. Allison Koenecke, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups, John R. Rickford, Dan Jurafsky, and Sharad Goel, “Racial disparities in automated speech recognition,” Proceedings of the National Academy of Sciences, pp. 7684–7689, 2020. ↗
  3. Valentin Vielzeuf and Grigory Antipov, “Are E2E ASR models ready for an industrial usage?,” arXiv e-prints, p. arXiv:2112.12572, 2021. ↗
  4. Alëna Aksënova, Antoine Bruguier, Amanda Ritchart-Scott, and Uri Mendlovic, “Algorithmic Exploration of American English Dialects,” in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 7374–7378. find
  5. Pranav Dheram, Murugesan Ramakrishnan, Anirudh Raju, I-Fan Chen, Brian King, Katherine L. Powell, Melissa Saboowala, Karan Shetty, and Andreas Stolcke, “Toward fairness in speech recognition: Discovery and mitigation of performance disparities,” in Interspeech, 2022. find
  6. Zion Mengesha, Courtney Heldreth, Michal Lahav, Juliana Sublewski, and Elyse Tuennerman, “I don’t think these devices are very culturally sensitive. impact of automated speech recognition errors on african americans,” Frontiers in Artificial Intelligence, p. 169, 2021. find
  7. Bernd Kortmann, Kerstin Lunkenheimer, and Katharina Ehret, Eds., eWAVE, 2020. find
  8. Arun Narayanan, Rohit Prabhavalkar, Chung-Cheng Chiu, David Rybach, Tara N. Sainath, and Trevor Strohman, “Recognizing long-form speech using streaming end-to-end models,” in 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2019, pp. 920–927. find
  9. Bo Li, Dongseong Hwang, Zhouyuan Huo, Junwen Bai, Guru Prakash, Tara N Sainath, Khe Chai Sim, Yu Zhang, Wei Han, Trevor Strohman, et al., “Efficient domain adaptation for speech foundation models,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5. find
  10. Prajit Ramachandran, Barret Zoph, and Quoc V. Le, “Searching for activation functions,” CoRR, 2017. ↗
  11. Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al., “Conformer: Convolution-augmented transformer for speech recognition,” arXiv preprint arXiv:2005.08100, 2020. ↗
  12. Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu, “Self-supervised learning with random-projection quantizer for speech recognition,” in International Conference on Machine Learning. PMLR, 2022, pp. 3915– 3924. ↗
  13. Tyler Kendall and Charlie Farrington, “The corpus of regional african american language,” Version, p. 1, 2018. find
  14. Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber, “Common Voice: A Massively-Multilingual Speech Corpus,” in Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), 2020, pp. 4211–4215. ↗
  15. Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh, “Set transformer: A framework for attention-based permutation-invariant neural networks,” in International conference on machine learning. PMLR, 2019, pp. 3744–3753. ↗
  16. Alena Aksenova, Zhehuai Chen, Chung-Cheng Chiu, Daan van Esch, Pavel Golik, Wei Han, Levi King, Bhuvana Ramabhadran, Andrew Rosenberg, Suzan Schwartz, and Gary Wang, “Accented speech recognition: Benchmarking, pre-training, and diverse data,” ArXiv, 2022. ↗
  17. Walt Wolfram and Natalie Schilling, American English: Dialects and Variation, John Wiley & Sons, 2015. find
  18. Ehsan Variani, David Rybach, Cyril Allauzen, and Michael Riley, “Hybrid autoregressive transducer (hat),” in ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 6139–6143. find
  19. Bo Li, Anmol Gulati, Jiahui Yu, Tara N. Sainath, Chung-Cheng Chiu, Arun Narayanan, Shuo-Yiin Chang, Ruoming Pang, Yanzhang He, James Qin, Wei Han, Qiao Liang, Yu Zhang, Trevor Strohman, and Yonghui Wu, “A better and faster end-to-end model for streaming asr,” in ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 5634–5638. ↗
  20. Mike Schuster and Kaisuke Nakajima, “Japanese and Korean Voice Search,” in International Conference on Acoustics, Speech and Signal Processing, 2012, pp. 5149–5152. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/garg-2023-improving-speech-recognition-african-american/