Elamur Research⌕
☰
home / papers / 2024 / Quantification of stylistic differences in human- and ASR-produced transcripts of African American English

Quantification of stylistic differences in human- and ASR-produced transcripts of African American English

Paper · 2024 · Annika Heuser, Tyler Kendall, Miguel del Rio et al.

Summary

WER is the usual yardstick for speech recognition, but it mixes real mistakes with legitimate differences in how transcribers choose to write speech down. This matters most for underrepresented varieties such as African American English (AAE), whose spelling conventions are less settled. The authors took 27 CORAAL interview files (about 10 hours of audio) and collected six transcripts of each. One is the original CORAAL transcript. Three come from commercial services: Rev, Rev with an African American accent tag, and Amberscript. Two come from ASR systems: Rev ASR and Whisper large-v2. They aligned every ordered pair with fstalign. Rule-based tests then assigned each mismatch to one of three sources:

  • dropped or altered AAE morphosyntactic features,
  • different spellings of reductions and contractions,
  • verbatim choices such as fillers, restarts and repetitions.

Professional human transcripts of the same audio differ noticeably. ASR output reflects the habits of the people who transcribed its training data. One reference transcript is therefore not enough to rank systems fairly. The transcripts and code are publicly released.

Key points

  • Data: 27 CORAAL files (~10 h), 6 versions each (4 human, 2 machine). CORAAL linguists worked on whole files through several editing passes. The commercial services split audio into chunks for different transcribers, with checks by a senior transcriber.
  • Method: fstalign with default settings aligns all permutations of transcript pairs. Differences are then tested against three hypotheses: morpho-syntactic, reductions and verbatim.
  • The morpho-syntactic hypothesis has 17 tests built from the AAE feature lists of Rickford (1999) and Spears (2019). Some features cannot be tested from writing, e.g. stressed "stay", or participle-for-past forms that clash with irregular verbs.
  • Reduction tests cover pronoun contractions ("she'll" vs "she will", etc.) and the full/reduced pairs in the CORAAL user guide. Verbatim tests cover:
    • filler deletion,
    • filler swaps ("uh" vs "um"),
    • restart marking ("you-" vs "you"),
    • repetition deletion.
  • Whisper deletes more words than Rev ASR, which the authors read as Whisper sitting further toward the non-verbatim end. They also suggest that Rev's style guide should say clearly how non-standard grammar is to be transcribed.

Interesting ideas

  • I1. Instead of one error score, each aligned difference is attributed to a named hypothesis by a targeted test, so "error" splits into style, dialect and real mistakes.
  • I2. The hypotheses are ordered by how much they reveal or distort what the speaker meant: grammar erasure matters more than contraction spelling, which matters more than fillers.
  • I3. Reduced and full forms ("gonna"/"going to", "she'll"/"she will") are treated as an explicit equivalence class of stylistic variants.
  • I4. A single reference is judged insufficient, and several equally valid references of the same input are collected.
  • I5. The paper openly lists dialect features that cannot be detected from written form, rather than forcing a test for them.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when witness parsers or silver and gold trees disagree, classify the disagreement first (tokenization convention, UD annotation choice, true attachment error) so that only real errors feed the self-induced repair rules.
  • I2 (learn): weight evaluation differences by meaning impact, so a lost negation or habitual marker costs more than a disfluency mismatch.
  • I3 (tokens): map such variants to the same lemmas and features so that transcript style does not change the resulting tree or frames.
  • I4 (learn): evaluate parsers against more than one valid annotation or UD conversion of the same sentences, so that matching one annotator's habits is not mistaken for accuracy.
  • I5 (tree): when a construction is ambiguous on the surface, mark it as unresolved and keep the dialect form intact instead of silently normalizing it to standard English.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksSpeech & spoken language · 480 works

Cite

Cite as Annika Heuser, Tyler Kendall, Miguel del Rio, Quinten McNamara, Nishchal Bhandari, Corey Miller and Miguel Jette. 2024. Quantification of stylistic differences in human- and ASR-produced transcripts of African American English. arXiv:2409.03059.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.21437/interspeech.2024-2300 · arXiv 2409.03059

@misc{heuser2024quantification,
  title = {{Quantification of stylistic differences in human- and ASR-produced transcripts of African American English}},
  author = {Annika Heuser, Tyler Kendall, Miguel {del Rio}, Quinten McNamara, Nishchal Bhandari, Corey Miller and Miguel Jette},
  year = {2024},
  doi = {10.21437/interspeech.2024-2300},
  eprint = {2409.03059},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.21437/interspeech.2024-2300}
}

References

20 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. Bucholtz, “The politics of transcription,” Journal of Pragmatics, vol. 32, no. 10, pp. 1439–1465, 2000. ↗
  2. T. Kendall and C. Farrington, “The corpus of regional African American Language,” Version 2023.06, 2023. ↗
  3. S. Lanehart, The Oxford Handbook of African American Language. Oxford University Press, 2015. find
  4. L. D. Shriberg and G. L. Lof, “Reliability studies in broad and narrow phonetic transcription,” Clinical Linguistics & Phonetics, vol. 5, no. 3, pp. 225–279, 1991. ↗
  5. B. R. Patterson, N. C. Neupauer, P. A. Burant, S. C. Koehn, and A. T. Reed, “A preliminary examination of conversation analytic techniques: Rates of inter-transcriber reliability,” Western Journal of Communication (includes Communication Reports), vol. 60, no. 1, pp. 76–91, 1996. ↗
  6. D. Loakes, “Does automatic speech recognition (ASR) have a role in the transcription of indistinct covert recordings for forensic purposes?” Frontiers in Communication, vol. 7, p. 803452, 2022. ↗
  7. R. Love and D. Wright, “Specifying challenges in transcribing covert recordings: Implications for forensic transcription,” Frontiers in Communication, vol. 6, p. 797448, 2021. ↗
  8. M. Del Rı́o, C. Miller, J. Profant, J. Drexler-Fox, Q. McNamara, N. Bhandari, N. Delworth, I. Pirkin, M. Jetté, S. Chandra et al., “Accents in speech recognition through the lens of a world englishes evaluation set,” Research in Language, pp. 225–244, 2023.
  9. T. Jones, J. R. Kalbfeld, R. Hancock, and R. Clark, “Testifying while black: An experimental study of court reporter accuracy in transcription of African American English,” Language, vol. 95, no. 2, pp. e216–e252, 2019. ↗
  10. A. Koenecke, A. Nam, E. Lake, J. Nudell, M. Quartey, Z. Mengesha, C. Toups, J. R. Rickford, D. Jurafsky, and S. Goel, “Racial disparities in automated speech recognition,” Proceedings of the National Academy of Sciences, vol. 117, no. 14, pp. 7684–7689, 2020. ↗
  11. A. B. Wassink, C. Gansen, and I. Bartholomew, “Uneven success: automatic speech recognition and ethnicity-related dialects,” Speech Communication, vol. 140, pp. 50–70, 2022. ↗
  12. J. D. Fox and N. Delworth, “Improving contextual recognition of rare words with an alternate spelling prediction model,” arXiv preprint arXiv:2209.01250, 2022. ↗
  13. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in International Conference on Machine Learning. PMLR, 2023, pp. 28 492–28 518. ↗
  14. M. Del Rio, N. Delworth, R. Westerman, M. Huang, N. Bhandari, J. Palakapilly, Q. McNamara, J. Dong, P. Zelasko, and M. Jetté, “Earnings-21: A practical benchmark for asr in the wild,” arXiv preprint arXiv:2104.11348, 2021. ↗
  15. J. R. Rickford, African American Vernacular English. Blackwell Publishers, 1999, ch. Phonological and Grammatical Features of African American Vernacular English (AAVE). find
  16. A. K. Spears, “Rickford’s list of African American English grammatical features: an update,” in The Routledge companion to the work of John R. Rickford. Routledge, 2019, pp. 79–89. find
  17. J. Davydova and K. Hazen, “The role of linguistic structure in the perceptions of vernacular speech: Evidence from L1 English and English as a foreign language,” English World-Wide, vol. 42, no. 3, pp. 273–298, 2021. find
  18. L. Harrington, R. W. Rhodes, and V. Hughes, “Style variability in disfluency analysis for forensic speaker comparison,” International Journal of Speech, Language and the Law, pp. 31–58, 2021. ↗
  19. H. Fogel and L. C. Ehri, “Teaching African American English forms to standard American English-speaking teachers: Effects on acquisition, attitudes, and responses to student use,” Journal of Teacher Education, vol. 57, no. 5, pp. 464–480, 2006. ↗
  20. C. Miller, D. Silverman, V. Jurica, E. Richerson, R. Morris, and E. Mallard, “Embedding register-aware MT into the CAT workflow,” in Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 2: User Track), J. Campbell, A. Yanishevsky, J. Doyon, and D. Jones, Eds. Boston, MA: Association for Machine Translation in the Americas, Mar. 2018, pp. 275–282. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/heuser-2024-quantification-stylistic-differences-human-asr/