📄 English Accent Accuracy Analysis in a State-of-the-Art Automatic Speech Recognition System

Summary

The paper asks whether a modern, heavily trained speech recognizer still performs unevenly across English accents. The authors took an ASR model pretrained on 44.5K hours of English audiobook speech from Multilingual LibriSpeech (MLS) and ran it, without adaptation, on Common Voice (CV). CV is a crowd-sourced corpus of read speech in which many recordings carry accent labels, while MLS itself has no accent annotation. They computed WER for each test utterance and then averaged it within each labeled accent group, reporting the standard error as error bars. American and Canadian speech got the lowest error rates, followed by Irish, English and New Zealand speech. Accents further from US and UK English, especially Asian varieties, were up to about 10 absolute WER points worse. This matches the likely makeup of the training data, which is dominated by North American and British readers. The study is a short workshop-style analysis. Its value is in showing that scale alone does not remove dialect bias, and that metadata-rich corpora like CV make this bias measurable.

Key points

• The recognizer is the MLS English model trained on 44.5K hours of LibriVox-derived audio. MLS is an 8-language extension of LibriSpeech.
• The LibriVox accents table suggests a strong skew among MLS readers: 48 US/Canada speakers and 26 UK speakers, versus only 1–2 from countries such as India or the Philippines.
• In the evaluation corpus, Common Voice, about 321K samples carry accent labels, from roughly 900 hours of volunteer-read scripted text.
• Utterance counts per accent vary widely:
  • large groups: US 171K, England 47K, India 34K, Australia 24K, Canada 21K
  • small groups: Bermuda 0.2K, Wales 0.3K, Malaysia 0.6K
• The metric is WER = (S + D + I) / N. It is averaged per accent group, and the worst groups trail the best by up to roughly 10 absolute points.

Interesting ideas

• I1. Break a single benchmark score into subgroup scores using metadata from an independent labeled corpus; this exposes biases that one global number hides.
• I2. Test a model on a corpus it never saw, drawn from a different source and recording style, to probe real generalization rather than in-domain fit.
• I3. Explain subgroup gaps by checking how well each group was represented in training, even when the training data lacks labels and only indirect evidence is available.
• I4. Show error bars per subgroup, because group sizes differ by orders of magnitude and small groups give noisy means.
• I5. Treat the observed gap as a starting point for asking which specific traits cause errors, not just as a call to collect more data.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): Report LAS separately per genre or English variety across UD treebanks, so we can see where the parser quietly underperforms instead of trusting one averaged figure.
• I2 (learn): Keep held-out evaluation sets from sources unrelated to the silver-data pipeline, so witness-parser agreement is not mistaken for true robustness.
• I3 (learn): When a tagger or parser fails on some register, estimate that register's share of the training treebanks before adding rules, so data gaps and model flaws are told apart.
• I4 (answer): When scoring readers on fables or maths problems by category, show per-category uncertainty so a bad score on a tiny category is not over-interpreted.
• I5 (tokens): Group errors by concrete features such as spelling variants or regional morphology to find targeted fixes in the en crate's tokenizer and lemmatizer.