📄 Artie Bias Corpus: An Open Dataset for Detecting Demographic Bias in Speech Applications

Summary

Speech recognizers often work worse for some speaker groups than others, usually because their training data over-represents certain voices. Until this paper, there was no openly licensed resource for measuring that gap. The authors build the Artie Bias Corpus by taking English Common Voice test clips that carry self-reported age, gender or accent labels and re-checking each audio/transcript pair with expert annotators. The released corpus has 1,712 clips (about 2.4 hours) under CC0, along with open code for bias testing.

They define bias as a difference between the character error rate distributions of two speaker groups and test it with ANOVA. Applied to Mozilla's DeepSpeech v0.5.1, the method finds a significant accent gap, with US English transcribed better than Indian English. The gender gap is not significant. Fine-tuning improves several groups, but one fine-tuning step makes the gender gap significant.

The work shows that a small, carefully validated, demographically tagged test set can detect both bias and its unintended side effects.

Key points

• Source and filtering: the corpus starts from 1,903 demographically labeled clips in the English Common Voice test split (June 2019 release). A clip was kept only if two of three expert votes accepted it.
  • 167 clips were rejected as invalid.
  • Children's speech (22 clips), one sung clip and one clip with problematic text were also excluded.
• Annotator agreement: pairwise Kappa was 0.51, 0.28 and 0.29, which counts as fair to moderate. Rejection statistics showed no clear annotation bias against any group.
• Labels and skew: the corpus covers 3 gender classes, 17 accents and 8 age bands, but the distribution is uneven.
  • Gender: 1,431 male clips versus 257 female.
  • Accent: NA 562, US 558, Indian 264, England 131.
• Baseline model: DeepSpeech v0.5.1, trained on Fisher, LibriSpeech and Switchboard. CER was chosen over WER because it is more language-agnostic.
• Fine-tuning recipe: two stages, producing seven models in total (baseline, all-Common-Voice and five group-specific).
  • Stage 1: all of Common Voice for 100 epochs, learning rate 1e-5, dropout 15%.
  • Stage 2: a single demographic group for 20 more epochs, learning rate 1e-6.
• Fine-tuning results:
  • English English never improved significantly.
  • After the first stage, the gender gap grew to 3.55% CER and became significant (p = 0.042).
  • After female-only tuning the gap shrank to 2.8% but stayed significant.

Interesting ideas

• I1. Bias is treated as a statistical difference between per-group error distributions rather than a raw gap in means, which separates real gaps from noise in small slices.
• I2. Crowd-validated data gets a second expert pass, with agreement reported and rejections checked for skew toward any one group, so that cleaning does not introduce its own bias.
• I3. Adapting a model to new data can create a gap that did not exist before, so every slice should be re-measured after each tuning stage, not just the target one.
• I4. A small test set may simply lack the power to detect a real effect, so a non-significant result is not proof that no bias exists.
• I5. Character-level error was preferred because word boundaries are not universal, which keeps the metric comparable across languages.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): slice parser LAS by genre, dialect or construction and run a significance test before claiming a weakness or a fix.
• I2 (learn): audit silver data rejected by the witness parsers to see whether discards cluster on particular constructions, which would quietly bias training.
• I3 (tree): when fine-tuning gparse on a domain treebank, keep fixed per-slice regression checks so gains on one dialect do not hide losses on another.
• I4 (answer): with small school-test or fable-QA sets, report uncertainty and avoid reading "no difference" into tiny samples.
• I5 (tokens): score tokenizer and lemma errors at the character level, so results stay comparable when tokenization conventions differ between UD dialects.