📄 Discovering Canonical Indian English Accents: A Crowdsourcing-based Approach

Summary

Speech recognizers lose accuracy on accents absent from their training data. Adapting acoustic models per accent requires knowing which accents exist. For Indian English, no inventory of canonical accents or labeled accent corpus existed. The authors therefore used listeners as the measuring instrument. In a mobile app, participants heard Indian English speech clips and marked on a map of India where they thought each speaker came from, choosing either one of five regions or one of the states. The speaker's geographic location stood in for their first language. Agreement and confusion patterns between listener guesses and true locations suggest which accents are distinct and which blur together. South Indian speech was the most reliably placed, followed by North Indian speech, while Central India was often mistaken for the North. State-level agreement was generally low.

The team then built a gamified web version, seeded with recordings that pilot participants made along with their self-reported first language, to collect both judgments and new accented speech at scale. This matters because it offers a cheap route to an accent taxonomy and adaptation data where no ground truth exists.

Key points

• Pilot data: 250 clips (10 per city from 25 cities and towns across 5 regions), drawn from in-house spoken queries to a speech recognizer.
  • Clips were manually filtered to be long enough to judge.
  • Clips that mentioned places were excluded.
• Pilot participants: 60 people each judged 15 clips (3 per region), so every file received at least three labels.
  • Participants could skip clips.
  • They could optionally give demographics: first language, other languages, education, and places lived.
• Findings:
  • About 32% of judgments were made at state level; the rest were region-level.
  • South India showed the highest agreement and North India the second highest.
  • Central India was confused with North India.
  • Uttar Pradesh, Tamil Nadu and Karnataka were placed more accurately than other states.
• Recordings: Roughly 60 participants recorded the GMU Accent Project elicitation paragraph on phones, so the audio includes realistic background noise.
• Web game: Each player judges 10 files and scores 10 points for a correct state and 5 for a correct region. Scores fall into five 20-point bands with humorous labels.
  • All 10 lab testers scored 40 or less.
  • Testers blamed background noise and mild urban accents for the low scores.

Interesting ideas

• I1. Discover categories from confusions: Rather than fixing accent classes in advance, the authors let listener confusion matrices reveal which classes are separable and which merge. Low separability is itself informative, since it hints at a "neutral" variety.
• I2. Two-level annotation with inferred coarse labels: Annotators choose a coarse or fine label as they feel confident, and fine answers are rolled up to the coarse level for analysis. The share of fine-grained answers also measures annotator confidence.
• I3. Remove items that leak the answer: Clips mentioning a place were excluded so judgments rested on the accent alone, not on content.
• I4. Hierarchical partial credit in scoring: An exact state match earns full points and a correct region earns half, so near misses are rewarded proportionally.
• I5. Per-item feedback with replay: Showing players which items they missed, letting them replay those clips, and mapping right and wrong answers turned annotation into engaging, self-improving play.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Confusion matrices between witness parsers or LLM teacher tiers could show which label distinctions are reliably separable and which should be merged or flagged as unreliable in silver data.
• I2 (learn): Teacher or human annotation of frames could permit a broad role or class when unsure and a specific one when sure, mapping both onto a hierarchy and tracking the fine-label rate as a confidence signal.
• I3 (answer): Test items in school-course or fable QA should be screened so the answer cannot be read off surface cues, which works as a built-in negative control against shortcut reading.
• I4 (world): Evaluating entity typing or ontology placement could give partial credit for a correct ancestor in WordNet or Wikidata, separating near misses from wrong answers.
• I5 (learn): Self-play and repair-rule induction could produce per-item error reports grouped by phenomenon, making clear where a learner fails rather than giving only an aggregate score.