📄 The MGB-2 Challenge: Arabic Multi-Dialect Broadcast Media Recognition

Summary

MGB-2 is a shared evaluation for automatic recognition of Arabic broadcast speech, held for SLT-2016. The earlier English MGB focused on variety across TV genres; this edition focuses on variety across Arabic dialects. The organisers collected about 1,200 hours of Aljazeera TV recordings from 2005–2015, drawn from 19 programme series of conversations, interviews and reports. The broadcaster's transcripts were edited, not verbatim, and had no timestamps. To make them usable, the organisers decoded the audio with the QCRI Arabic LVCSR system and matched the output against the text using Smith–Waterman local alignment. They kept per-segment quality scores instead of discarding imperfect data. Participants also received over 110M words of Aljazeera.net text for language modelling, a grapheme lexicon, a phoneme lexicon, and an open-source baseline on GitHub. The challenge had two tasks: transcription and word-level alignment. Of 13 teams, eight submitted final systems. The best system (QCRI) reached 14.7% WER, compared with a 34% baseline. Nobody entered the alignment task. The result is a large, closed-condition benchmark for dialectal Arabic ASR.

Key points

• Training data: about 1,200 hours split into more than 550,000 segments.
  • By genre: conversation 63%, interview 19%, report 18%.
  • By language: over 70% is MSA; the rest is EGY, GLF, LEV and NOR dialects.
  • About 300 hours of short highlight clips were removed.
• Alignment quality: measured with AnchorRate, the share of transcript words matched to ASR output.
  • Results: 48% exact match, 15% approximate match, 37% no match.
  • Each segment's metadata includes AWD, PMER and WMER.
• Lexicons: a grapheme lexicon with over 900K entries, and a V2P-based phoneme lexicon with over 500K words averaging about four pronunciations per word.
• Evaluation sets: dev and eval are 10 hours each, taken from the final month of 2015. They were transcribed verbatim and segmented by hand. Four WER variants were reported; the official one uses the GLM and excludes overlapped speech.
• Official results:
  • Transcription WER: QCRI 14.7%, LIUM 16.7%, MIT 17.3%, NDSC 18.2%.
  • Alignment baseline, with a 100 ms tolerance window: precision 0.83, recall 0.70, F 0.76.

Interesting ideas

• I1. Light supervision from approximate transcripts: decode the audio, align the result to edited human text using both exact and fuzzy matching, and fill unmatched spans by linear time interpolation. This turns existing subtitles, parliament records or news transcripts into ASR training data without new annotation.
• I2. Release quality scores instead of a filtered set: each segment ships with WMER, PMER and AWD, so users set their own thresholds. For example, NDSC kept segments with zero WMER and AWD between 0.3 and 0.7 s, leaving about 680 hours. Average word duration is also a cheap signal for spotting misaligned segments in any corpus.
• I3. Layered WER scoring: the organisers scored raw text, then text without punctuation and diacritics, then GLM-mapped text, then text with normalised letter variants. This separates spelling-convention mismatches from real recognition errors, which helps for any language with inconsistent orthography.
• I4. Comparability by design: all teams trained under a closed condition on the same data. Dev and eval came from the most recent month, so they could not overlap with training. Overlapped speech was scored but kept out of the main ranking.