Speech Recognition Challenge in the Wild: Arabic MGB-3

Summary

The Arabic MGB-3 Challenge is a shared evaluation that moves Arabic speech recognition from clean broadcast news to dialectal, multi-genre web video. The organisers took the first 12 minutes of 80 Egyptian YouTube videos across seven genres, including cooking, fashion, drama, sports and TEDx talks. This gave 16 hours, divided into adaptation, development and test portions. That amount is too small to train a recogniser alone, so participants reused the 1,200-hour MGB-2 Al Jazeera training set and treated the Egyptian material as adaptation data. Egyptian Arabic has no fixed spelling, so four annotators transcribed the data and systems were scored with multi-reference WER (MR-WER). A second task covered five-way Arabic dialect identification, with audio, lexical features and i-vectors supplied. A Kaldi TDNN baseline reached 22.6% WER on MGB-2 dev but about 63% average WER on Egyptian dev data. The best submissions achieved:

37.5% AV-WER and 29.3% MR-WER on MGB-3,
13.2% WER on MGB-2 test,
about 80% dialect accuracy.

These results set a benchmark for low-resource speech without standardised spelling.

Key points

MGB-3 ASR data: 80 YouTube videos from Egypt, spanning seven genres: drama, comedy, sports, fashion, cooking, content for families and children, and science/TEDx. Splits are 4.6 h adaptation, 4.8 h dev and 6.0 h test, and overlapping speech is marked separately.
Annotator disagreement: Four independent transcriptions differ pairwise by roughly 13% WER after normalising the alef, yah and hah characters. The MR-WER scoring code is released at github.com/qcri/multiRefWER.
Baseline: Kaldi recipe trained on MGB-2 (1,200 h, tripled via speed/volume perturbation), using a lattice-free MMI TDNN, a 950k-word grapheme lexicon and a trigram LM.
- MGB-2 dev: 22.6% WER.
- Unadapted MGB-3 dev: 63.2% AV-WER and 58.0% MR-WER.
ADI task: Five classes (EGY, GLF, LAV, NOR, MSA), with about 53.6 h of training data and roughly 10 h each for dev and test, all from Al Jazeera. The organisers supplied ASR-derived lexical features and 400-dim i-vectors built on bottleneck features. The baseline is a multi-class SVM, and data and code are at github.com/qcri/dialectID.
Outcomes: Thirteen teams submitted ten systems. Helpful ASR techniques were sub-word LMs and merging multiple references via confusion networks for training. Strong ADI systems combined acoustic and lexical cues and fought domain mismatch with GANs, Siamese networks and recursive whitening.

Interesting ideas

I1. Score against several valid transcripts: Counting a word as correct if any reference contains it avoids penalising legitimate spelling variants. This fits any language or register without settled orthography, such as mixed or colloquial speech.
I2. Quantify disagreement before reporting results: Measuring pairwise WER between annotators, both raw and after light character normalisation, gives a ceiling for interpreting system errors. It also shows how much "error" is really orthographic noise.
I3. Pair a large general corpus with a small in-domain set: Big out-of-domain training data plus a few hours of genre-balanced in-domain data for adaptation and tuning is a cheap, realistic recipe for new dialects or domains.
I4. Use multiple references as training signal: Combining alternative transcriptions into confusion networks for training, not just evaluation, extracts extra value from costly multi-annotator data.
I5. Release precomputed features with raw data: Shipping i-vectors and ASR-derived lexical features alongside the audio lowers the entry barrier for participants. It also makes acoustic and text-based classifiers directly comparable on the same identification task.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

64 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 59 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1709.07276 · arXiv 1709.07276

@misc{ali2017speech,
  title = {{Speech Recognition Challenge in the Wild: Arabic MGB-3}},
  author = {Ali, Ahmed and Vogel, Stephan and Renals, Steve},
  year = {2017},
  doi = {10.48550/arxiv.1709.07276},
  eprint = {1709.07276},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1709.07276}
}

References

20 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

P. Bell, M. JF Gales, T. Hain, J. Kilgour, P. Lanchantin, X. Liu, A. McParland, S. Renals, O. Saz, M. Wester, and P. Woodland, “The MGB challenge: Evaluating multi-genre broadcast media recognition,” in ASRU, 2015. find
A. Ali, P. Bell, J. Glass, Y. Messaoui, H. Mubarak, S. Renals, and Y. Zhang, “The MGB-2 Challenge: Arabic multi-dialect broadcast media recognition,” in SLT, 2016. find our card
A. Ali, P. Nakov, P. Bell, and S. Renals, “WERd: Using social text spelling variants for evaluating dialectal speech recognition,” in ASRU, 2017. find
A. Ali, W. Magdy, P. Bell, and S. Renals, “Multi-reference WER for evaluating ASR for languages with no orthographic rules,” in ASRU, 2015. find
S. Malmasi, M. Zampieri, N. Ljubešic, P. Nakov, A. Ali, and Jörg Tiedemann, “Discriminating between similar languages and Arabic dialect identification: A report on the third DSL shared task,” VarDial 3, p. 1, 2016. find our card
M. Zampieri, S. Malmasi, N. Ljubešic, P. Nakov, A. Ali, J. Tiedemann, Y. Scherrer, and N. Aepli, “Findings of the VarDial evaluation campaign 2017,” VarDial 2017, p. 1, 2017. find our card
S. Khurana and A. Ali, “QCRI Advanced Transcription System (QATS) for the Arabic multi-dialect broadcast media recognition: MGB-2 Challenge,” in SLT, 2016. find
A. Ali, N. Dehak, P. Cardinal, S. Khurana, S. Harsha Yella, J. Glass, P. Bell, and S. Renals, “Automatic dialect detection in Arabic broadcast speech,” in Interspeech, 2016. find
S. Wray and A. Ali, “Crowdsource a little to label a lot: Labeling a speech corpus of Dialectal Arabic,” in Interspeech, 2015. find
A. Ali, Y. Zhang, and S. Vogel, “QCRI advanced transcription system (QATS),” in SLT, 2014. find
T. Ko, V. Peddinti, D. Povey, and S. Khudanpur, “Audio augmentation for speech recognition.,” in Interspeech, 2015. find
D. Povey, V. Peddinti, D. Galvez, P. Ghahrmani, Vimal M., X. Na, Y. Wang, and S. Khudanpur, “Purely sequence-trained neural networks for ASR based on lattice-free MMI,” Interspeech, 2016. find
P. Smit, S. Gangireddy, S. Enarvi, S. Virpioja, and M. Kurimo, “Aalto system for the 2017 Arabic multi-genre brodcast challenge,” in ASRU, 2017. find
K. Yang, W. Zhang, D. Qu, and W. Zhang, “The NDSC-THUEE transcription system for the 2017 multi-genre broadcast challenge,” in ASRU, 2017. find
V. Manohar, D. Povey, and S. Khudanpur, “JHU Kaldi system for Arabic MGB-3 ASR challenge using diarization, audio-transcript alignment and transfer learning,” in ASRU, 2017. find
D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz, et al., “The Kaldi speech recognition toolkit,” in ASRU, 2011. find
M. Najafian, W. Hsu, A. Ali, and J. Glass, “Automatic speech recognition of Arabic multi-genre broadcast media,” in ASRU, 2017. find
K. Vesely, B. Karthick, M. Diez, and K. Benes, “MGB-3 BUT system: Low-resource ASR on Egyptian YouTube data,” in ASRU, 2017. find
S. Shon, A. Ali, and J. Glass, “MIT-QCRI Arabic dialect identification system for the 2017 multi-genre broadcast challenge,” in ASRU, 2017. find
A. Bulut, Q. Zhang, C. Zhang, F. Bahmaninezhad, and J. Hansen, “UTD-CRSS submission for MGB-3 Arabic dialect identification: Front-end and back-end advancements on broadcast speech,” in ASRU, 2017. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/ali-2017-speech-recognition-challenge-wild-arabic/

📄 Speech Recognition Challenge in the Wild: Arabic MGB-3