Elamur Research⌕
☰
home / papers / 2017 / MIT-QCRI Arabic Dialect Identification System for the 2017 Multi-Genre Broadcast Challenge

MIT-QCRI Arabic Dialect Identification System for the 2017 Multi-Genre Broadcast Challenge

Paper · 2017 · Suwon Shon, Ahmed Ali and James Glass

Summary

This paper addresses Arabic Dialect Identification (ADI) in the MGB-3 challenge at ASRU 2017. Each Aljazeera news recording must be labelled as Egyptian, Levantine, Gulf, North African or Modern Standard Arabic. The main difficulty is a domain mismatch: the large training set was recorded under different conditions from the development and test data, and only a small in-domain DEV set is available for adaptation.

The MIT-QCRI team combined acoustic and linguistic evidence. On the audio side they started from the organizers' 400-dimensional i-vectors. A Siamese convolutional network learned a projection that separates dialects. They adapted to the target domain in two ways: by blending dialect models built from training and DEV data with a single weight γ, and by recursive whitening. On the text side they added word, character and phoneme n-gram features derived from ASR transcripts and phone recognizers.

The fused primary submission reached 75% accuracy on the 10-hour test set, the best among participants. The official i-vector baseline scored about 57%. The work shows that a small matched sample can substantially repair a model trained on mismatched data.

Key points

  • Data splits: TRN has 13,825 utterances (53.6 h), DEV has 1,524 (10 h) and TST has 1,492 (10.1 h). There are five classes (EGY, LEV, GLF, NOR, MSA), scored by accuracy, average precision and average recall.
  • Baselines: A multi-class SVM on bottleneck-based 400-dim i-vectors gave 57.3% accuracy, 60.8% precision and 58.0% recall. Lexical bigrams gave only 48.4%, 51.0% and 49.3%.
  • Siamese network: Two weight-sharing CNNs map i-vectors to 200-dim vectors, with no nonlinearity on the final fully connected layer. The loss is the squared gap between the pair label (+1 same dialect, −1 different) and the cosine score.
  • Domain adaptation: Interpolating TRN and DEV dialect means worked best at γ = 0.91 on DEV, while the test-set optimum was 0.83; the team used 0.91. Recursive whitening was applied at 1–3 levels.
  • Linguistic features: Among four phone recognizers, Hungarian was best (47% accuracy) and English worst (33.3%). Character features reached 57.28% on DEV, compared with 48.43% for the word baseline.

Interesting ideas

  • I1. Instead of fusing scores from two systems, build each class prototype as a weighted average of out-of-domain and in-domain examples inside one model, controlled by a single interpretable knob.
  • I2. Train a weight-shared network on same/different pairs so that a cosine score directly reflects class identity, and keep the final projection linear.
  • I3. Recursive whitening recomputes normalization at each round using the data subset that best resembles the target, removing leftover mismatch step by step.
  • I4. Text-derived features are unaffected by recording conditions, and character n-grams beat word n-grams on noisy ASR transcripts, so the two views complement each other.
  • I5. The paper openly reports both the γ chosen on DEV and the different optimum seen on the test set, exposing how much a dev-tuned setting misses.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When adapting to fables or school texts, prototype statistics from large generic treebanks could be blended with a small in-domain gold set, tuning one γ on held-out data.
  • I2 (world): A linear, inspectable projection learned from "same sense / different sense" pairs could score WordNet/FrameNet concept matches while staying explainable.
  • I3 (learn): Confidence features from witness parsers could be re-normalized per target domain in rounds, so silver-data agreement thresholds stay calibrated outside news text.
  • I4 (tokens): Character n-gram backoff could help the tagger and lemmatizer with typos, OOV words or dialectal spellings where word-level cues fail.
  • I5 (learn): Evaluation reports could list the dev-selected hyperparameter next to the oracle test value, as an honesty check alongside negative controls.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Massachusetts Institute of TechnologyCambridge · United States211 works · 69,000 citations🎓Hamad bin Khalifa UniversityDoha · Qatar141 works · 14,175 citations🔬MIT Computer Science and Artificial Intelligence LaboratoryCambridge · United States69 works · 30,953 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Suwon Shon, Ahmed Ali and James Glass. 2017. MIT-QCRI Arabic Dialect Identification System for the 2017 Multi-Genre Broadcast Challenge. arXiv:1709.00387.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1709.00387 · arXiv 1709.00387

@misc{shon2017mit,
  title = {{MIT-QCRI Arabic Dialect Identification System for the 2017 Multi-Genre Broadcast Challenge}},
  author = {Suwon Shon, Ahmed Ali and James Glass},
  year = {2017},
  doi = {10.48550/arxiv.1709.00387},
  eprint = {1709.00387},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1709.00387}
}

References

30 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Najim Dehak, Patrick J Kenny, Reda Dehak, Pierre Dumouchel, and Pierre Ouellet, “Front-End Factor Analysis for Speaker Verification,” IEEE Trans. on Audio, Speech, and Lang. Process., vol. 19, no. 4, pp. 788–798, may 2011. ↗
  2. Fred Richardson, Douglas Reynolds, and Najim Dehak, “A Unified Deep Neural Network for Speaker and Language Recognition,” in Interspeech, 2015, pp. 1146– 1150. find
  3. Yun Lei, Nicolas Scheffer, Luciana Ferrer, and Mitchell McLaren, “A Novel Scheme for Speaker Recognition using a Phonetically-aware Deep Neural Network,” in IEEE ICASSP, 2014, pp. 1714–1718. ↗
  4. David Snyder, Daniel Garcia-Romero, and Daniel Povey, “Time delay deep neural network-based universal background models for speaker recognition,” in IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), 2016, pp. 92–97. find
  5. Sameer Khurana and Ahmed Ali, “QCRI advanced transcription system (QATS) for the Arabic Multi-Dialect Broadcast media recognition: MGB-2 challenge,” in IEEE Workshop on Spoken Language Technology(SLT), 2016, pp. 292–298. find
  6. Stephen Shum, Douglas a. Reynolds, Daniel Garcia-Romero, and Alan McCree, “Unsupervised Clustering Approaches for Domain Adaptation in Speaker Recognition Systems,” in Proceedings of Odyssey - The Speaker and Language Recognition Workshop, 2014, pp. 265–272. find
  7. Md Hafizur Rahman, Ahilan Kanagasundaram, David Dean, and Sridha Sridharan, “Dataset-invariant covariance normalization for out-domain PLDA speaker verification,” in Interspeech, 2015, pp. 1017–1021. find
  8. Elliot Singer and Douglas A. Reynolds, “Domain Mismatch Compensation for Speaker Recognition Using a Library of Whiteners,” IEEE Signal Processing Letters, vol. 22, no. 11, pp. 2000–2003, 2015. ↗
  9. Daniel Garcia-Romero, Xiaohui Zhang, Alan McCree, and Daniel Povey, “Improving Speaker Recognition Performance in the Domain Adaptation Challenge Using Deep Neural Networks,” in IEEE Spoken Language Technology Workshop (SLT), 2014, pp. 378–383. find
  10. Daniel Garcia-Romero and Alan McCree, “Supervised domain adaptation for I-vector based speaker recognition,” in IEEE ICASSP, 2014, pp. 4047–4051. find
  11. Daniel Garcia-Romero, Alan McCree, Stephen Shum, Niko Brummer, and Carlos Vaquero, “Unsupervised Domain Adaptation for I-Vector Speaker Recognition,” in Proceedings of Odyssey - The Speaker and Language Recognition Workshop, 2014, pp. 260–264. find
  12. Ondrej Glembek, Jeff Ma, Pavel Matejka, Bing Zhang, Oldrich Plchot, Lukas Burget, and Spyros Matsoukas, “Domain adaptation via within-class covariance correction in i-vector based speaker recognition systems,” in IEEE ICASSP, 2014, pp. 4060–4064. find
  13. Hagai Aronowitz, “Inter dataset variability compensation for speaker recognition,” in IEEE ICASSP, 2014, pp. 4002–4006. find
  14. Suwon Shon, Seongkyu Mun, Wooil Kim, and Hanseok Ko, “Autoencoder based Domain Adaptation for Speaker Recognition under Insufficient Channel Information,” in Interspeech, 2017, pp. 1014–1018. find
  15. Suwon Shon, Seongkyu Mun, and Hanseok Ko, “Recursive Whitening Transformation for Speaker Recognition on Language Mismatched Condition,” in Interspeech, 2017, pp. 2869–2873. find
  16. Suwon Shon and Hanseok Ko, “KU-ISPL Speaker Recognition Systems under Language mismatch condition for NIST 2016 Speaker Recognition Evaluation,” ArXiv e-prints arXiv:1702.00956, 2017. ↗
  17. Ahilan Kanagasundaram, David Dean, and Sridha Sridharan, “Improving out-domain PLDA speaker verification using unsupervised inter-dataset variability compensation approach,” in IEEE ICASSP, 2015, pp. 4654– 4658. find
  18. Hagai Aronowitz, “Compensating Inter-Dataset Variability in PLDA Hyper-Parameters for Robust Speaker Recognition,” in Proceedings of Odyssey - The Speaker and Language Recognition Workshop, 2014, pp. 280– 286. find
  19. Ahmed Ali, Najim Dehak, Patrick Cardinal, Sameer Khurana, Sree Harsha Yella, James Glass, Peter Bell, and Steve Renals, “Automatic dialect detection in Arabic broadcast speech,” in Interspeech, 2016, vol. 08-12-Sept, pp. 2934–2938.
  20. Abualsoud Hanani, Aziz Qaroush, and West Bank, “Identifying dialects with textual and acoustic cues,” in VarDial, 2017, pp. 93–101.
  21. Radu Tudor Ionescu and Andrei M Butnaru, “Learning to Identify Arabic and German Dialects using Multiple Kernels,” in VarDial, 2017, pp. 200–209.
  22. Taraka Rama, “T ubingen system in VarDial 2017 shared task : experiments with language identification and cross-lingual parsing,” in VarDial, 2017, pp. 146– 155. find
  23. Shervin Malmasi and Marcos Zampieri, “Arabic Dialect Identification Using iVectors and ASR Transcripts,” in VarDial, 2017, number 2015, pp. 178–183.
  24. Ahmed Ali, Stephan Vogel, and Steve Renals, “Speech Recognition Challenge in the Wild: ARABIC MGB-3 ( DRAFT ),” in IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), 2017, p. to be appeared.
  25. Jane Bromley, James W. Bentz, Léon Bottou, Isabelle Guyon, Yann Lecun, Cliff Moore, Eduard Säckinger, and Roopak Shah, “Signature Verification Using a Siamese Time Delay Neural Network,” International Journal of Pattern Recognition and Artificial Intelligence, vol. 07, no. 04, pp. 669–688, 1993. ↗
  26. Stephen Shum, Najim Dehak, Reda Dehak, and James R Glass, “Unsupervised Speaker Adaptation based on the Cosine Similarity for Text-Independent Speaker Verification,” in Proceedings of Odyssey - The Speaker and Language Recognition Workshop, 2010. find
  27. Daniel Garcia-Romero and Carol Y Espy-Wilson, “Analysis of i-vector Length Normalization in Speaker Recognition Systems.,” in Interspeech, 2011, pp. 249– 252. find
  28. Petr Schwarz, Pavel Matejka, and Jan Cernocky, “Hierarchical structures of neural networks for phoneme recognition,” in IEEE ICASSP. IEEE, 2006. find
  29. Ahmed Ali, Peter Bell, James Glass, Yacine Messaoui, Hamdy Mubarak, Steve Renals, and Yifan Zhang, “The mgb-2 challenge: Arabic multi-dialect broadcast media recognition,” in IEEE Spoken Language Technology Workshop (SLT), 2016.
  30. Niko Brümmer and Edward de Villiers, “The BOSARIS Toolkit: Theory, Algorithms and Code for Surviving the New DCF,” in NIST SRE’11 Analysis Workshop, apr 2011. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shon-2017-mit-qcri-arabic-dialect-identification/