📄 Arabic Dialect Identification Using iVectors and ASR Transcripts

Summary

The paper describes the systems that the MAZA team submitted to VarDial 2017's shared task on Arabic Dialect Identification (ADI). The task was to label Arabic utterances as Modern Standard Arabic (MSA) or one of four dialects: Egyptian, Gulf, Levantine or North-African. It was the first ADI edition to provide both acoustic data and ASR transcripts. The data are Al Jazeera broadcast, debate and discussion recordings from the dataset of Ali et al. (2016). The authors trained one linear SVM per feature type:

• word unigrams,
• character n-grams of length 1–8, taken from raw transcripts with no preprocessing,
• the supplied iVector audio features.

They then combined these classifiers in three ways: plain voting, averaging of Platt-scaled probabilities, and a stacked Random Forest meta-classifier. In cross-validation the audio iVectors were clearly the strongest single feature, and every combination beat every individual classifier. On the test set the meta-classifier was best, reaching about 71.7% accuracy and placing second of six teams. The work shows that mixing speech-derived and text-derived evidence pays off for telling apart closely related varieties.

Key points

• Data: 25,311 train/dev samples were merged for training, and the test set had 1,492 instances. The test majority-class baseline was 23.1%.

• Single features in cross-validation: iVectors scored 85.3%, while the best transcript feature (character 6-grams) scored 76.2%. Character 4-grams and longer outperformed word unigrams.

• Ensembles in cross-validation: voting reached 85.4%, mean probability 95.0% and the meta-classifier 97.7%. In the voting system, 1,165 samples (4.60%) ended in ties that were broken arbitrarily.

• Test results:

  System	Accuracy	Weighted F1
  Run 1 (voting)	0.6086	0.6073
  Run 2 (mean probability)	0.6689	0.6679
  Run 3 (meta-classifier)	0.7165	0.7170

  Teams were ranked by weighted F1.

• Errors by class: MSA was recognised much more reliably than any dialect. Gulf and Levantine were the pair most often confused.

Interesting ideas

• I1. Averaging class probabilities lets a classifier contribute support for the right label even when that label is only its second choice, which makes this combiner more robust than hard voting.
• I2. Stacking trains a second-level model, here a Random Forest, on the base classifiers' outputs via cross-validation, so it learns which witness to trust in which situation.
• I3. With an even number of voters, ties are common (4.6% of training samples here) and were resolved arbitrarily, which silently adds noise.
• I4. Cross-validation accuracy of 97.7% fell to 71.7% on held-out test data, a warning that in-sample stacking estimates can be badly optimistic.
• I5. Character n-grams taken directly from unnormalised text capture sub-word cues such as affixes and syllables without needing a tokenizer.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): when several witness parsers vote on silver arcs, combining their scored alternatives instead of single top picks could recover agreement that hard votes miss.
• I2 (learn): a small explainable meta-model over parser confidences could decide when silver data is trustworthy, while still exposing which parser drove each decision.
• I3 (learn): silver-agreement pipelines should count ties explicitly and route them to an LLM teacher tier rather than breaking them at random.
• I4 (learn): evaluations of combined systems should always include a truly unseen split and negative controls, not just cross-validation on training material.
• I5 (tokens): character-level evidence could back up the tagger and lemmatizer on unknown or noisy words, while staying an inspectable feature.