📄 Mawdoo3 AI at MADAR Shared Task: Arabic Fine-Grained Dialect Identification with Ensemble Learning

Summary

The paper describes Mawdoo3 AI's entry to sub-task 1 of the MADAR shared task (WANLP 2019). The task is fine-grained Arabic dialect identification: each sentence from the travel domain is assigned to one of 26 classes, which are 25 city dialects plus MSA. The team represents each sentence with TF–IDF vectors over word n-grams and character n-grams. On top of these features they stack class probabilities from first-level classifiers and, in one run, scores from 96 KenLM language models. They submitted three systems:

• a logistic regression over the stacked features;
• a hard-voting ensemble of heterogeneous classifiers;
• the same ensemble with LM scores added.

The ensemble scored best on DEV, at about 69.3% macro F1. The LM-augmented ensemble scored 67.20% macro F1 on TEST, which placed third of 19 teams and only 0.12 points behind the winner. Most errors came from confusing dialects that are geographically close. The work shows that careful smoothing, stacking and voting over sparse n-gram features stay competitive for closely related language varieties.

Key points

• Data. CORPUS-26 has 1,600 training and 200 DEV sentences per class. CORPUS-6 covers 5 cities plus MSA. CORPUS-26 contains about 295K space-separated tokens, of which about 85K are unique.
• Features. The combined vector has about 236K dimensions. It joins word unigrams and bigrams with character 1–5-grams that are kept inside word boundaries, which worked better than letting n-grams cross word boundaries.
• Baseline. A Multinomial Naive Bayes with Lidstone smoothing (α = 0.1 rather than 1) reached about 68.3% on DEV. This beat the comparable MNB setup reported by Salameh et al. (2018) on the same features.
• Ensemble. The voting ensemble combines a one-vs-rest MNB, an SVM, Bernoulli NB, a distance-weighted kNN and a prior-based dummy classifier. Each is trained on TF–IDF plus first-level probabilities.
• Language models. There are 96 KenLM language models, at character and word level and in 5-gram and bigram variants. Their sentence scores were rescaled to the 0–1 range so they sit alongside the TF–IDF features.
• Results and errors. On TEST the ensemble with LMs scored 67.20% macro F1, against 67.32% for the winning system, ArbDialectID. The confusion matrix shows Cairo, Alexandria and Aswan being mixed up, and Mosul being confused with MSA.

Interesting ideas

• I1. Stacking with probabilities as features: first-level classifier probabilities are appended to the raw features, so the second layer can learn the systematic biases and error patterns of the first.
• I2. Weak dummy voter as a control: a prior-only classifier sits in the ensemble and gives a floor that any real model must clearly beat.
• I3. Character n-grams kept within words: restricting character n-grams to single words cuts out-of-vocabulary sparsity from rich morphology while avoiding noise from spans that cross word boundaries.
• I4. Smaller smoothing for huge sparse vectors: lowering the additive-smoothing pseudo-count matters when most dimensions are absent from any given input.
• I5. Confusion-matrix error analysis by relatedness: grouping errors by how close the confused classes are shows whether a model fails on near-variants or on genuinely different classes.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): witness parsers' per-arc confidence could be fed as features into an arbiter that learns when each witness tends to be wrong, rather than relying on simple agreement alone.
• I2 (learn): a majority-label or prior-only "witness" could serve as a built-in negative control in evaluations, showing how much each real component actually adds.
• I3 (tokens): word-internal affix n-grams could back off unknown-word tagging and lemma guessing for rare or misspelled English forms.
• I4 (tokens): count-based tagger or morphology statistics could be tuned the same way, checking whether α below 1 reduces noise from unseen features.
• I5 (tree): parser errors on UD dialect-converted treebanks could be broken down by source variety, to locate where conversion rules or the parser blur closely related constructions.