📄 Hierarchical Deep Learning for Arabic Dialect Identification

Summary

This workshop paper addresses fine-grained Arabic dialect identification in the MADAR shared task. Each sentence must be assigned to one of 26 classes: the dialects of 25 Arab cities plus Modern Standard Arabic (MSA). The authors compare two systems.

The first is a two-level neural cascade. A recurrent network first predicts one of 8 broad regional groups, formed from the cities' geography and dialect similarity. A separate small recurrent model for that group then selects the exact city. Its input is word2vec embeddings trained on a large tweet collection, harvested using dialect-specific keywords from MADAR.

The second is a hard-voting ensemble of Multinomial Naive Bayes and a Random Forest, both working on TF-IDF word features.

On their own held-out split, the hierarchy only slightly beat a flat neural network, while the simpler voting ensemble was clearly best. That ensemble reached 63.02% F1 on the official subtask evaluation.

The work matters as a practical reminder. With roughly two thousand sentences per class, count-based statistical models can beat deep networks. Closely related neighbouring dialects remain the main source of errors.

Key points

• Data. MADAR corpus: 2,000 parallel sentences per dialect, 52,000 in total. The authors split it per dialect into 80% train, 10% dev and 10% test. Sequences were capped at 40 words, and non-Arabic characters were removed.
• Embeddings. word2vec was trained on more than 32 million tweets. These were fetched using the highest-TF-IDF dialect words from MADAR, so the vectors lean toward dialectal usage.
• Hierarchy.
  • Level 1: a B-LSTM with 128 units, a dense layer of 64 and an 8-way softmax.
  • Level 2: seven group-specific models, each using B-LSTM or B-GRU with 32–64 units and its own layer sizes.
  • Training: RMSProp and Adamax, cross entropy loss, batch size 100.
• Internal macro F1. Flat DNN 0.56, hierarchical DNN 0.58, voting classifier 0.64.
  • The group router averaged 0.78 F1.
  • The Levantine group (G7: Jerusalem, Amman, Salt, Beirut, Damascus, Aleppo) was the weakest second-level model, at only 0.55 F1.
• Best ensemble settings. Naive Bayes with alpha 0.3, and a Random Forest with 250 estimators and depth 200. Tuning the forest added about 4 F1 points.

Interesting ideas

• I1. Split a many-way decision into coarse-to-fine stages: a router picks a family of labels, then a specialist chooses within it, which shrinks each model's label space when data is scarce.
• I2. Give each specialist a capacity matched to how many classes and how much training data its subset has, instead of one fixed size for all.
• I3. Grow an unlabeled corpus by mining the small labeled set for its most distinctive words (by TF-IDF) and using them as retrieval keys, so pretraining text is tilted toward the target variety.
• I4. Report accuracy separately for the router and for each second-level model, which exposes exactly which branch (here the Levantine group) drags down the whole cascade.
• I5. Probe the system with near-identical sentence pairs from neighbouring classes (the Tunis vs. Sfax example) to show where its distinctions break down.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): The tagger could first commit to a coarse UPOS and then call small per-category models for XPOS and morphological features, so each feature decision can be explained by its parent choice.
• I2 (learn): When training per-relation or per-construction components of gparse or the perceptron parser, size them to their silver-data volume to avoid overfitting rare branches.
• I3 (learn): Distinctive words from a domain such as fables or school tests could seed the collection of raw text for witness parsers, yielding domain-matched silver data.
• I4 (learn): Every staged step (tokens → tree → frames) should be scored on its own, so errors that pass from one stage to the next can be located rather than hidden in one end-to-end number.
• I5 (learn): Minimal pairs differing in one word could serve as negative controls in evaluation, checking that a parse or frame change is triggered by the real difference.