📄 Arabic Dialect Identification with Deep Learning and Hybrid Frequency Based Features

Summary

The paper addresses fine-grained Arabic dialect identification: assigning a sentence to one of 25 city dialects or Modern Standard Arabic (MSA). This is harder than country-level classification because neighbouring cities often write almost identically. The authors work with the MADAR parallel corpora, mainly CORPUS-26 (1,600 training samples per class), plus the larger five-dialect CORPUS-6.

Preprocessing strips punctuation, diacritics and letter elongation. The training data is enlarged with simple text perturbations. The authors first tune a Multinomial Naive Bayes baseline on TF-IDF character and word features. They then test several neural architectures: character CNNs with LSTMs, fastText-based LSTMs, a classifier on top of BERT, AraVec embeddings, and character language models.

The strong frequency-based baseline proved very hard to beat. The best system feeds the same TF-IDF vectors into a small feed-forward network and averages its output with the Naive Bayes model. It reached 65.66 macro F1 on the test set, under two points behind the shared-task winner.

The work matters because it shows where the remaining errors come from: close sub-regional dialects, and out-of-vocabulary dialect words that pretrained embeddings handle poorly.

Key points

• Data: MADAR CORPUS-26 has 2,000 parallel sentences in 25 city dialects plus MSA. CORPUS-6 adds 10,000 sentences for five dialects. No external data was used.

• Baseline: The tuned MNB baseline uses TF-IDF character 1–5-grams plus word unigrams, chosen by grid search in scikit-learn. It scored 64.94% dev F1.

• Augmentation: Four tricks were used:

  • removing duplicate words;
  • randomly masking up to 20% of words;
  • randomly swapping up to 20% of words;
  • concatenating two short documents.

  For non-neural models, augmentation roughly quadrupled the data and added about 1% accuracy.

• Submissions:

  • The TF-IDF + NN + MNB ensemble reached 65.66% test F1, ranking 7th of 19. The winner scored 67.32%.
  • The CharCNN/LSTM + fastText LSTM + MNB ensemble reached 65.36%.
  • The BERT + document pooling system reached only 35.14%.

• Diagnostics: Classifying by region instead of city on CORPUS-26 gives at least 80% dev accuracy. Out-of-vocabulary words made up 10–20% of tokens, and they are exactly the dialect-specific ones.

Interesting ideas

• I1. Neural layer on classic features. Instead of replacing a strong sparse n-gram model, the authors train a small dense network on the very same TF-IDF vectors and average its log-probabilities with the original model, gaining a little over either alone.
• I2. Random augmentation per epoch. For each training sample in each epoch, the system picks at random either one augmentation or the untouched original, which curbed overfitting on a small corpus.
• I3. Diagnosis by collapsing labels. Re-running the task with coarse region labels instead of cities showed that most errors come from near-identical sibling classes, not from lack of data.
• I4. Unknown words carry the signal. Averaging or zeroing unknown-word vectors discards precisely the words that identify a dialect, so subword or character-level handling is needed.
• I5. Per-group confusion matrices. Confusion matrices computed within a single region expose which pairs of close dialects (e.g. Cairo vs. Alexandria) are inseparable.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): A lightweight neural scorer over the tagger's existing hand-built feature vectors could be blended with the current model, keeping the interpretable features as the backbone.
• I2 (learn): When training the perceptron or gparse on scarce silver data, sampling light perturbations per epoch could reduce memorisation without inflating the stored dataset.
• I3 (learn): Evaluations could routinely collapse fine labels (e.g. UD relation subtypes, frame roles) into coarser groups to separate genuine confusion between close categories from general weakness.
• I4 (tokens): Out-of-vocabulary words in the morphology and lemma step should get character-based analysis rather than a default fallback, since rare forms often carry the distinguishing meaning.
• I5 (tree): When converting between UD dialects or treebanks, per-pair confusion matrices of labels could pinpoint which conventions the parser cannot tell apart and need explicit conversion rules.