📄 Simple But Not Naïve: Fine-Grained Arabic Dialect Identification Using Only N-Grams

Summary

The paper tackles sentence-level Arabic dialect identification at city granularity. The task is hard because neighbouring dialects share script, much vocabulary and grammar, and cities in the same country can be nearly indistinguishable. Working within the MADAR shared task (25 city dialects plus MSA, 26 classes), the Qatar University team deliberately limited themselves to lexical surface features: word unigrams/bigrams and character 2–5-grams. Preprocessing was whitespace tokenization plus removal of stop words and punctuation. They compared several standard learners, and only the two Naïve Bayes variants clearly stood out. Their submissions were:

• a Multinomial NB,
• a probability-based vote between Multinomial and Bernoulli NB,
• a Multinomial NB trained on a Random Forest–pruned feature set.

The voting run scored best, at about 64.6% accuracy on the held-out test set, with per-dialect F1 between 0.52 and 0.84. The work shows that a transparent, inexpensive n-gram baseline gets close to richer systems. The error analysis shows that most mistakes stay inside a country or region.

Key points

• Data: MADAR Corpus-26, which consists of BTEC travel sentences rendered in 25 city dialects and MSA, with 1,600/200/200 sentences per dialect for train/dev/test. Corpus-6 was not used. Tools: pyarabic, NLTK and scikit-learn.

• Classifier screening was done on word unigrams only (dev accuracy):

  • Bernoulli NB: 63.37%
  • Multinomial NB: 63.21%
  • SGD: 52.79%
  • Gaussian NB: 49.37%
  • Perceptron variants: about 47%

• The best feature mix was word unigrams combined with character 4- and 5-grams, giving 66.25% dev accuracy with MNB.

• Official test results:

  • Voting: 64.58% accuracy, F1 0.6445
  • MNB: 64.40%
  • MNB-FS: 62.62%

  Random Forest feature selection reduced the feature count from 228,585 to 203,200 (about 11%) and training time from 1:42h to 1:31h. On the dev set this cost only about 0.6 accuracy points.

• Error patterns:

  • The largest confusion was Tunis↔Sfax (49 and 29 sentences).
  • Cairo had the weakest F1, 0.53, and was often mislabeled as Aswan or Alexandria.
  • By region, Maghrebi dialects scored relatively well, Levantine weaker, and Gulf worst.

Interesting ideas

• I1. Before exploring feature combinations, they ran every candidate learner on one minimal feature set and kept only the clear winners, which shrinks the search space at low cost.
• I2. Character 4–5-grams served as a stand-in for affixes, picking up morphology that whole words miss.
• I3. Two related classifiers were combined by their probabilities rather than hard labels, which gained a little because the models were confident in different places.
• I4. Errors were laid out in a confusion matrix ordered by country and region, which showed that mistakes cluster within related varieties.
• I5. They explained a specific misclassification by pointing to one word's skewed frequency across two classes in the training data (41 vs 14 occurrences).

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Our evaluation harness could screen new parser or tagger variants on a stripped-down baseline input first, then spend full training runs only on the configurations that survive.
• I2 (tokens): Suffix/prefix character n-gram features could help the tagger and lemmatizer guess the POS and lemma of out-of-vocabulary or nonstandard English words, without using them as a bag of words for meaning.
• I3 (learn): When several witness parsers produce silver data, weighting agreement by each parser's confidence could surface better trees than a plain majority vote.
• I4 (tree): Parser errors across UD treebanks or converted dialects could be grouped by genre or variety family, to tell genuine closeness of varieties apart from real parser bugs.
• I5 (learn): Self-induced repair rules could start from the same kind of diagnosis, tracing a wrong decision to the training counts of the lexical item behind it, which keeps every fix explainable.