📄 Arabic Dialects Identification for All Arabic countries

Summary

The paper addresses country-level Arabic dialect identification for Twitter text, covering all 21 Arab countries. This is subtask 1 of the NADI 2020 shared task. The NADI tweets are noisy: they include Persian and Kurdish posts, as well as MSA and Classical Arabic mixed in with dialect. The authors therefore build a five-stage pipeline:

1. Normalization.
2. Noise filtering, using a Naïve Bayes foreign-language detector and a semi-supervised k-means step that separates dialectal from non-dialectal tweets.
3. Removal of standard-Arabic clitics and stop words, keeping dialect-specific ones.
4. Features: TF-IDF words plus selected prefixes and suffixes.
5. Classification by a vote of Naïve Bayes, Logistic Regression and Decision Tree.

Voting is applied either as one 21-way classifier or as 21 one-vs-rest detectors. Ambiguous cases go to a specialist model trained only on the candidate countries. On the development set, the best setup (binary detectors plus MSA-cleaned training data) reaches a macro F-measure of 52.38. The test scores are much lower, which highlights how hard fine-grained dialect identification is when labels and data are noisy.

Key points

• Data: the NADI Twitter set has 21,000 training, 4,957 development and 5,000 test tweets. Each tweet is labeled by country (21 classes) and by province (100 classes). Roughly 504 training tweets were found to be non-Arabic.
• Foreign-tweet detector: Naïve Bayes using word unigrams and character bigrams. On 132 development foreign tweets it scored P 0.985, R 1.0, F 0.992.
• Dev F-measure: 27.17 for voting without clustering (the official submission), 41.34 with clustering, and 52.38 for 21 binary detectors with clustering. The matching test scores are 12.45, 17.61 and 20.05.
• Forced labels for evaluation: foreign tweets were labeled Iraq, MSA tweets Egypt, and tweets rejected by all detectors Egypt. The authors note this lowers macro scores.
• Error sources identified by the authors:
  • Labels reflect user location rather than actual dialect.
  • Neighbouring dialects are close to each other.
  • Some cities speak like another country (e.g., Basrah sounds closer to Kuwait).
  • Some tweets look like MSA but could be read in many dialects.

Interesting ideas

• I1. About 2,000 hand-labeled tweets serve as fixed anchors in k-means, so cluster membership spreads to unlabeled data while the seeds never move; this cleans a noisy training set cheaply.
• I2. A cascade first runs one-vs-rest detectors; only when several fire does a specialist trained solely on those candidates decide, and when none fire the input is explicitly "unknown".
• I3. Normalization strips standard-language clitics and stop words but deliberately keeps dialect-specific clitics and function words, because those carry the signal.
• I4. The authors make the cost of forcing out-of-scope inputs (foreign, MSA) into real classes visible, noting it drags down macro scores.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Clamped-seed clustering could filter silver parses or frames before training, with a small trusted gold core anchoring which disputed examples to drop.
• I2 (tokens): The tagger could resolve hard ambiguities by calling a small model trained only on the competing tags, keeping each decision local and explainable.
• I3 (tokens): Tokenizer and lemmatizer normalization should keep register and dialect markers (contractions, nonstandard forms) as features instead of erasing them before parsing.
• I4 (learn): Evaluations should keep an explicit "out of scope / no answer" bucket and report it separately, similar to a negative control, so forced guesses do not hide real accuracy.