📄 ST MADAR 2019 Shared Task: Arabic Fine-Grained Dialect Identification

Summary

The paper presents the authors' entry in the MADAR 2019 shared task on fine-grained Arabic dialect identification. Each short sentence must be assigned to one of 26 classes: 25 city-level dialects plus MSA. The data is the MADAR Travel Domain corpus, built by having native speakers translate BTEC travel phrases into each variety. The authors used deliberately simple machinery. Preprocessing was minimal, consisting of whitespace tokenization and punctuation removal. Features were word n-grams, character n-grams and within-word character n-grams (char_wb), weighted with TF-IDF. These were fed to a range of scikit-learn classifiers: SVM, Naive Bayes variants, LR, SGD, Passive Aggressive and Perceptron. Multinomial Naive Bayes on word unigrams performed best, at about 62.9% macro-F1 on dev and 62.1% on test. This falls short of the official baseline (F1 69.00). The same setup reaches about 90.5% on a coarser 6-way version of the data, which shows how much harder the city-level distinction is. The authors attribute the difficulty to closely related dialects, non-standard spelling and the presence of MSA material.

Key points

• The data contains 1,600 training sentences per variety and 200 each for dev and test, giving 41,600 / 5,200 / 5,200 sentences in total. Sentences are 4–15 words long.
• Classifiers and dev macro-F1 on Corpus-26:
  • MNB: 62.87
  • BNB: 62.72
  • linear SVM (SVM l): 61.94
  • Perceptron: weakest, at 55.89
• On test, MNB scored F1 62.12 and accuracy 62.17, below the baseline's 69.00 F1 and 67.90 accuracy.
• Preliminary experiments with POS-tagging and lemmatization features produced no significant improvement over surface n-grams.
• Results varied widely by dialect and region:
  • Mosul (MOS) was easiest, with F1 of 80 on dev and 78 on test.
  • AMM and MUS were hardest, at roughly 48–51 F1.
  • By region, Yemen averaged about 75% and the Maghreb over 67%.

Interesting ideas

• I1. The classification task was run on the same data at two levels of granularity (6-way and 26-way), turning the gap between them into a direct measure of how confusable the fine labels are.
• I2. Character n-grams were restricted to word interiors with space padding at the edges (char_wb), which captures prefixes and suffixes while tolerating unstable spelling.
• I3. The authors noted that richer linguistic features (POS, lemmas) failed to beat plain surface n-grams, an ablation worth reporting rather than hiding.
• I4. Results were broken down per class and per regional group, exposing which neighbouring varieties collapse into each other.
• I5. The corpus is parallel: the same content is expressed in every variety, so the classifier must rely on variety signal rather than topic.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Scoring evaluation at both coarse and fine label sets (e.g., UPOS vs. full feature bundles, frame vs. VerbNet subclass) would show whether errors are true confusions or near-misses.
• I2 (tokens): Word-internal padded char n-grams could serve as backoff features in the tagger and lemmatizer for misspelled, informal or rare English words.
• I3 (learn): Every added linguistic layer should be justified by an ablation against a surface-only control, so that complexity is kept only where it measurably pays off.
• I4 (tree): Breaking down LAS by source treebank or converted UD dialect would reveal which conversions or genres are systematically confused by the perceptron parser and gparse.
• I5 (tree): A small parallel set of identical sentences in different English registers or treebank conventions could test whether UD dialect conversion yields matching trees.