📄 LIUM-MIRACL Participation in the MADAR Arabic Dialect Identification Shared Task

Summary

The paper reports the LIUM and MIRACL team's entry in subtask 1 (Travel Domain Dialect Identification) of the MADAR shared task at WANLP 2019. The task is to assign one of 26 labels to a sentence: 25 city-level Arabic dialects plus MSA. City-level labels make the problem much harder than the usual five regional groups, because neighbouring varieties share most of their vocabulary. The authors used only the provided corpus-26 data, which consists of BTEC travel sentences translated into each dialect. They compared several approaches:

• Multinomial Naive Bayes over TF-IDF word and character n-grams
• choosing the dialect whose KenLM language model gives the lowest perplexity
• MNB with LM scores added as extra features
• deep learning models, which did poorly

The best system was plain MNB on unnormalized text, reaching an F1 of 65.41 on the official test set. A native-speaker review of the Tunis/Sfax confusions found that most misclassified sentences could plausibly belong to either city. This suggests the task is close to its natural ceiling for very similar dialects.

Key points

• Data: Corpus-26-train has about 41.6k lines and 343.7k tokens. Dev and test each contain 5.2k sentences, with 200 per class. Corpus-6 was available but not used for the submission.
• Best configuration: MNB with word unigrams plus character 1→5-grams scored 66.41 F1 on dev and 65.41 on test. This used raw data. With character normalization and removal of vowels, tatweel and punctuation, the same setup scored only 60.97 and 59.21.
• LM classifiers: 5-gram KenLM models with Kneser–Ney smoothing came in word/character and forward/backward variants. They stayed below MNB, and the best was the backward character LM at 60.63 test F1. Feeding the 26 LM scores into MNB lowered results in every combination tried.
• Per-dialect results: F1 ranged from 0.84 (MOS) down to 0.53 (CAI). Easier dialects tended to be those with high token dissimilarity from the others, such as ALG, SAN and MOS.
• TUN vs. SFX confusion:
  • Of 200 SFX test sentences, 153 were correct and 18 were labelled TUN.
  • Of 200 TUN test sentences, 123 were correct and 41 were labelled SFX.
  • A native speaker judged nearly all of these 59 errors to be genuinely ambiguous.

Interesting ideas

• I1. Cleaning the text hurt accuracy, and the authors suspect the raw data let the classifier pick up each translator's personal spelling and punctuation habits, so "noise" can be a leaked annotator signal rather than real linguistic evidence.
• I2. The team asked a native speaker whether the misclassified sentences were truly distinguishable, which turned an error count into an estimate of the achievable upper bound.
• I3. Each class gets its own LM, a sentence is assigned to the model with the lowest perplexity, and a character model trained on reversed strings performed best among these, presumably because it focuses on word endings.
• I4. A simple Naive Bayes model on mixed word and character n-grams beat deep models when training data was small and labels were fine-grained.
• I5. The authors checked that per-class difficulty matched a measurable property, pairwise lexical overlap, so the confusion pattern could be predicted from the data itself.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Run every evaluation with and without normalization as a negative control. A large gap would warn that a model may be fitting treebank or annotator style instead of syntax.
• I2 (learn): Disagreements among witness parsers could go through the same check by a human or the LLM teacher, separating real ambiguity from model mistakes so neither is trained against.
• I3 (tokens): Small reversed character LMs could score suffix patterns to guess the POS and lemma of unknown English words in a transparent, inspectable way.
• I4 (tokens): Cheap character n-gram features could serve as an explainable fallback in the tagger for rare forms before a heavier GPU model is called.
• I5 (learn): Relating per-relation or per-construction errors to overlap statistics in the treebank could flag which confusions are expected and which point to real bugs.