📄 Comparison between Voting Classifier and Deep Learning methods for Arabic Dialect Identification

Summary

The paper describes the SORBONNE team's entry to the NADI 2020 shared task, subtask 1. The goal is to guess which of 21 Arab countries a tweet's dialect comes from. The data is heavily skewed: Egypt has about 21% of the tweets, while Bahrain and Djibouti have under 1% each. The authors compare three systems:

• a hard-voting ensemble of five linear and Naive Bayes classifiers over TF-IDF word and character n-grams;
• an LSTM network on pre-trained word embeddings (AraVec, plus their own tweet-trained Word2Vec);
• a plain character 5-gram logistic regression baseline with no cleaning.

The ensemble did best: 18.8% macro F1 and 36.54% accuracy on the official test set, ranking 9th. The neural model overfit when given more training data.

The authors also audited the gold labels. A whole Tunisian province subset turned out to be mostly Libyan or Sudanese tweets. Most Lebanese tweets "misclassified" as Egyptian were in fact Egyptian. The lesson is that on small, noisy, imbalanced data, simple diverse ensembles and label checking matter more than model depth.

Key points

• Data: NADI country-level data has 21,000 train, 4,957 dev and 5,000 test tweets across 21 countries. Classes are very unbalanced: Egypt has 5,543 tweets, while Bahrain and Djibouti have roughly 220 each.
• Run1 (best): a scikit-learn FeatureUnion builds three TF-IDF feature sets: word 1–5-grams, character 1–4-grams, and boundary-marked character 1–5-grams. These feed a hard VotingClassifier over SGDClassifier, LinearSVC, Multinomial NB, Bernoulli NB and RidgeClassifier. Trained on train+dev, it reached 18.8 macro F1 and 36.54 accuracy on test.
• Run2 (LSTM): two LSTM layers with dropout, then dense layers, on embeddings.
  • AraVec (100-dim Twitter model) gave 16.18 F1 on test when trained on train only. Adding dev data lowered this to 14.94.
  • Their own 300-dim skip-gram, trained on 10M unlabelled NADI tweets, did worse.
• Run3 (baseline): the MICHAEL method from MADAR 2019, a character 5-gram logistic regression with no preprocessing, scored about 14.6 F1.
• Preprocessing and per-class results:
  • Cleaning removed punctuation, digits, non-Arabic words, repeated characters and emoticons.
  • Stop-words were kept, and diacritics were not stripped, because removing either hurt performance.
  • Bahrain and Djibouti got zero true positives.
  • The manual audit found 212 mislabeled "Mahdia" tweets, and 63 of 70 Lebanon→Egypt confusions were annotation errors.

Interesting ideas

• I1. When a confusion-matrix cell is suspiciously large, inspecting its examples by hand can show that the gold labels are wrong rather than the model, which changes how errors should be counted.
• I2. A cheap ensemble of diverse linear classifiers with majority voting beat a deeper LSTM on small, noisy data, and it kept improving with more data while the LSTM overfit.
• I3. Combining word n-grams with character n-grams, including a variant with explicit word-boundary markers, captures both lexicon and sub-word spelling cues in one feature vector.
• I4. "Standard" cleaning steps such as stop-word removal or normalization can destroy exactly the signal a task needs, so each step should be tested rather than assumed.
• I5. Checking whether adding the dev set to training helps or hurts on test is a quick overfitting probe that separates robust models from brittle ones.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Evaluation could flag dense disagreement clusters between Mova and a gold treebank for manual audit before treating them as parser errors.
• I2 (learn): Silver-data agreement among several simple witness parsers may be more robust than trusting one large model, especially in low-data domains.
• I3 (tokens): The tagger and morphology could add boundary-marked character n-gram features to handle rare words, misspellings and dialectal spellings.
• I4 (tokens): Normalization in the tokenizer should be kept reversible and ablated per task, since function words and spelling variants carry dialect and register information.
• I5 (learn): Training runs for gparse could include this "more data" control as a negative check alongside held-out evaluation.