📄 LTG-ST at NADI Shared Task 1: Arabic Dialect Identification using a Stacking Classifier

Summary

The paper tackles country-level Arabic dialect identification for the NADI 2020 shared task (sub-task 1): each tweet must be assigned to one of 21 Arab countries. The author built a deliberately simple scikit-learn system. Three base learners (a logistic regression model, a linear support vector classifier and a multinomial variant of naive Bayes) read weighted TF-IDF features over word bigrams, character n-grams and skip-grams. A second Logistic Regression then combines their cross-validated outputs. The system reaches a macro F1 of 17.71 on test, roughly level with heavier approaches the author also tried, which did no better. Much of the paper is error analysis of the corpus itself. The class distribution is heavily skewed, and most word types are shared across dialects, often because they are MSA words. Some tweets carry a dialect label even though they are written almost entirely in MSA. The author argues that these data properties, more than model choice, limit performance. Filtering out or relabelling MSA content could therefore help future systems.

Key points

• NADI corpus splits: 21,000 / 4,957 / 5,000 tweets (train/dev/test), with 270,574 / 60,700 / 64,458 tokens excluding punctuation.
• Scores: dev macro F1 17.87 with accuracy 37.26; test macro F1 17.71 with accuracy 36.22.
• Feature weights were tuned by grid search and the feature blocks joined with sklearn's Feature Union:
  • word bigrams: 0.5
  • character 2–5-grams bounded by words: 0.5
  • character 3–5-grams: 0.5
  • word and character skip-grams: 0.3
• Class imbalance and vocabulary overlap:
  • Egyptian makes up over 21% of training tweets, while Djibouti, Bahrain, Sudan, Mauritania and Somalia each have about 1%.
  • Only Egyptian and Iraqi have a majority of unique word types (58%); the other 19 dialects share over half their vocabulary.
• Error pattern: errors collapse onto the four largest classes (Egypt, Saudi Arabia, Algeria, Iraq), and Qatar, Bahrain, Djibouti and Kuwait get zero F1. Training on just those four classes yields F1 above 60%. Resampling, BiLSTMs, CNNs and embeddings trained on unlabeled tweets did not help.

Interesting ideas

• I1. Base classifiers are fitted on all features, but the meta-learner sees only their out-of-fold predictions, so it learns how far to trust each one without leakage.
• I2. For each class, the author measures what share of its vocabulary is unique, shared with a few other classes, or shared widely, to explain why some classes are inseparable.
• I3. To test whether imbalance is the cause, the author retrains on only the four best-represented classes and checks whether scores jump.
• I4. Rare classes show decent precision but near-zero recall, which reveals a conservative model rather than a confused one.
• I5. Labels can be noisy because "dialect" tweets are often in the standard variety; filtering or relabelling such items is proposed as a fix.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): When merging several witness parsers into silver data, an arbiter could be trained on held-out witness outputs rather than in-sample ones, to learn per-witness reliability honestly.
• I2 (learn): A similar overlap profile over lexical or construction features per genre or treebank could flag which evaluation slices are inherently hard before blaming the parser.
• I3 (learn): Restricting a task to well-resourced subsets is a cheap diagnostic control and fits our evaluation-with-negative-controls practice for separating data effects from model effects.
• I4 (answer): Reporting per-question-type precision and recall separately would show whether a student abstains too often or answers wrongly, which call for different fixes.
• I5 (tree): During UD dialect conversion and treebank mixing, sentences that are in fact standard-register could be detected and set aside so they do not blur dialect-specific parsing statistics.