📄 Country-level Arabic Dialect Identification Using Small Datasets with Integrated Machine Learning Techniques and Deep Learning Models

Summary

The paper tackles country-level Arabic dialect identification for the NADI 2021 shared task (subtask 1.2), where each tweet must be assigned to one of 21 countries using only a modest labelled set. The author compares classical learners (SVM, Logistic Regression, Random Forest), a stacked LR→SVM setup, a hard-voting ensemble, and two neural models (GRU, BiLSTM). Inputs include character and word n-grams, TFIDF, and features derived from Pointwise Mutual Information. PMI is used to build per-dialect word lists automatically, with MSA vocabulary filtered out, and to score each tweet against every dialect. Two ways of using the roughly 10M unlabelled tweets are also tested: co-training with three agreeing classifiers, and word2vec embeddings. The best system is an SVM with n-grams, dictionary flags and tweet-level PMI scores, reaching 18.94% macro-F1 on the test set. The stacked model has the highest precision (22.70%), and the neural models lag well behind. The work shows that cheap, interpretable lexical statistics outperform deep models when labelled data is scarce and unevenly distributed across classes.

Key points

• Data: the NADI corpus contains 31,000 labelled tweets in 21 country dialects, split into 21,000 for training, 5,000 for development and 5,000 for test. It comes with about 10M unlabelled tweets, and six dialects have fewer than 250 training examples.
• Dictionaries: a word is added to a dialect's list when its PMI with that dialect exceeds 1.5. Words found in the subtask 1.1 MSA corpus are excluded, and each tweet also receives 21 summed PMI values, one per dialect.
• Results: the SVM with c2-5, w1-3, dict and PMI features scored 18.71 macro-F1 on dev and 18.94 on test. The integrated LR→SVM model scored about 18.1 F1 on test with higher accuracy (38.47), while BiLSTM and GRU with learned embeddings reached only about 13 F1 on dev.
• Co-training: the unanimous agreement of LR, SVM and RF was used to label 300K unlabelled tweets, producing the CoTraining labelled corpus. Training on this corpus (BiLSTM 12.63 F1) performed about the same as using embeddings pre-trained on it (12.85 F1).
• Error analysis: many errors were short tweets with no distinctive dialect words. Class imbalance also distorted PMI: a word shared by Egypt and Yemen received a high score only for Yemen, because Egypt had 4283 samples and Yemen 429.

Interesting ideas

• I1. Class-specific lexicons are built by keeping only words whose PMI passes a threshold, then removing words that also appear in a generic reference corpus (MSA), so each list keeps only the truly marked vocabulary. This is a cheap, inspectable alternative to learned cue detectors.
• I2. An entire input is condensed into one summed association score per class, giving a very small, readable feature vector that sits alongside sparse n-grams. Each number can be traced back to the individual words that produced it.
• I3. In the stacking design, one model's predictions become extra features for a second model, which raised precision while costing some recall. This is a simple knob for systems where a false positive is worse than a miss.
• I4. Automatic labelling by unanimous agreement among three different classifiers was directly compared with simply pre-training embeddings on the same text, and the two came out level. This shows that agreement-based silver data should be benchmarked against a cheap alternative.
• I5. The error analysis shows that association statistics computed on imbalanced sources favour the smaller class for shared words. This is a general warning for any corpus-derived lexicon.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): when inducing repair rules from silver data, cue words could be chosen by PMI with the error class and then filtered against a neutral corpus. This would keep each rule's trigger explainable.
• I2 (frames): lemma-to-speech-act or lemma-to-irony PMI sums could act as transparent prior signals for prag. Each score could be justified by listing its contributing words.
• I3 (tree): gparse's predicted heads and labels could be fed as features into the perceptron parser. The final, explainable parser could then decide when to trust the transformer.
• I4 (learn): witness-parser silver data could be evaluated against a baseline that uses the same raw text without agreement. This would act as a negative control showing what the agreement filter actually contributes.
• I5 (world): when compiling verb classes or value lexicons from uneven corpora, scores could be smoothed or class-balanced. This would stop sparsely covered sources from claiming shared words.