📄 SQU-CS @ NADI 2022: Dialectal Arabic Identification using One-vs-One Classification with TF-IDF Weights Computed on Character n-grams

Summary

The paper tackles country-level Arabic dialect identification on Twitter as part of the NADI 2022 shared task. The task has 18 country labels and a strongly skewed class distribution. The author submitted two systems. The first, OVO-LR, is a deliberately simple baseline: it normalizes the text, represents each tweet with TF-IDF weights over character n-grams of length 2 to 5, and trains a one-vs-one set of binary logistic-regression classifiers with SGD, L2 regularization and class weights balanced against label frequency. The second fine-tunes AraBERT (bert-base-arabertv02-twitter). AraBERT is clearly stronger, but the cheap character-level model stays within a few points of it. The analysis is the most useful part. It breaks down performance per country, shows that rare Gulf dialects are hardest, and traces a complete failure on Bahraini to a topic and pronoun mismatch between TRAIN and DEV. It also finds that many of the most discriminative n-grams are simply country names, a known weakness of bag-of-features models.

Key points

• Data: TRAIN has 20,398 tweets and DEV has 4,871, with Egypt the largest class at 4,283 in TRAIN and Bahrain, Qatar and Sudan the smallest at about 215 each. Unlabeled evaluation sets are TEST-A (4,758) and TEST-B (1,474); TEST-B covers an unknown subset of the countries.
• Preprocessing: Printable ASCII and Arabic diacritics are removed, runs of 3+ identical letters are cut to two, Alif variants are unified to ا, ة is mapped to ه, and ى is mapped to ي.
• OVO-LR implementation: The scikit-learn setup is TfIdfVectorizer(char, 2–5), SGDClassifier(log loss, balanced weights) and OneVsOneClassifier. The TF-IDF vocabulary was built from TRAIN, DEV and TEST text together.
• AraBERT setup: Only the prediction layer was retrained on TRAIN, for 10 epochs with lr 2e-5 and batch size 16. The underlying model was pretrained on 60M tweets.
• Macro-F1 results:

The team's official average of 22.42 ranked 10th of 19 teams. Both systems scored 0 on Bahrain. OVO-LR beat AraBERT on Qatar and Kuwait, which the author credits to the class weighting.

Interesting ideas

• I1. When a class scores zero, compare that class's TRAIN and DEV samples by hand; here, sports and masculine pronouns versus social topics and feminine pronouns explained the collapse better than any model change would have.
• I2. Listing a linear model's top-weighted features exposes shortcut cues, such as country names standing in for dialect, that inflate scores without capturing the real phenomenon.
• I3. Cutting runs of three or more repeated letters down to two removes expressive elongation while still keeping a mark that elongation happened.
• I4. Setting class weights inversely to label frequency gave measurable gains on rare classes even though it lowered results on frequent ones.
• I5. A tuned character-n-gram linear model, cheap and fully inspectable, lands only about 7 points below a fine-tuned transformer, which makes it a useful honest baseline.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Add a per-label shift audit to evaluation, checking genre and pronoun or gender mix between silver training data and test data before blaming the parser or the teacher tier.
• I2 (learn): Inspect the highest-weight features of the perceptron parser and the repair rules for lexical shortcuts, and build negative controls that remove those cues to see whether accuracy holds.
• I3 (tokens): Apply the same normalization to informal English ("soooo", "nooo") before lemma lookup, and keep the original form as a feature that the irony and emphasis components can use.
• I4 (tree): Weight rare UD relations (e.g., vocative, dislocated) more heavily when training the perceptron parser or gparse, so they are not drowned out by nsubj and obj.
• I5 (learn): Keep a simple, explainable baseline alongside gparse in every evaluation, so that any claimed gain from the heavier model is measured against something interpretable.