📄 Arabic Dialect Identification and Sentiment Classification using Transformer-based Models

Summary

The authors (team Pythoneers) describe two AraBERT-based systems for the NADI 2022 shared task. The first sub-task is country-level identification of Arabic dialects in tweets. The second is sentiment classification of dialectal tweets. For dialects, they derive two coarser labels from each country: a binary Western/Eastern region and one of four areas. They then train one shared encoder with three classification heads, cycling through the losses one batch at a time. They also oversample so that every country contributes the same number of examples. For sentiment, where training data is tiny, they fine-tune ten AraBERT models in a K-fold scheme and average their logits. That averaged ensemble then serves as a teacher for a single student model. The systems ranked 6th and 9th on the two dialect test sets and 4th on sentiment. Country-level dialect identification remains hard, with macro-F1 in the low thirties. The distilled sentiment model scored better on the test set than on dev, which the authors take as a sign of good generalization.

Key points

• Backbone for both systems: bert-large-arabertv02-twitter, which builds on AraBERTv0.2-large and is pretrained with MLM on 60M multi-dialect tweets. Pre-processing strips non-Arabic words, emojis and URLs.
• Data sizes:
  • Subtask 1 has ~20,398 training tweets across 18 countries and 4,871 dev tweets. TEST-A has 4,758 tweets covering all 18 countries. TEST-B has 1,474 tweets covering an undisclosed subset of k countries.
  • Subtask 2 has 1,500 training tweets, 500 dev tweets and 3,000 test tweets, labelled positive, negative or neutral.
• Multi-task setup:
  • Each head is a dropout layer (0.1) plus a linear layer over the CLS embedding, giving 2, 4 or 18 outputs.
  • Training uses 500 tweets sampled with replacement per country, Adam with lr 1e-5, batch size 64 and 5 epochs.
• Distillation:
  • The loss is (1−α)·CE against gold labels plus α·MSE between student and teacher logits, with α = 0.95 and K = 10.
  • Training uses lr 5e-6, batch size 32 and 6 epochs.
• Macro-F1 results:
  • On dev, region scores 74.64, area 60.65 and country 27.57.
  • Country scores 32.63 on TEST-A and 15.61 on TEST-B.
  • Sentiment on the test set reaches 73.40, scored over positive and negative classes only.
  • Most errors are Levantine vs. Peninsular Gulf, and confusions stay mostly within the same area.

Interesting ideas

• I1. Derive coarser auxiliary labels (region, area) automatically from the fine label and train them as extra heads on one shared encoder, so the label hierarchy acts as free supervision.
• I2. Instead of summing several task losses, update on just one head's loss per batch and rotate between them, sidestepping the problem of weighting losses that measure different things.
• I3. Resample the training set to a fixed count per class, with replacement, so rare classes are seen as often as frequent ones.
• I4. Train K models on K−1 folds each, average their logits into a teacher, then distill into one student. This is most useful when labelled data is very small.
• I5. Read the confusion matrix along the label hierarchy, i.e. whether errors stay inside the same family, to judge whether the hierarchy was actually learned.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): gparse could add auxiliary heads that predict coarse relation families (e.g. core argument vs. modifier vs. function word) derived from the full UD deprel. This would inject the label hierarchy at no annotation cost.
• I2 (learn): when training jointly on tagging, lemmas and parsing, alternating objectives per batch avoids hand-tuning loss weights. That makes each objective's contribution easier to attribute.
• I3 (tokens): rare morphological features or POS tags in the en tagger could be balanced this way during training. The effect should be checked against an unbalanced baseline.
• I4 (learn): silver data agreed by several witness parsers could be turned into soft averaged scores. A single fast student parser would then be distilled from them rather than trained on hard votes alone.
• I5 (learn): evaluation reports could group parser or frame-role errors by family. This would show whether mistakes are near-misses or structural failures, making errors more explainable.