📄 Ahmed and Khalil at NADI 2022: Transfer Learning and Addressing Class Imbalance for Arabic Dialect Identification and Sentiment Analysis

Summary

The paper describes a team's system for both subtasks of the NADI 2022 shared task: country-level Arabic dialect identification in tweets (18 classes) and three-way sentiment classification of tweets written in dialects and Modern Standard Arabic. The authors first examine the label distributions. Dialect data are strongly skewed, with the biggest class about twenty times the smallest. Sentiment data are fairly balanced but small. They compare three Arabic BERT-style encoders (MarBERT, CamelBERT, AraBERT) with a simple classification head, along with several loss functions. MarBERT combined with focal loss works best. Tuning the focusing parameter γ shows that a large value gives the highest macro-F1 on dialect identification. Since external labelled data were forbidden, the sentiment model starts from the dialect-identification model and is then fine-tuned on the small sentiment set. The system placed mid-to-upper on both leaderboards. The work shows that cheap loss-level fixes for imbalance and task-to-task transfer can help in low-resource dialect settings.

Key points

• Subtask 1 data: 20,398 training tweets and 4,871 validation tweets over 18 countries. Egypt has 4,283 training samples, while Bahrain, Sudan and Qatar have only 215 each. Subtask 2 has 1,500 training and 500 validation tweets.
• Setup: fine-tuned BERT plus one classification layer (extra dense layers did not help), learning rate 4·10⁻⁵, 10 epochs, Adam with weight decay, and batch size 32 (subtask 1) or 8 (subtask 2), implemented in PyTorch.
• Losses compared: SelfAdjDiceLoss, NLLLoss, CrossEntropyLoss with and without class weights, and FocalLoss. FocalLoss was best on both subtasks.
• γ values tried were 5, 10, 20 and 25. γ = 25 gave the lowest validation loss, but γ = 20 gave the best accuracy and F1. Validation scores peaked at epoch 2, which indicates later overfitting.
• Validation macro-F1 was 0.3305 for subtask 1 and 0.7334 for subtask 2. On test, the results section reports 7th of 19 teams for subtask 1 and 5th of 10 for subtask 2. The abstract states these ranks in the opposite order.

Interesting ideas

• I1. Focal loss down-weights examples the model already gets right, so training concentrates on hard, rare-class cases, and it beat explicit class weighting here.
• I2. When outside labelled data are unavailable, a model trained on a large related task can serve as the starting point for a much smaller target task.
• I3. The checkpoint with the lowest validation loss was not the one with the best F1, so model selection should follow the metric that actually matters.
• I4. Reporting macro-F1, plus a variant that ignores the neutral class, keeps rare or less informative classes from being hidden by the majority.
• I5. The authors suggest, without testing it, domain-adversarial training so that sentiment features become dialect-agnostic and transfer across varieties.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): Rare UD relations such as vocative, dislocated or reparandum could get focal loss in gparse's label classifier, which would lift macro-level LAS on infrequent labels without resampling the treebank.
• I2 (frames): A speech-act or irony classifier in prag could be initialized from the encoder already trained for parsing or tagging, so the scarce pragmatic annotations go further.
• I3 (learn): Training runs should choose checkpoints and hyperparameters by task metrics (LAS, per-label F1) rather than by loss, and log both so these disagreements are visible.
• I4 (learn): Evaluation reports could add per-class macro scores for rare dependency labels or frame roles, so that rare-class regressions surface instead of disappearing into micro averages.
• I5 (tree): A similar adversarial signal could push the parser toward genre- or dialect-invariant representations when training on converted treebanks from different English varieties.