📄 Adapting MARBERT for Improved Arabic Dialect Identification: Submission to the NADI 2021 Shared Task

Summary

The paper addresses fine-grained Arabic dialect identification for the NADI 2021 shared task. Systems must guess where a short tweet comes from, at both country and province level, for dialectal Arabic (DA) and Modern Standard Arabic (MSA). That makes four subtasks, covering 21 countries and 100 provinces. The authors start from MARBERT, a BERT-base model pretrained on a billion Arabic tweets, and compare two training approaches. One fine-tunes the whole network; the other trains only small Houlsby-style adapter layers. They also add Vertical Attention, a module that lets the classifier draw on the CLS representation from every layer rather than only the top one. Their final system combines several of these variants by multiplying their softmax outputs. It reaches a dev macro-F1 of 34.03 on country-level DA, well above the earlier MARBERT baseline, and the authors report state-of-the-art results on all four subtasks. Error analysis shows two patterns: neighbouring countries are often confused, and short tweets are much harder to classify. This makes the work a useful reference for classifying closely related language varieties on noisy, imbalanced data.

Key points

• The data has two tweet datasets (DA and MSA), each with about 21k training, 5k dev and 5k test examples drawn from separate users. The labels are heavily imbalanced: Egypt has 4283 examples and Somalia only 172.
• The base model is MARBERT (163M parameters, WordPiece). The classification head uses a much higher learning rate (1e-2) than the encoder (5e-6), with warmup, inverse-square-root decay, AdamW, and early stopping on dev macro-F1.
• Due to limited compute, all hyperparameters were tuned only on the Subtask 1.2 dev set and then reused unchanged for the other three subtasks.
• On Subtask 1.2:
  • The best single model (full fine-tuning plus VAtt) scored 32.90 dev F1.
  • The ensemble reached 34.03 dev and 32.26 test F1, a gain of about 1.1 points over the best single model.
• Province-level DA remains very hard, with roughly 10 dev F1 and 8.6 test F1. Correctly classified tweets were noticeably longer, with an Erlang-fit median of about 5.5 tokens versus 3.4 for errors.

Interesting ideas

• I1. Vertical Attention learns which layer depth to read: the top CLS vector serves as the query, and keys come from learned level embeddings over the per-layer CLS vectors.
• I2. The ensemble multiplies the softmax distributions of the variants (product of experts), so a label survives only if every model gives it reasonable probability.
• I3. The authors split prediction errors by input length and fit a distribution to the lengths, showing that short inputs carry too little signal.
• I4. Adapters keep a broadly pretrained model frozen and store only small task-specific layers, which preserves the general knowledge learned in pretraining.
• I5. The confusion matrix revealed geographically structured errors, meaning the mistakes follow real similarity between classes rather than random noise.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): gparse could attend across its transformer layers rather than using only the last one, letting arc and label scorers pick the abstraction level they need, with the attention weights inspectable for explainability.
• I2 (learn): witness parsers could be combined multiplicatively when producing silver data, so a head or label is accepted only if no witness strongly rejects it.
• I3 (learn): evaluation reports could break down accuracy by sentence length, and very short inputs could be treated as low-confidence instead of being trusted blindly.
• I4 (tree): each UD dialect or treebank conversion could get its own lightweight adapter on top of a shared gparse encoder, avoiding interference between treebanks.
• I5 (world): when Mova grounds place or origin entities, near-miss confusions could be scored against geographic proximity from Wikidata, separating plausible errors from absurd ones.