📄 Building an Ensemble of Transformer Models for Arabic Dialect Classification and Sentiment Analysis

Summary

The paper reports the UniManc team's systems for the NADI 2022 shared task. The task had two parts: identifying which of 18 Arab countries a dialectal tweet comes from, and classifying the sentiment of dialectal tweets. Both systems start from MARBERT, a BERT-style model pre-trained on MSA and dialectal tweets.

• Dialect identification: the classes are heavily skewed, with Egypt dominating. The authors therefore built class-balanced copies of the training data by random undersampling, plus duplication-based oversampling for the larger caps. They then fine-tuned models on both the original and the balanced data.
• Sentiment: they first continued MLM pre-training of MARBERT on 10 million unlabelled tweets supplied by the organisers, and only then fine-tuned.

For each subtask they varied seeds and learning rates, kept the strongest variants, and tried every subset of them as a probability-averaging ensemble, judging each subset on the dev set. Both systems placed second overall. Error analysis showed that very short tweets and neighbouring dialects cause most mistakes. From this the authors argue that dialect identification would be better framed as multi-label classification.

Key points

• Subtask 1 balancing: each class was capped at 215 examples, matching the smallest classes (Bahrain, Qatar, Sudan). Extra variants were capped at 250 and 300, with rare classes oversampled by duplication.
• Subtask 2 domain adaptation: MLM continuation used a 90/10 train/validation split of the unlabelled tweets, 8 epochs, batch size 8 and maximum length 512. It helped sentiment but not dialect ID.
• Fine-tuning grid: seeds from 20 to 300 were explored, and 200 worked best for both subtasks. Learning rates between 1.5e-5 and 2.5e-5 were used, with batch size 32 and 8 epochs.
• Ensembles:
  • Subtask 1 combined 8 candidates (4 trained on full data, 4 on balanced data), giving 255 possible subsets.
  • Subtask 2 combined 5 candidates, giving 31 subsets.
  • The top 3 ensembles by dev score were submitted.
• Results:
  • Subtask 1 reached macro-F1 34.778 on Test-A (3rd) and 18.948 on Test-B (1st of 19 teams), for a mean of 26.863 (2nd).
  • Subtask 2 reached macro-F1-PN 73.544 (2nd of 10 teams).
  • Dev tweets of four tokens or fewer (864 samples) scored only 25.180 macro-F1, against 37.385 for longer ones.

Interesting ideas

• I1. Ensembling models trained on the natural label distribution with models trained on rebalanced data combines their strengths: the first handles frequent classes well, the second can actually predict rare ones.
• I2. Treating ensemble design as an exhaustive search over all non-empty subsets of a small pool of candidates, scored on held-out data, is cheap when the pool is small and avoids guessing which members help.
• I3. Bucketing evaluation by input length showed that the overall score hid a sharp drop on very short inputs, where context is too thin to decide.
• I4. Continued MLM pre-training on unlabelled in-domain text paid off for one downstream task but not the other, so domain adaptation should be checked per task rather than assumed.
• I5. When classes genuinely overlap and even humans disagree, forcing a single label creates errors that are not really errors; allowing several labels per item matches the data better.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): gparse could average dependency-label probabilities from a model trained on the plain treebank and one trained on data rebalanced toward rare relations. This could recover rare deprels without hurting frequent ones.
• I2 (learn): the same subset search could pick which witness parsers should vote when building silver data. Negative controls should check that the chosen subset is not simply overfitting the dev set.
• I3 (learn): parser and reader evaluations could report results per sentence-length bucket. This would show whether failures concentrate on fragments and headlines rather than on full sentences.
• I4 (tree): before adopting an encoder further pre-trained on fables or school texts, its parsing LAS should be compared against the base encoder. This should be done separately for each downstream consumer.
• I5 (frames): speech-act, irony or implicature judgments could carry multiple scored labels rather than a single forced choice. Downstream world-building could then keep the ambiguity visible instead of committing to one reading too early.