📄 iCompass Working Notes for the Nuanced Arabic Dialect Identification Shared task

Summary

This working-notes paper describes the iCompass entry for Subtask 1 of NADI 2022, the shared task on country-level Arabic dialect identification. Given a tweet, a system must assign one of eighteen country labels. The organizers supplied training data of roughly 20k tweets plus a development set of about 4.9k. Class sizes are very uneven, with Egypt and Iraq far ahead of the other countries. The authors cleaned the text by stripping non-Arabic tokens, placeholders and emojis, splitting hashtags, and collapsing extra whitespace. They then compared four set-ups built on the dialect-oriented MARBERT model:

• MARBERT v1 embeddings with a CNN head
• MARBERT v2 embeddings with a CNN head
• MARBERT v2 embeddings with a QRNN head
• plain fine-tuning of MARBERT v2

Plain fine-tuning of MARBERT v2 performed best on the development data and became the single submitted run. It finished 4th on Test-A, 7th on Test-B, and 5th in the overall ranking, a few macro-F1 points below the winner. The paper shows that a dialect-focused pretrained model, fine-tuned directly, beats heavier classifier heads on this data. It also shows how class imbalance pulls macro-F1 down.

Key points

• The data covers 18 country dialects, with 20,398 training and 4,871 development sentences. Egypt and Iraq dominate the label distribution.
• Development results (macro-F1 / accuracy) for each set-up:
  • MARBERT v1 + CNN: 0.12 / 0.39
  • MARBERT v2 + CNN: 0.14 / 0.40
  • MARBERT v2 + QRNN: 0.26 / 0.41
  • fine-tuned MARBERT v2: 0.33 / 0.50
• The submitted model was MARBERT v2 trained for 4 epochs, with learning rate 2e-5, batch size 32 and maximum sequence length 64. Training ran in Google Colaboratory.
• Official results:
  • Test-A (4,758 tweets, 18 known dialects): macro-F1 33.70, accuracy 51.91, rank 4.
  • Test-B (unknown country-level dialects): macro-F1 16.94, rank 7.
  • Average macro-F1: 25.32, rank 5. The winning team, rematchka, scored 27.06.
• Confusion matrices show the classes with the most training data (Egypt, Iraq, KSA) are recognized best. Many Gulf tweets are misassigned to KSA, and Moroccan tweets are often labelled Algerian.

Interesting ideas

• I1. The authors report macro-F1 next to accuracy and read the confusion matrices, which exposes a CNN head that keeps predicting the large classes despite decent accuracy.
• I2. The organizers added a second test set (Test-B) drawn from dialects not covered in training, and scores fell sharply there compared with the in-distribution set.
• I3. The authors compared a light head on top of pretrained embeddings against full fine-tuning, and direct fine-tuning won clearly on a small labelled set.
• I4. During preprocessing, hashtags are split into their component words instead of being dropped, so their content still reaches the model.
• I5. MARBERT's pretraining kept tweets that mix Arabic with foreign words, because dialect text is often code-switched, and the authors plan a model that also handles Latin-script dialect writing such as TUNIZI.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Mova's evaluation could report per-class or macro scores and confusion tables for tagger labels and relation types. This would catch a model that only gets frequent labels right.
• I2 (learn): Mova could keep a held-out out-of-domain test split, such as a different genre or English variety, as a standing check on the silver-trained parsers.
• I3 (tree): gparse experiments could include a fully fine-tuned baseline before adding extra layers on top of the transformer encoder.
• I4 (tokens): the en tokenizer could segment hashtags and camel-cased handles into words, so the tagger and lemmatizer see real tokens.
• I5 (tokens): the en pipeline could flag code-switched or non-standard-script spans explicitly instead of forcing them into English tags, which keeps later steps explainable.