📄 NLPeople at NADI 2023 Shared Task: Arabic Dialect Identification with Augmented Context and Multi-Stage Tuning

Summary

The paper reports the winning entry for Subtask 1 of the NADI 2023 shared task: assigning a tweet to one of 18 country-level Arabic dialects. Tweets are short, noisy and morphologically rich, so the team stacked several improvements on a BERT-style classifier, which feeds the [CLS] vector into a two-layer feedforward head. First, they compared pretrained encoders and found that Arabic-specific and Twitter-trained ones work best, with MARBERTv2 on top. Second, they enlarged each input by appending its nearest training tweets, together with their gold labels, found through embeddings from the fine-tuned model. Third, they trained in stages: on NADI-2020, then NADI-2021, then the 2023 data, instead of pooling everything. Finally, they combined many models through score-level ensembling. Each component raised macro-F1 on the development set. The final ensemble reached 87.27 macro-F1 on the test set and ranked first of 16 teams. The work shows that careful data ordering and retrieved labelled neighbours can compensate for very short inputs.

Key points

• The NADI-2023 data is balanced: 1000 training and 100 development tweets for each of 18 countries, with 3600 test items. The older NADI-2020 and NADI-2021 sets are imbalanced and contain no UAE label.
• Development macro-F1 by encoder:
  • MARBERTv2: 86.05.
  • arabertv02-twitter: 80.61, versus 76.62 for plain arabertv02.
  • AraT5-tweet-base, generating the label: 78.53, versus 75.67 for AraT5-base.
  • Multilingual XLM-R: only 68.44.
• Retrieval used Euclidean k-NN over fine-tuned MARBERTv2 vectors, with neighbour labels added as special tokens. F1 rose from 86.05 to 86.58 with one neighbour, 86.71 with five and 86.79 with ten.
• Data sources:
  • Simply adding the 2020 or 2021 data to the 2023 data lowered F1 to about 83.0–83.5.
  • The three-stage schedule (2020 → 2021 → 2023) raised it to 87.02.
• Ensembling:
  • With 10 models, score ensembling reached 86.88 and majority voting reached 86.74.
  • The submitted system came from a random search over model subsets. It used 15 MARBERTv2 and arabertv02-twitter models and scored 89.56 on development.
  • Surprisingly, no retrieval-augmented models ended up in this ensemble.
  • On development, Morocco was correct for 99 of 100 tweets and UAE stayed below 80%. The biggest confusion was Kuwait labelled as Bahrain, at 12%.

Interesting ideas

• I1. Enrich a short input by attaching its nearest labelled training examples, labels included, so the classifier sees concrete precedents alongside the target.
• I2. Train on noisier, mismatched data first and finish on clean in-domain data, rather than pooling everything into one set, which hurt here.
• I3. Test each extra data source on its own against a target-only baseline; here more data lowered the score by about 2.5–3 F1 points.
• I4. Average the raw score vectors of several models instead of counting their votes; this did slightly better at equal ensemble size.
• I5. Analyse which labels get confused with which (here Kuwait with Bahrain) to locate the hard boundaries, rather than relying on one aggregate score.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): when assigning a VerbNet frame or speech act to a short sentence, retrieve similar annotated sentences with their frame labels as hints, and show them as the evidence behind the decision.
• I2 (learn): pretrain the parser on multi-witness silver trees, then fine-tune on gold UD treebanks as a final stage instead of mixing silver and gold.
• I3 (learn): add a fixed check that every new silver or converted corpus must beat the gold-only baseline before it enters training.
• I4 (tree): combine arc and label scores from several parser runs before decoding, rather than voting over finished trees.
• I5 (learn): report the most confused label pairs for dependency relations or frames, so that repair rules can target exactly those boundaries.