📄 TF-IDF or Transformers for Arabic Dialect Identification? ITFLOWS participation in the NADI 2022 Shared Task

Summary

The paper describes the ITFLOWS team's entry to NADI 2022 Subtask 1, which asks systems to label tweets with the country-level Arabic dialect they come from. The labels come from geolocation, not linguistic annotation. The training set was small and imbalanced, and external labeled data was forbidden. The authors compared three systems:

• a classical TF-IDF "fingerprint" classifier built on character n-grams, with an optional transliteration step;
• MARBERT fine-tuned in full;
• MARBERT trained through adapters on a training set where minority classes were augmented with contextual word insertion.

The adapter model was strongest on the development data and on TEST-B. However, it did worst on TEST-A, where the TF-IDF system had the best accuracy. Together the submissions placed second in the shared task on average. The authors conclude that much dialect signal probably sits below the word level, which MARBERT's tokenization hides. They suggest that subword- or transliteration-aware Arabic transformers are worth building. The work matters because a cheap, transparent baseline matched a large pretrained model on one test set.

Key points

• Data: 20,398 training tweets and 4,872 validation tweets cover 18 country dialects. The unlabeled test sets are TEST-A (4,758 tweets) and TEST-B (1,473 tweets, with an undisclosed set of dialects). Evaluation uses macro-averaged F1.
• TF-IDF system:
  • All tweets of one label are merged into a single "document," and 1–7-gram patterns serve as fingerprints.
  • A pattern must exceed a frequency threshold inside its dialect and be missing from at least k dialects.
  • (3, 9) patterns gave the best accuracy, and prediction is by maximum likelihood.
  • Transliteration helped, while balancing classes by random sampling hurt accuracy.
• Augmentation:
  • Every class below 10% of the data received one or two words inserted via MARBERT contextual embeddings, following the EDA insertion idea.
  • Duplicates were then removed, so that, for example, "uae" grew from 642 to 1,284 tweets.
• Training setups:
  • Full fine-tuning: MARBERT feeds max pooling and dense layers, trained for 5 epochs with early stopping on balanced accuracy.
  • Adapters: task adapters use PfeifferConfig, with the last-layer adapter dropped (MAD-X 2.0). Settings were lr 1e-4, batch 16 and 6 epochs, and the checkpoint at step 4500 was used.
• Errors:
  • The model over-predicts large classes and confuses neighboring dialects; for example, UAE tweets were often tagged as Oman or KSA.
  • Qatar and Bahrain received zero correct DEV predictions.

Interesting ideas

• I1. Visually identical words can hide different Unicode code points, such as Arabic presentation forms versus base letters, so normalizing or transliterating to one alphabet before n-gram counting removes false distinctions.
• I2. Treating each class as one document gives readable fingerprints: patterns frequent in a class but absent from at least k other classes, scored by likelihood. This makes a fully inspectable classifier.
• I3. Augmenting only under-represented classes, by inserting one or two context-predicted words and then deduplicating, enlarges rare classes without flooding common ones.
• I4. The best model on dev was the worst on one test split, while the simple baseline stayed consistent across two differently sampled test sets. This shows the value of scoring on several splits and tracking stability.
• I5. The confusion matrix showed errors drifting toward similar, larger classes, which separates prior bias from genuine confusability.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): Add a Unicode normalization pass (NFKC, ligatures, curly quotes, full-width forms) ahead of the en tokenizer, and log each mapping so the step stays explainable.
• I2 (tree): Such fingerprints could detect which treebank style or English variety an input resembles, so the right UD dialect conversion is picked with a visible justification.
• I3 (learn): Rare frames or construction types in silver data could be expanded this way, provided the witness parsers still agree on the modified sentences.
• I4 (learn): Mova's evaluation could always report variance across held-out sets with different sampling, keeping a simple transparent baseline as a control.
• I5 (frames): Speech-act or semantic-role confusion matrices could check whether errors collapse into frequent neighboring labels, guiding targeted repair rules.