📄 MA at AraFinNLP2024: BERT-based Ensemble for Cross-dialectal Arabic Intent Detection

Summary

The paper tackles banking intent detection across Arabic dialects for subtask-1 of the AraFinNLP 2024 shared task. ArBanking77, the only Arabic banking intent dataset, covers just MSA and Palestinian, so systems trained on it struggle with other dialects. The authors expanded the training data by machine-translating the MSA queries into further dialects with the distilled 600M NLLB model. They then removed stray diacritics and symbols that the translation introduced. On this enlarged data they fine-tuned six BERT classifiers built on AraBERTv0.2-Twitter and MARBERTv2. Some classifiers combined cross-entropy with an NT-Xent contrastive term. The six models were made deliberately different in backbone, loss, data cleaning and dialect coverage. Their predicted probabilities are averaged with equal weight. The ensemble reached an F1 of about 0.877 and ranked first in the subtask, beating every single member. The work shows that cheap synthetic dialect data and a carefully diversified ensemble can make up for missing dialect resources in a narrow domain.

Key points

• ArBanking77 has 31,404 MSA and Palestinian queries over 77 intents. After augmentation, the training set holds 85,957 samples across eight varieties: MSA, PAL, South and North Levantine, Egyptian, Tunisian, Moroccan and Saudi, with roughly 10.7k samples each. The shared-task test set contains 11,721 queries.
• The ensemble has six members: four AraBERTv0.2-Twitter models (about 136M parameters) and two MARBERTv2 models (about 160M parameters). Each member has a single linear classification head, and outputs are combined by averaging the probabilities.
• The loss is cross-entropy plus α·NT-Xent, with α set to 10. A custom batch sampler limits how many distinct classes appear in each batch, so every sample has both positives and negatives.
• Ensemble test F1 is 0.8773 in the results table; the abstract reports 0.8771. The best single model is m3 (AraBERT, preprocessed data, contrastive loss) at 0.8578. The worst is m4, trained only on MSA and PAL, at 0.7913.
• Preprocessing helped, raising F1 from 0.8434 (m2) to 0.8547 (m1). With identical settings, AraBERTv0.2-Twitter outperformed MARBERTv2, 0.8434 (m2) against 0.8198 (m5).

Interesting ideas

• I1. Use an MT model to turn labelled standard-language examples into dialect copies that keep their labels, giving cheap coverage of unseen varieties despite translation noise.
• I2. Build ensemble diversity on purpose by varying the backbone, the loss, cleaned versus raw data, and dialect subsets, because members that err differently make averaging more effective.
• I3. Train some models on raw text and others on cleaned text, so the group as a whole tolerates imperfect input.
• I4. Control batch composition so that a contrastive objective always has in-class positives and out-of-class negatives, instead of relying on random batches.
• I5. Keep a control member trained only on the original narrow data (m4) to measure directly what augmentation adds.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Paraphrasing silver sentences into informal, regional or non-native English could widen training coverage. Labels that survive the rewrite would need to be checked by witness-parser agreement before use.
• I2 (learn): Witness parsers for silver data should be chosen to fail in different ways. Their agreement then carries more evidence than votes from near-identical models.
• I3 (tokens): Training tagger and lemmatizer variants on both normalized and noisy text could make the pipeline robust to typos and odd symbols. A disagreement between the variants would also flag a suspicious token in an explainable way.
• I4 (frames): If frames or VerbNet classes are ever learned with a contrastive objective, a class-limited sampler would ensure rare roles meet real contrasts in every batch.
• I5 (learn): Every augmentation or repair-rule experiment should be compared against a baseline trained without it, which serves as a built-in negative control.