📄 rematchka at NADI 2023 shared task: Parameter Efficient tuning for Dialect Identification and Dialect Machine Translation

Summary

The paper describes one team's entry to NADI 2023, which had three subtasks. The first was country-level Arabic dialect identification over 18 countries. The second and third were sentence-level translation from four dialects into MSA. For identification, the author compared several BERT-based Arabic encoders: MARBERT, AraBERT, QARiB, AraELECTRA and CAMeLBERT. Each was trained in different ways, including standard fine-tuning, prompt-tuning with a verbalizer of Arabic country names, soft prompts, prefix-tuning, adapters, few-shot contrastive training, and continued pre-training. The final systems were weighted ensembles whose weights were optimized on the dev set. For translation, AraT5v2 was trained with full fine-tuning, LoRA, and prompt-tuning. The identification ensemble reached F1 86.18 and placed second. The translation system reached BLEU 11.37, placing fourth and third. The main lesson is that parameter-efficient methods beat full fine-tuning on dialect classification by a clear margin. They are also cheap to store, because one frozen backbone can serve many tasks.

Key points

• Subtask 1 data: about 18K training, 1.8K dev and 3.6K test instances, covering 18 country dialects.
• Translation data: MADAR-parallel-corpus, with 12,000 training, 400 validation and 2,000 test examples.
• PEFT vs. fine-tuning: on dev, PEFT variants beat conventional fine-tuning by 3–8%.
  • Prompt-tuning of MARBERT was the strongest single approach, at roughly 0.86 F1.
  • A short prompt ("language is [MASK]") beat a longer, more descriptive prompt by about 1%.
• Translation behaviour: AraT5v2 was very sensitive to the learning rate. BLEU was about 8, 9 and 11 at 1e-6, 3e-6 and 6e-6.
  • LoRA used rank 1 and scaling factor 4.
  • The submitted translation run used conventional fine-tuning.
• Error analysis and data choices:
  • Main confusions were between neighbouring dialects: Kuwaiti vs. Bahraini, and Syrian vs. Lebanese.
  • Adding the previous year's NADI data to training lowered performance.

Interesting ideas

• I1. Turning classification into masked-word prediction (a prompt plus a verbalizer of label words) lets the pretrained LM head do the labelling, and a terse prompt can outperform a wordier one.
• I2. Selecting an ensemble by exhaustively searching model subsets and optimizing weights on dev beat simply averaging all models, though at high search cost.
• I3. A compact tag prompt ("CAI => MSA") worked better than a natural-language instruction for steering a seq2seq model, which suggests structured control codes can beat prose instructions.
• I4. Reporting how BLEU moves across several learning rates shows that a single run can misstate a method's value; ranking also changed between dev and test.
• I5. The finding that extra data from an older edition of the task hurt performance is a reminder that distribution shift can outweigh volume.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): A cloze-style probe could help the tagger with ambiguous cases, scoring candidate tags through label words, while keeping prompt wording as a tunable, logged parameter.
• I2 (learn): When combining several witness parsers for silver data, a weighted subset of witnesses may agree more reliably than a vote of all of them.
• I3 (back): Generation from the meaning graph could condition on short register or variety codes rather than verbose instructions.
• I4 (learn): Evaluations should include sweeps or repeated runs as a stability control, so that small dev gains are not taken as real improvements.
• I5 (tree): When merging treebanks or converted UD dialects, each source should be checked against a held-out target set before being added, not merged blindly.