📄 Arabic Train at NADI 2024 shared task: LLMs' Ability to Translate Arabic Dialects into Modern Standard Arabic

Summary

This paper describes a team entry for Subtask 3 of the NADI 2024 shared task: translating sentences from four Arabic dialects (Egyptian, Emirati, Jordanian, Palestinian) into Modern Standard Arabic (MSA). The authors compared several systems: Jais, NLLB, GPT-3.5, GPT-4 and a Google Translate round-trip baseline.

They tested prompting in zero-shot and few-shot modes. They also fine-tuned two models on a custom dataset. To build it, they sent Arabic sentences through Google Translate into English or French and back, then paired these noisy outputs with gold MSA. This turns translation into a correction problem. NLLB was fine-tuned this way, and Jais with LoRA. MADAR was also used as a resource.

The best setup was Jais with a mixed prompting scheme, one-shot for some dialects and zero-shot for others. It reached 19.48 average BLEU on validation and 20.44 on test, which placed the team second. Fine-tuning hurt rather than helped. The study suggests that a model pre-trained on Arabic beats larger general-purpose LLMs on this task.

Key points

• The validation set has 400 sentences (100 per dialect) and the test set 2,000 (500 per dialect). Some sentences are long (one has 58 words) or metaphorical.
• Validation average BLEU by system:
  • Jais combination approach: 19.48
  • Jais 0-shot: 18.72
  • GPT-3.5 2-shot: 16.67
  • GPT-4 2-shot: 16.53
  • Google Translate: 11.93
  • Jais fine-tuned on custom data: 9.58
  • NLLB fine-tuned on custom data: 8.77
• In the combination approach, Egyptian and Emirati inputs received one-shot prompts, while Jordanian and Palestinian inputs were given no examples at all. Jais already appeared to handle the last two well without examples.
• Generation settings were temperature 0.1 and top_p 0.4. The few-shot examples stayed fixed within each dialect: 2-shot for GPT and 1-shot for Jais.
• The submitted GPT-3.5 2-shot run scored 0 BLEU on the test set, which the authors attribute to hallucination. Most remaining errors came from named entities and culture-specific words such as dish names.

Interesting ideas

• I1. Choose the prompting regime separately for each language variety from validation scores, since examples help on some varieties and hurt on others.
• I2. Recast translation as correction: give the model a rough machine draft and train it to repair the draft toward the gold target.
• I3. Push sentences through a pivot language and back to get paraphrased, noisy variants of the same content.
• I4. Report scores for each variety alongside the overall average, so a strong mean cannot hide a weak subgroup.
• I5. The error analysis found that culture-bound items such as local dishes and named entities cause most mistakes.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): LLM teacher tiers could use per-genre or per-register prompt configurations selected on held-out data, instead of one global prompt.
• I2 (back): text generation from the meaning graph could produce a crude draft first and then apply explainable repair steps, logging each fix.
• I3 (learn): round-trip variants could serve as robustness tests and negative controls, checking that parses and frames stay stable when only the surface wording changes.
• I4 (tree): parser evaluation could always break down LAS per treebank and per UD dialect conversion, not only give a pooled number.
• I5 (world): the system could flag such entities and resolve them through the Wikidata/WordNet ontology before downstream reasoning, rather than treating them as ordinary words.