📄 Alson at NADI 2024 shared task: Alson - A fine-tuned model for Arabic Dialect Translation

Summary

The paper describes a system for Subtask 3 of the NADI 2024 shared task, which is sentence-level translation from Arabic dialects into Modern Standard Arabic (MSA). The target dialects are Egyptian, Emirati, Jordanian and Palestinian. Parallel dialect–MSA data is scarce, so the authors built a training set from three sources:

• dialect–MSA pairs from the MADAR corpus;
• the Emi-NADI corpus, which covers Emirati, a dialect missing from MADAR;
• several hundred Jordanian and Palestinian sentences from NADI 2021, translated into MSA with ChatGPT (GPT-3.5) and then manually checked.

They fine-tuned three AraT5v2 variants and compared them. AraT5v2-base-1024 was clearly the strongest, reaching BLEU 0.1650 on the dev set and 0.1746 on the test set.

Two other findings stand out. A smaller training set limited to the four target dialects worked better than the full multi-city MADAR data. Errors grew as dialect words moved further from MSA. The work matters as a practical, low-resource recipe: pretrained model selection, LLM-assisted data augmentation and targeted data curation together give usable dialect normalization.

Key points

• Data sources: MADAR (25 city dialects plus English and MSA), Emi-NADI for Emirati, and 846 NADI 2021 sentences (420 Palestinian, 426 Jordanian) translated by GPT-3.5.
• Training set sizes: about 91.7k pairs for MADAR + Jor-Pal-NADI, versus about 20.5k for the 4-dialect + Jor-Pal-NADI set.
• Preprocessing: URLs, punctuation, digits, extra whitespace and diacritics (vowel marks) were removed.
• Model comparison: AraT5v2-msa-small peaked at about 0.024 BLEU and AraT5v2-msa-base at about 0.056. AraT5v2-base-1024 reached 0.1650 on dev and 0.1746 on test.
• Best configuration: learning rate 7e-5, batch size 16, only 2 epochs.
• Evaluation splits: the dev set had 400 sentences (100 per dialect) and the test set 2,000 sentences (500 per dialect, no references released).
• Per-dialect results: Emirati scored lowest, while Jordanian and Palestinian scored highest and close to each other.

Interesting ideas

• I1. A curated subset covering only the target varieties outperformed a corpus over four times larger spanning many dialects, so matching the training distribution to the evaluation distribution can matter more than raw volume.
• I2. Gaps for underrepresented varieties were filled by having an off-the-shelf LLM produce translations, followed by a human quality check before training.
• I3. The error analysis links failures to how far a variety's words sit from the standard form, and reports scores per variety rather than one aggregate.
• I4. The best model came after just two epochs, and longer training sometimes lowered scores while costing more compute.
• I5. The authors noticed that verb tense and repeated structures were often mistranslated, which shows that a decent BLEU can hide systematic grammatical losses.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): When training gparse or the perceptron parser, compare a register-matched subset of treebanks against the full mix rather than assuming more treebanks always help.
• I2 (learn): This mirrors LLM teacher tiers producing silver data; a cheap manual review pass on a small LLM-generated slice could gate its admission alongside witness-parser agreement.
• I3 (learn): Evaluation could report results per genre or register (tales, school tests, informal text) and test whether accuracy drops with distance from edited standard English.
• I4 (learn): Track dev scores per epoch and stop early by default when fine-tuning gparse, especially on small silver sets where overfitting is likely.
• I5 (back): In the text → graph → text compressor and meta-world translation, add explicit checks that tense and repetition survive the round trip, rather than relying on overlap metrics alone.