Elamur Research⌕
☰
home / papers / 2024 / OSACT 2024 Task 2: Arabic Dialect to MSA Translation

OSACT 2024 Task 2: Arabic Dialect to MSA Translation

Summary

This paper describes a system for the OSACT 2024 shared task on translating Arabic dialects into Modern Standard Arabic (MSA). The task covers five dialect groups: Gulf, Egyptian, Levantine, Iraqi and Maghrebi. Organizers supplied 1001 parallel sentences for development and 1888 blind test sentences. The authors compared three families of approaches:

  • Supervised fine-tuning. AraT5 and NLLB 3.3B were fine-tuned on the MADAR corpus, which was regrouped into the task's five regions. AraT5 was trained both as one general model and as separate dialect-specific models.
  • Off-the-shelf NLLB. NLLB 3.3B was run with no tuning.
  • Prompting. GPT-3.5 was used in zero-shot and 3-shot settings, with a final variant adding step-by-step self-correction instructions to the prompt.

Prompting won clearly. The best result came from 3-shot GPT-3.5 with self-correction, at 29.61 BLEU. All fine-tuned seq2seq models stayed around 9–12 BLEU. The work matters for low-resource MT across closely related varieties. It suggests that a general LLM with a few well-chosen examples can beat small specialised models trained on a mismatched parallel corpus.

Key points

  • Data splits:

    • Validation: 1001 pairs (200 Egyptian, 200 Maghrebi, 200 Levantine, 201 Gulf, 200 Iraqi).
    • Test: 1888 pairs (314 Egyptian, 343 Maghrebi, 568 Levantine, 77 Iraqi, 586 Gulf).

  • Hard cases include Arabised English (English words written in Arabic script), words with repeated letters, and long sentences over 128 characters (22 in validation, 45 in test).

  • MADAR fine-tuning corpus: 25 city varieties with 2,000 sentences each plus MSA equivalents. The cities were mapped into the five task regions for dialect-specific training.

  • Test BLEU scores:

    SystemBLEU
    NLLB-3.3B fine-tuned9.00
    AraT5 fine-tuned9.41
    AraT5 dialect-specific10.41
    NLLB-3.3B untuned11.96
    ChatGPT 0-shot21.84
    ChatGPT 3-shot28.00
    ChatGPT 3-shot + self-correction29.61

    The abstract attributes 11.96 to a fine-tuned NLLB, which conflicts with the results table.

  • AraT5 without fine-tuning failed to produce MSA output. Its pretraining targeted dialect→English translation, not dialect→MSA.

Interesting ideas

  • I1. Telling the LLM to check and repair its own draft, step by step inside one prompt, added about 1.6 BLEU over plain 3-shot prompting at no training cost. — For Mova (learn): LLM teacher tiers could use a built-in "verify then fix" instruction when producing silver parses or frames, cutting noise before witness-parser agreement.
  • I2. Few-shot examples were kept fixed per dialect instead of re-sampled, so outputs within a variety are produced under identical conditions. — For Mova (learn): fixed per-genre exemplars for teacher prompts would make silver data reproducible and let regressions be traced to model changes rather than prompt drift.
  • I3. The 25 fine-grained city varieties were pooled into five regional groups, each trained separately, as a middle ground between one global model and many tiny ones. — For Mova (tree): treebanks of English registers or dialects could be clustered into a few groups for per-group parser fine-tuning or UD dialect conversion rules, balancing data size against specificity.
  • I4. The evaluation data explicitly flags hard phenomena (transliterated foreign words, letter elongation, very long sentences) as separate challenge cases. — For Mova (tokens): normalising elongated spellings (e.g. "soooo") before lemma lookup, and keeping such cases as dedicated test slices, would show where the tokenizer and morphology break.
  • I5. Fine-tuning NLLB on a mismatched corpus scored below the untuned model, a useful negative result. — For Mova (learn): every training run should be compared against the untouched baseline as a negative control, so harmful domain shifts are caught rather than assumed to help.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

6 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 4 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{atwany2024osact,
  title = {{OSACT 2024 Task 2: Arabic Dialect to MSA Translation}},
  author = {Atwany, Hanin and Rabih, Nour and Mohammed, Ibrahim and Waheed, Abdul and Raj, Bhiksha},
  booktitle = {Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation @ LREC-COLING 2024},
  year = {2024},
  doi = {10.63317/4zrvabtejteo},
  url = {https://doi.org/10.63317/4zrvabtejteo}
}

References

16 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Hamdy Mubarak, Shammur Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Samir Abdaljalil, Yassine El Kheir, Daniel Izham, Fahim Dalvi, et al. 2024. Larabench: Benchmarking arabic ai with large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 487–520. find our card
  2. Muhammad Abdul-Mageed, Chiyu Zhang, Abdel-Rahim Elmadany, Houda Bouamor, and Nizar Habash. 2022. NADI 2022: The third nuanced Arabic dialect identification shared task. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 85–97, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find
  3. Rania Al-Sabbagh. 2024. The negative transfer effect on the neural machine translation of egyptian arabic adjuncts into english: The case of google translate. International Journal of Arabic-English Studies, 24(1):95–118. find
  4. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018a. The MADAR Arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  5. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, et al. 2018b. The madar arabic dialect corpus and lexicon. In Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018). find
  6. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. find
  7. Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. ↗
  8. Salima Harrat, Karima Meftouh, and Kamel Smaïli. 2017. Machine translation for arabic dialects (survey). Information Processing Management, 56:262–273. find
  9. Karima Kadaoui, Samar M Magdy, Abdul Waheed, Md Tawkat Islam Khondaker, Ahmed Oumar El-Shangiti, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. Tarjamat: Evaluation of bard and chatgpt on machine translation of ten arabic varieties. arXiv preprint arXiv:2308.03051. ↗ our card
  10. Yasir Abdelgadir Mohamed, Akbar Khanan, Mohamed Bashir, Abdul Hakim HM Mohamed, Mousab AE Adiel, and Muawia A Elsadig. 2024. The impact of artificial intelligence on language translation: A review. IEEE Access, 12:25553– 25579. find
  11. Youness Moukafih, Nada Sbihi, Mounir Ghogho, and Kamel Smaïli. 2021. Improving machine translation of arabic dialects through multi-task learning. In International Conference of the Italian Association for Artificial Intelligence, pages 580– 590. Springer. find our card
  12. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2022. Arat5: Text-to-text transformers for arabic language generation. find our card
  13. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. ↗
  14. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2023. Exploring the limits of transfer learning with a unified text-to-text transformer. find
  15. Malik Sallam and Dhia Mousa. 2024. Evaluating chatgpt performance in arabic dialects: A comparative study showing defects in responding to jordanian and tunisian general health prompts. Mesopotamian Journal of Artificial Intelligence in Healthcare, 2024:1–7. find
  16. Mohamed Ali Sghaier and Mounir Zrigui. 2020. Rule-based machine translation from tunisian dialect to modern standard arabic. Procedia Computer Science, 176:310–319. Knowledge-Based and Intelligent Information Engineering Systems: Proceedings of the 24th International Conference KES2020. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/atwany-2024-osact-2024-task-2-arabic/