Elamur Research⌕
☰
home / papers / 2024 / Sirius_Translators at OSACT6 2024 Shared Task: Fin-tuning Ara-T5 Models for Translating Arabic Dialectal Text to Modern Standard Arabic

Sirius_Translators at OSACT6 2024 Shared Task: Fin-tuning Ara-T5 Models for Translating Arabic Dialectal Text to Modern Standard Arabic

Paper · 2024 · Salwa Saad Alahmari

Summary

The paper addresses translating sentences in Arabic dialects into Modern Standard Arabic (MSA). The task comes from Shared Task 2 of OSACT6 (LREC-COLING 2024) and covers five dialect groups: Gulf, Egyptian, Levantine, Iraqi and Maghrebi. Most earlier work built one system per dialect. The Sirius_Translators team (Alahmari, with Atwell and Saadany) instead trained a single model for all five dialects.

They pooled about 180k dialect–MSA sentence pairs from five public corpora: MADAR, PADIC, Dial2MSA, Arabic STS and SADID. They also added extra pairs to help the model with vocabulary that dialects share with MSA. Four AraT5 variants were fine-tuned with identical hyper-parameters so they could be compared fairly.

AraT5v2-base-1024 performed best, with BLEU 21.0 on the development set. Its score fell to 9.57 on the hidden test set. The authors attribute the weak results mainly to uneven data coverage, especially for Iraqi, and to having no time to tune hyper-parameters. The work offers a simple, reproducible multi-dialect baseline and points to data scarcity as the main bottleneck.

Key points

  • Data splits:
    • Train: 180,211 sentences.
    • Dev: 1,001 sentences, roughly 200 per dialect.
    • Test: 1,888 sentences, roughly 377 per dialect.
  • Training pairs per dialect after pooling:
    • Levantine: 52,421
    • Maghrebi: 37,112
    • Gulf: 36,168
    • Egyptian: 35,910
    • Iraqi: 18,600
  • MADAR was the only training corpus covering all five dialects. PADIC was used in full, including translations that had never been manually validated.
  • Dev BLEU by model:
    • AraT5v2-base-1024: 21.0
    • AraT5 base: 19.26
    • AraT5-MSA-Base: 16.88
    • AraT5-MSA-Small: 15.97
  • The best model reached only 9.57 BLEU on the test set.
  • Shared fine-tuning setup:
    • learning rate 5e-5
    • source and target length 128
    • batch size 16
    • 2 epochs
    • Hugging Face / PyTorch Transformers

Interesting ideas

  • I1. Training one model jointly on all dialect varieties, instead of one model per dialect, lets scarce dialects borrow strength from richer ones through shared structure.
  • I2. Extra pairs were added so the model learns which words stay unchanged between a dialect and the standard form. This guards against "translating" tokens that are already correct.
  • I3. All candidate models were fine-tuned with exactly the same hyper-parameters, so differences in score reflect the architecture rather than tuning luck.
  • I4. Counting training pairs per variety made it possible to trace weak results to a specific under-represented slice, in this case Iraqi.
  • I5. The large drop from development to test score shows that a dev set close to the training data can hide poor generalization.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): The same idea could guide gparse: train one parser on several English varieties or UD dialect conversions together, rather than keeping a separate parser per source.
  • I2 (back): In translation through the meta-world, identity or near-identity examples could teach the generator to keep shared lexical items verbatim instead of paraphrasing them.
  • I3 (learn): When comparing witness parsers or teacher tiers, holding the training recipe fixed keeps the comparison honest and the outcome explainable.
  • I4 (learn): Per-category coverage tables for silver data (genre, construction, verb class) would show which gaps cause errors before retraining starts.
  • I5 (learn): Evaluation should always include held-out, differently sourced control sets, so that dev-set gains are not mistaken for real progress.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 1,484 works

Cite

Cite as Salwa Saad Alahmari. 2024. Sirius_Translators at OSACT6 2024 Shared Task: Fin-tuning Ara-T5 Models for Translating Arabic Dialectal Text to Modern Standard Arabic. Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation @ LREC-COLING 2024.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{alahmari2024sirius,
  title = {{Sirius\_Translators at OSACT6 2024 Shared Task: Fin-tuning Ara-T5 Models for Translating Arabic Dialectal Text to Modern Standard Arabic}},
  author = {Salwa Saad Alahmari},
  booktitle = {Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation @ LREC-COLING 2024},
  year = {2024},
  doi = {10.63317/59fi88he3h8x},
  url = {https://doi.org/10.63317/59fi88he3h8x}
}

References

21 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2022. NADI 2022: The third nuanced Arabic dialect identification shared task. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 85–97, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  2. Wael Abid. 2020. The sadid evaluation datasets for low-resource spoken language machine translation of arabic dialects. In International Conference on Computational Linguistics.
  3. Roqayah M. Al-Ibrahim and Rehab Duwairi. 2020. Neural machine translation from jordanian dialect to modern standard arabic. 2020 11th International Conference on Information and Communication Systems (ICICS), pages 173–178. find
  4. Mahmoud A Al-Sobh, Abdel-Rahman H Abu-Melhim, and Nedal A Bani-Hani. 2015. Diglossia as a result of language variation in arabic: Possible solutions in light of language planning. Journal of Language Teaching and Research, 6(2):274. ↗
  5. Mansour Al Sulaiman, Abdullah M. Moussa, Sherif Abdou, Hebah Elgibreen, Mohammed Faisal, and Mohsen Rashwan. 2022. Semantic textual similarity for modern standard and dialectal arabic using transfer learning. PLOS ONE, 17(8):1–14. find
  6. Laith H. Baniata, Seyoung Park, and Seong-Bae Park. 2018. A multitask-based neural machine translation model with part-of-speech tags integration for arabic dialects. Applied Sciences, 8(12). ↗
  7. Houda Bouamor, Nizar Habash, and Kemal Oflazer. 2014. A multidialectal parallel corpus of arabic. In LREC, pages 1240–1245.
  8. Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. The MADAR shared task on Arabic fine-grained dialect identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 199–207, Florence, Italy. Association for Computational Linguistics.
  9. Injy Hamed, Nizar Habash, Slim Abdennadher, and Ngoc Thang Vu. 2023. Investigating lexical replacements for Arabic-English code-switched data augmentation. In Proceedings of the The Sixth Workshop on Technologies for Machine Translation of Low-Resource Languages (LoResMT 2023), pages 86–100, Dubrovnik, Croatia. Association for Computational Linguistics. find
  10. Saméh Kchaou, Rahma Boujelbane, Emna Fsih, and Lamia Hadrich-Belguith. 2022. Standardisation of dialect comments in social networks in view of sentiment analysis : Case of Tunisian dialect. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 5436–5443, Marseille, France. European Language Resources Association. find
  11. Abdullah Khered, Ingy Abdelhalim, Nadine Abdelhalim, Ahmed Soliman, and Riza Batista-Navarro. 2023. UniManc at NADI 2023 shared task: A comparison of various t5-based models for translating Arabic dialectical text to Modern Standard Arabic. In Proceedings of ArabicNLP 2023, pages 658–664, Singapore (Hybrid). Association for Computational Linguistics. find
  12. K. Meftouh, S Harrat, and Kamel Smaïli. 2018. PADIC: extension and new experiments. In 7th International Conference on Advanced Technologies ICAT, Antalya, Turkey. find
  13. Hamdy Mubarak. 2018. Dial2msa: A tweets corpus for converting dialectal arabic to modern standard arabic. OSACT, 3:49. find
  14. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2021. Investigating code-mixed Modern Standard Arabic-Egyptian to English machine translation. In Proceedings of the Fifth Workshop on Computational Approaches to Linguistic Code-Switching, pages 56–64, Online. Association for Computational Linguistics. find
  15. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2022a. AraT5: Text-to-text transformers for Arabic language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 628–647, Dublin, Ireland. Association for Computational Linguistics.
  16. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2022b. TURJUMAN: A public toolkit for neural Arabic machine translation. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 1–11, Marseille, France. European Language Resources Association. ↗
  17. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. ↗
  18. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. ↗
  19. Houcemeddine Turki, Emad Adel, Tariq Daouda, and Nassim Regragui. 2016. A conventional orthography for maghrebi arabic. In International Conference on Language Resources and Evaluation. find
  20. Omar F. Zaidan and Chris Callison-Burch. 2014. Arabic dialect identification. Computational Linguistics, 40(1):171–202. ↗
  21. Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1568–1575, Austin, Texas. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alahmari-2024-sirius-translators-osact6-2024-shared/