Elamur Research⌕
☰
home / papers / 2023 / UniManc at NADI 2023 Shared Task: A Comparison of Various T5-based Models for Translating Arabic Dialectical Text to Modern Standard Arabic

UniManc at NADI 2023 Shared Task: A Comparison of Various T5-based Models for Translating Arabic Dialectical Text to Modern Standard Arabic

Paper · 2023 · Abdullah Khered, Ingy Abdelhalim, Nadine Abdelhalim et al.

Summary

The paper tackles sentence-level translation from four Arabic dialects (Egyptian, Emirati, Jordanian, Palestinian) into Modern Standard Arabic (MSA) for the NADI 2023 shared task. Such parallel data is scarce, and MSA-trained systems handle dialects poorly. The UniManc team fine-tuned three T5-family models: AraT5 v2, mT5 and mT0. They compared two set-ups: one joint model covering all regional dialects (J-R) and a separate model per dialect (I-R). They also tested beam sizes from 1 to 5.

The closed track allowed only the MADAR corpus. The open track added PADIC, Dial2MSA, an Arabic STS dataset and Emi-NADI, a new Emirati-to-MSA set the authors built with GPT 3.5 Turbo and partly checked by hand.

AraT5, which was pretrained on Arabic only, consistently beat the multilingual models. The team ranked first in both tracks: 14.76 BLEU in the closed track (I-R AraT5, beam 3) and 21.10 in the open track (J-R AraT5). The work shows that a language-specific pretrained model plus modest data engineering is effective for low-resource dialect normalisation.

Key points

  • Models compared: AraT5 v2 (UBC-NLP/AraT5v2-base-1024), mT5 and mT0. Training used learning rate 5e-5, batch 16, at most 40 epochs and 128-token limits, keeping the checkpoint with the best dev score.
  • Emi-NADI contains 2,712 Emirati tweets (taken from NADI Subtask 1 data) translated into MSA by GPT 3.5 Turbo. Native speakers verified 1,000 of them, and both checked and unchecked pairs were used for training.
  • Related varieties were borrowed to fill gaps: Gulf data stood in for Emirati and Levantine data for Jordanian/Palestinian. Totals were roughly 32.9k Egyptian, 20.9k Gulf and 31.4k Levantine pairs.
  • On the dev sets, per-dialect (I-R) models usually beat the joint model. On the open-track test set, however, the joint AraT5 came out on top. A post-evaluation rerun of that model with beam 5 reached 21.87 BLEU.
  • In error analysis, several low-BLEU outputs were actually correct paraphrases. One Palestinian failure came from code-mixing: a loanword ("bravo") written in Arabic script.

Interesting ideas

  • I1. For every dialect→MSA pair, the authors also added an MSA→MSA copy pair, so the model learns to keep words the two varieties share instead of rewriting them.
  • I2. When a variety lacks data, they substituted a closely related one (Gulf for Emirati, Levantine for Jordanian) rather than leaving the gap empty.
  • I3. The new parallel set was produced by an LLM, and only a subset was human-checked before everything was used in training.
  • I4. Manual review of the worst-scoring outputs showed that BLEU penalised translations with the right meaning but different wording.
  • I5. Per-variety specialist models won on dev, but the joint model won on test, and the specialists also need the input's dialect identified first.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): When training a converter between UD dialects or treebank conventions, include trees already in the target convention mapped to themselves, so the converter learns not to over-edit correct structure.
  • I2 (tree): Parser training for an under-covered English genre (e.g., tweets, fable prose) could draw on the nearest-genre treebanks, kept as a separately tagged source.
  • I3 (learn): This fits the LLM-teacher tier. Recording which silver items were human-verified would let evaluation measure how much the unverified portion helps or hurts.
  • I4 (back): Text→graph→text output should be scored by comparing the meaning graphs of output and reference, with surface-overlap scores only as a secondary signal.
  • I5 (learn): Before adopting a split-by-domain design, check that its dev advantage holds on held-out data, and count the routing step as an added explainable dependency.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of ManchesterManchester · United Kingdom87 works · 44,938 citations🎓King Abdulaziz UniversityJeddah · Saudi Arabia31 works · 508 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Abdullah Khered, Ingy Abdelhalim, Nadine Abdelhalim, Ahmed Soliman and Riza Batista-Navarro. 2023. UniManc at NADI 2023 Shared Task: A Comparison of Various T5-based Models for Translating Arabic Dialectical Text to Modern Standard Arabic. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.71

@inproceedings{khered2023unimanc,
  title = {{UniManc at NADI 2023 Shared Task: A Comparison of Various T5-based Models for Translating Arabic Dialectical Text to Modern Standard Arabic}},
  author = {Abdullah Khered, Ingy Abdelhalim, Nadine Abdelhalim, Ahmed Soliman and Riza Batista-Navarro},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.71},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.71}
}

References

18 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, Chiyu Zhang, ElMoatez Billah Nagoudi, Houda Bouamor, and Nizar Habash. 2023. NADI 2023: The Fourth Nuanced Arabic Dialect Identification Shared Task. In Proceedings of The First Arabic Natural Language Processing Conference (ArabicNLP 2023).
  2. Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020. NADI 2020: The first nuanced Arabic dialect identification shared task. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 97–110, Barcelona, Spain (Online). Association for Computational Linguistics. ↗
  3. Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2021. NADI 2021: The second nuanced Arabic dialect identification shared task. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 244–259, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. ↗
  4. Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2022. find
  5. NADI 2022: The third nuanced Arabic dialect identification shared task. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 85–97, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  6. Mahmoud Al-Sobh, Abdel-Rahman Abu-Melhim, and Nedal Bani Hani. 2015. Diglossia as a result of language variation in arabic: Possible solutions in light of language planning. Journal of Language Teaching and Research, 6:274. ↗
  7. Mansour Al Sulaiman, Abdullah M. Moussa, Sherif Abdou, Hebah Elgibreen, Mohammed Faisal, and Mohsen Rashwan. 2022. Semantic textual similarity for modern standard and dialectal arabic using transfer learning. PLOS ONE, 17(8):1–14. find
  8. Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. The MADAR shared task on Arabic fine-grained dialect identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 199–207, Florence, Italy. Association for Computational Linguistics.
  9. Markus Freitag and Yaser Al-Onaizan. 2017. Beam search strategies for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 56–60, Vancouver. Association for Computational Linguistics.
  10. Salima Harrat, Karima Meftouh, and Kamel Smaili. 2019. Machine translation for arabic dialects (survey). Information Processing Management, 56(2):262–273. Advance Arabic Natural Language Processing (ANLP) and its Applications. find
  11. Karima Meftouh, Salima Harrat, and Kamel Smaïli. 2018. PADIC: extension and new experiments. In 7th International Conference on Advanced Technologies ICAT. find
  12. Hamdy Mubarak. 2018. Dial2MSA: A Tweets Corpus for Converting Dialectal Arabic to Modern Standard Arabic. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), OSACT2018 Workshop, pages 49–53. find
  13. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, and Teven et al. Le Scao. 2023. Crosslingual generalization through multitask finetuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15991– 16111, Toronto, Canada. Association for Computational Linguistics. ↗
  14. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2022. AraT5: Text-to-text transformers for Arabic language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 628–647, Dublin, Ireland. Association for Computational Linguistics.
  15. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the ACL, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  16. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. ↗
  17. Abdulhadi Shoufan and Sumaya Alameri. 2015. Natural language processing for dialectical Arabic: A survey. In Proceedings of the Second Workshop on Arabic Natural Language Processing, pages 36–48, Beijing, China. Association for Computational Linguistics.
  18. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/khered-2023-unimanc-nadi-2023-shared-task/