Elamur Research⌕
☰
home / papers / 2024 / ASOS at NADI 2024 shared task: Bridging Dialectness Estimation and MSA Machine Translation for Arabic Language Enhancement

ASOS at NADI 2024 shared task: Bridging Dialectness Estimation and MSA Machine Translation for Arabic Language Enhancement

Summary

This NADI 2024 submission tackles two linked Arabic tasks. The first is predicting a sentence's Arabic Level of Dialectness (ALDi) as a continuous score. The second is translating four dialects (Egyptian, Emirati, Jordanian, Palestinian) into Modern Standard Arabic (MSA).

For ALDi, the authors fine-tuned several Arabic encoders as regressors on the 100k-sentence NADI training set. The MARBERT-based model scored best, with a test RMSE of 0.1403.

For translation, they first trained models in the opposite direction, MSA to dialect. They used these to turn 20k monolingual MSA sentences per dialect into synthetic dialect text, then added MADAR data where it existed. On this combined data they fine-tuned AraT5 V2, Turjuman and mt5 in the dialect-to-MSA direction. AraT5 V2 led with a test BLEU of 0.1713.

Finally, they chained the two tasks: translated MSA outputs were scored by the ALDi model to see how standard they look. This gives a reference-free check of translation quality, and the scores clustered near low dialectness.

Key points

  • ALDi regression results:

    • MARBERT reached a dev RMSE of 0.12801 and a test RMSE of 0.14031.
    • ARBERT looked fine on dev (0.14972) but collapsed on test (0.30607), suggesting overfitting.
    • AraElectra scored 0.18266 on test.

  • ALDi training setup: 1 epoch, batch size 32, 80/20 train/validation split, early stopping with patience 10, and no frozen layers. The training labels lean heavily toward low (MSA-like) scores.

  • Translation data per dialect: 20k synthetic sentences from MSA2Dia models, plus MADAR data (Egyptian 18k, Jordanian 4k, Palestinian 4k, Emirati none).

  • Dia2MSA BLEU scores:

    ModelDev BLEUTest BLEU
    AraT5 V20.28740.1713
    Turjuman0.21120.1560
    mt50.11340.1377

    Emirati had an unusually high dev score of 0.6851.

  • Error analysis by score bucket: splitting ALDi error by percentile segments showed the best accuracy in the middle range (RMSE 0.08669) and weaker results at both extremes. The authors attribute this to sparse high-dialect training data.

Interesting ideas

  • I1. Train a model in the reverse direction first, then use it to turn plentiful monolingual text into synthetic parallel pairs for the direction you actually need; this bootstraps training data cheaply for low-resource varieties. — For Mova (learn): Mova could generate silver sentence–tree pairs by running its graph→text generator over existing meaning graphs, then train parsing on those pairs after witness parsers agree on them.
  • I2. Use an independent scorer, here a dialectness regressor, to judge whether translated output has the target register, without needing reference translations. — For Mova (back): a separate register/formality scorer could check that text regenerated by the compressor or translator keeps the intended style, with each verdict logged as an explainable check.
  • I3. Treat a linguistic property as a continuous score rather than a binary label, which captures mixed and code-switched cases more faithfully. — For Mova (frames): graded scores for properties like irony strength or speech-act directness would let prag express partial cases instead of forcing yes/no frames.
  • I4. Break the error metric down by percentile buckets of the target value, which reveals where a model fails (here, the extremes) rather than hiding it in one average. — For Mova (learn): evaluation could report accuracy per difficulty or rarity bucket, such as sentence length or construction frequency, so repair rules target the weak tails.
  • I5. A sharp dev-to-test drop for one model (ARBERT) served as an overfitting warning that the dev score alone did not show. — For Mova (learn): always compare scores across a held-out set from a different distribution, alongside negative controls, before trusting a parser or reader improvement.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Machine translation

Cited by

2 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 1 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.83

@inproceedings{nacar2024asos,
  title = {{ASOS at NADI 2024 shared task: Bridging Dialectness Estimation and MSA Machine Translation for Arabic Language Enhancement}},
  author = {Nacar, Omer and Sibaee, Serry and Alharbi, Abdullah and Ghouti, Lahouari and Koubaa, Anis},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.83},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.83}
}

References

21 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, Chiyu Zhang, El Moatez Billah Nagoudi, Houda Bouamor, and Nizar Habash. 2023. NADI 2023: The fourth nuanced Arabic dialect identification shared task. In Proceedings of ArabicNLP 2023, pages 600– 613, Singapore (Hybrid). Association for Computational Linguistics. find
  2. Muhammad Abdul-Mageed, Amr Keleg, AbdelRahim Elmadany, Chiyu Zhang, Injy Hamed, Walid Magdy, Houda Bouamor, and Nizar Habash. 2024. NADI 2024: The Fifth Nuanced Arabic Dialect Identification Shared Task. In Proceedings of The Second Arabic Natural Language Processing Conference (ArabicNLP 2024). find
  3. Nizar Habash, Owen Rambow, Mona Diab, and Reem Kanjawi-Faraj. 2008. Guidelines for annotation of arabic dialectness. In Proceedings of the LREC Workshop on HLT & NLP within the Arabic world, pages 49–53. find
  4. Injy Hamed, Nizar Habash, Slim Abdennadher, and Ngoc Thang Vu. 2022. Investigating lexical replacements for arabic-english code-switched data augmentation. arXiv preprint arXiv:2205.12649. ↗
  5. Muhammad Abdul-Mageed, Chiyu Zhang, AbdelRahim Elmadany, Houda Bouamor, and Nizar Habash. 2022. Nadi 2022: The third nuanced arabic dialect identification shared task. arXiv preprint arXiv:2210.09582. ↗
  6. Salima Harrat, Karima Meftouh, and Kamel Smaïli. 2018. Maghrebi arabic dialect processing: an overview. Journal of International Science and General Applications, 1. find
  7. Roqayah Al-Ibrahim and Rehab M Duwairi. 2020. Neural machine translation from jordanian dialect to modern standard arabic. In 2020 11th International Conference on Information and Communication Systems (ICICS), pages 173–178. IEEE. find
  8. Amr Keleg, Sharon Goldwater, and Walid Magdy. 2023. Aldi: Quantifying the arabic level of dialectness of text. In The 2023 Conference on Empirical Methods in Natural Language Processing. find
  9. Hend Al-Khalifa, Tamer Elsayed, Hamdy Mubarak, Ab-dulmohsen Al-Thubaity, Walid Magdy, and Kareem Darwish. 2022. Proceedinsg of the 5th workshop on open-source arabic corpora and processing tools with shared tasks on qur’an qa and fine-grained hate speech detection. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection. find
  10. Laith H Baniata, Seyoung Park, and Seong-Bae Park. 2018. A neural machine translation model for arabic dialects that utilizes multitask learning (mtl). Computational Intelligence & Neuroscience. find
  11. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, et al. 2018. The madar arabic dialect corpus and lexicon. In Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018). find
  12. Mona Diab and Nizar Habash. 2007. Arabic dialect processing tutorial. In Proceedings of the human language technology conference of the NAACL, companion volume: tutorial abstracts, pages 5–6. find
  13. Heba Elfardy and Mona Diab. 2013. Sentence level dialect identification in arabic. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 456–461. find
  14. AbdelRahim Elmadany, Muhammad Abdul-Mageed, et al. 2022. Arat5: Text-to-text transformers for arabic language generation. In Proceedings of the 60th annual meeting of the association for computational linguistics (Volume 1: Long papers), pages 628–647. find our card
  15. Muhammad Abdul Mageed, Abdelrahim Elmadany, et al. 2021. Arbert & marbert: Deep bidirectional transformers for arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105. find our card
  16. Giovanni Molina, Fahad AlGhamdi, Mahmoud Ghoneim, Abdelati Hawwari, Nicolas Rey-Villamizar, Mona Diab, and Thamar Solorio. 2019. Overview for the second shared task on language identification in code-switched data. arXiv preprint arXiv:1909.13016. ↗
  17. Abdulhadi Shoufan and Sumaya Alameri. 2015. Natural language processing for dialectical arabic: A survey. In Proceedings of the second workshop on Arabic natural language processing, pages 36–48. find
  18. Thamar Solorio, Elizabeth Blair, Suraj Maharjan, Steven Bethard, Mona Diab, Mahmoud Ghoneim, Abdelati Hawwari, Fahad AlGhamdi, Julia Hirschberg, Alison Chang, et al. 2014. Overview for the first shared task on language identification in code-switched data. In Proceedings of the first workshop on computational approaches to code switching, pages 62–72. find our card
  19. Omar Zaidan and Chris Callison-Burch. 2011. The arabic online commentary dataset: an annotated dataset of informal arabic with high dialectal content. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 37–41. find our card
  20. Omar F Zaidan and Chris Callison-Burch. 2014. Arabic dialect identification. Computational Linguistics, 40(1):171–202. find our card
  21. Ashraf Elnagar, Sane M Yagi, Ali Bou Nassif, Ismail Shahin, and Said A Salloum. 2021. Systematic literature review of dialectal arabic: identification and detection. IEEE Access, 9:31010–31042. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nacar-2024-asos-nadi-2024-shared-task/