Elamur Research⌕
☰
home / papers / 2024 / ASOS at OSACT6 Shared Task: Investigation of Data Augmentation in Arabic Dialect-MSA Translation

ASOS at OSACT6 Shared Task: Investigation of Data Augmentation in Arabic Dialect-MSA Translation

Summary

This paper presents the ASOS team's submission to the OSACT 2024 shared task, which involved converting dialectal Arabic (Gulf, Egyptian, Levantine, Iraqi, Maghrebi) into Modern Standard Arabic. The main obstacle is the lack of data: the organizers supplied only about 200 parallel sentences per dialect.

To enlarge this seed set, the authors prompted GPT-3.5 and GPT-4 through the OpenAI API to write additional MSA renderings. The prompts asked the models to preserve meaning, respect MSA grammar and stay close to the source's word count. The team checked the synthetic pairs with METEOR and with GPT-4 acting as a judge, after showing that BLEU can collapse to zero over a trivial punctuation difference.

The enlarged data was used to fine-tune AraT5 V2, a T5-style Arabic sequence-to-sequence model, in several experimental configurations, one of which drew on the MADAR corpus. The best run scored 85.5 BLEU on validation but only 22.6 on the blind test. The authors attribute this gap to distribution and domain mismatch. The paper is a useful warning that LLM-generated training data can inflate in-distribution scores without comparable gains on unseen text.

Key points

  • Task: OSACT 2024 Dialect-to-MSA shared task covering five dialect groups. The blind test held 500 unseen sentences per dialect, and the official metric was BLEU.
  • Augmentation: GPT-3.5 and GPT-4 expanded the 200 seed sentences per dialect. A system message cast each model as a dialect-to-MSA translator focused on accuracy, grammar and length matching.
  • Quality of synthetic data: average METEOR against the gold MSA targets was 73.22% for GPT-4 outputs and 67.48% for GPT-3.5 outputs.
  • Model: AraT5 V2 fine-tuned on the augmented data reached 85.5 BLEU on validation versus 22.6 on the blind test. Combining GPT-3.5 and GPT-4 data (experiments 7–8) raised validation BLEU but gave no clear test gain.
  • Error analysis: GPT-4 rated translations along lexical, syntactic, semantic and orthographic dimensions. The MADAR-based run (experiment 3) was observed to drift from the source meaning more often.

Interesting ideas

  • I1. When generating synthetic targets with an LLM, the prompt also asks the output to keep roughly the source's word count, which discourages free rewriting and keeps pairs closely aligned. — For Mova (learn): LLM teacher tiers producing paraphrase silver data could carry a length/structure constraint, so witness parsers' trees on source and paraphrase stay token-alignable for agreement checks.
  • I2. Before trusting a metric, the authors expose its failure with a minimal case: an otherwise identical sentence that differs by one question mark gets zero BLEU. They then switch to a stem- and synonym-aware score. — For Mova (back): evaluation of the text → graph → text compressor should include such minimal-difference probes and use punctuation-tolerant, lemma-aware matching rather than raw n-gram overlap.
  • I3. The large gap between validation and blind-test scores shows that data generated from the same small seed mostly teaches the seed's own distribution. — For Mova (learn): keep a held-out evaluation set that no teacher-generated or self-play data was ever derived from, and treat it as a negative control against inflated scores.
  • I4. Errors are broken down into lexical, syntactic, semantic and orthographic categories instead of a single score, which makes each model's specific weakness visible. — For Mova (back): translation through the meta-world could log errors under the same four headings, pointing to whether a fault lies in lemmas, tree, frames or surface spelling.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Semitic languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{nacar2024asos,
  title = {{ASOS at OSACT6 Shared Task: Investigation of Data Augmentation in Arabic Dialect-MSA Translation}},
  author = {Nacar, Omer and Alharbi, Abdullah and Sibaee, Serry and Ahmed, Samar and Ghouti, Lahouari and Koubaa, Anis},
  booktitle = {Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation @ LREC-COLING 2024},
  year = {2024},
  doi = {10.63317/34u3y58xvz4e},
  url = {https://doi.org/10.63317/34u3y58xvz4e}
}

References

12 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Roqayah Al-Ibrahim and Rehab M Duwairi. 2020. Neural machine translation from jordanian dialect to modern standard arabic. In 2020 11th International Conference on Information and Communication Systems (ICICS), pages 173–178. IEEE. find
  2. Fatimah Alzamzami and Abdulmotaleb El Saddik. 2023. Osn-mdad: Machine translation dataset for arabic multi-dialectal conversations on online social media. arXiv preprint arXiv:2309.12137. ↗ our card
  3. Kareem Darwish, Nizar Habash, Mourad Abbas, Hend Al-Khalifa, Huseein T Al-Natsheh, Houda Bouamor, Karim Bouzoubaa, Violetta Cavalli-Sforza, Samhaa R El-Beltagy, Wassim El-Hajj, et al. 2021. A panoramic survey of natural language processing in the arab world. Communications of the ACM, 64(4):72–81. find
  4. Youness Moukafih, Nada Sbihi, Mounir Ghogho, and Kamel Smaïli. 2021. Improving machine translation of arabic dialects through multi-task learning. In International Conference of the Italian Association for Artificial Intelligence, pages 580– 590. Springer. find our card
  5. El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2022. AraT5: Text-to-text transformers for Arabic language generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 628–647, Dublin, Ireland. Association for Computational Linguistics. find our card
  6. Mahmoud Ali Qudah et al. 2017. A sociolinguistic study: Diglossia in social media. In Conference Proceedings. Innovation in Language Learning 2017. find
  7. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551. find our card
  8. Hassan Sajjad, Ahmed Abdelali, Nadir Durrani, and Fahim Dalvi. 2020. Arabench: Benchmarking dialectal arabic-english machine translation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5094–5107. find
  9. Mohamed Ali Sghaier and Mounir Zrigui. 2020. Rule-based machine translation from tunisian dialect to modern standard arabic. Procedia Computer Science, 176:310–319. find
  10. Connor Shorten, Taghi M Khoshgoftaar, and Borko Furht. 2021. Text data augmentation for deep learning. Journal of big Data, 8:1–34. find
  11. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934. 8. ↗ our card
  12. Language Resource References Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, et al. 2018. The madar arabic dialect corpus and lexicon. In Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018). find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nacar-2024-asos-osact6-shared-task-investigation/