Elamur Research⌕
☰
home / papers / 2014 / A Multidialectal Parallel Corpus of Arabic

A Multidialectal Parallel Corpus of Arabic

Summary

Arabic dialects are the everyday spoken and online language for most speakers, but they lack standard spelling and official status. NLP resources for them are sparse and concentrate on a few prominent varieties such as Egyptian. Parallel text across several dialects does not arise naturally, so the authors built the first multidialectal Arabic parallel corpus. They took 2,000 Egyptian sentences from the Egyptian–English corpus of Zbib et al. (2012). Native speakers then rendered these sentences into Tunisian, Syrian, Jordanian and Palestinian, and an Egyptian translator produced Modern Standard Arabic. The existing English translations complete the set, giving seven aligned versions of each sentence. A first lexical comparison, using the Overlap Coefficient, matches dialectological expectations: the Levantine varieties are closest to each other, and Tunisian stands furthest apart. The analysis also exposes biases introduced by the pivot language and by translator background. The resource is intended for dialect identification and machine translation and is to be released freely for research.

Key points

  • Languages covered: MSA, English and a set of five dialects, namely Tunisian (TN), Egyptian (EG), Syrian (SY), Jordanian (JO) and Palestinian (PA). The last three represent the Levantine group.
  • Source sentences: Egyptian was the pivot because it is broadly understood across the Arab world, thanks to Egyptian film and TV. The 2,000 sentences come from the Zbib et al. (2012) data, whose English side was produced by non-professional MTurk workers.
  • Corpus statistics (a 1,000-sentence sample per variety):
    • Token counts range from about 9.9K (JO) to about 11.6K (SY).
    • Unique tokens range from 3,675 (PA) to 4,588 (EG).
    • Average sentence length is roughly 8–10 tokens.
  • Lexical overlap results:
    • MSA overlaps most with Egyptian (39.12) and only about 26–28 with the other dialects.
    • Tunisian, the only Western dialect, shares the least with the rest.
    • Syrian and Jordanian form the most similar pair.
  • Planned extensions: more sentences and dialects, plus extra annotation layers: a CODA version, morphological tokenization, POS tags and manual word alignments.

Interesting ideas

  • I1. Pivot from a widely understood variety. Starting from a variety every translator understands well (here Egyptian) makes multi-way parallel data feasible when no natural parallel text exists. The trade-off, which the authors measured, is that outputs drift toward the pivot. This suggests checking for pivot bias in any similar multi-variety collection.
  • I2. Supply a second-language gloss for context. Translators saw each sentence in isolation, so they also received its English counterpart to resolve ambiguous readings. This is a cheap fix whenever sentences are translated out of context.
  • I3. Ask for internal spelling consistency instead of a full standard. Where no agreed orthography exists, each translator was only asked to spell the same word the same way and to avoid Latin script. This limits data sparsity without the cost of writing and enforcing guidelines, and could suit other non-standardized or mixed varieties.
  • I4. Use the Overlap Coefficient as a quick similarity probe. Dividing shared vocabulary by the smaller vocabulary gives a simple, size-robust way to compare varieties before doing deeper analysis.
  • I5. Test whether a ranking survives normalization, and record translator metadata. The authors recomputed similarity after unifying Alif/Hamza, Alif-Maqsura and Ta-Marbuta spellings and found the ranking unchanged, so spelling noise did not drive their conclusions. They also traced anomalies to translator profiles (an Egyptian MSA translator, a Syrian translator living in Egypt). Both practices apply to any human-built comparative resource.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

80 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 74 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{bouamor2014multidialectal,
  title = {{A Multidialectal Parallel Corpus of Arabic}},
  author = {Bouamor, Houda and Habash, Nizar and Oflazer, Kemal},
  booktitle = {Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)},
  year = {2014},
  doi = {10.63317/5mf7vmsqm8co},
  url = {https://doi.org/10.63317/5mf7vmsqm8co}
}

References

28 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abu-Melhim, Abdel-Rahman. (1991). Code-switching and Linguistic Accommodation in Arabic. In Perspectives on Arabic Linguistics III: Papers from the Third Annual Symposium on Arabic Linguistics, volume 80, pages 231–250. John Benjamins Publishing. ↗
  2. Al-Sabbagh, Rania and Girju, Roxana. (2010). Mining the Web for the Induction of a Dialectical Arabic Lexicon. In LREC, Valetta, Malta. find
  3. Bassiouney, Reem. (2009). Arabic Sociolinguistics. Edinburgh University Press. find
  4. Benmamoun, Elabbas. (2012). Agreement and Cliticization in Arabic Varieties from Diachronic and Synchronic Perspectives. In Bassiouney, Reem, editor, Al’Arabiyya: Journal of American Association of Teachers of Arabic, volume 44-45, pages 137– 150. Georgetown University Press. find
  5. Boujelbane, Rahma, Ellouze Khemekhem, Mariem, and Belguith, Lamia Hadrich. (2013). Mapping Rules for Building a Tunisian Dialect Lexicon and Generating Corpora. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 419–428, Nagoya, Japan. find
  6. Brustad, Kristen. (2000). The Syntax of Spoken Arabic: A Comparative Study of Moroccan, Egyptian, Syrian, and Kuwaiti Dialects. Georgetown University Press. find
  7. Chiang, David, Diab, Mona, Habash, Nizar, Rambow, Owen, and Shareef, Safiullah. (2006). Parsing Arabic Dialects. In Proceedings of EACL, Trento, Italy. ↗ our card
  8. Darwish, Kareem. (2013). Arabizi Detection and Conversion to Arabic. CoRR. ↗
  9. Diab, Mona and Habash, Nizar. (2007). Arabic Dialect Processing Tutorial. In NAACL. find
  10. Elfardy, Heba and Diab, Mona. (2013). Sentence Level Dialect Identification in Arabic. In Proceedings of the Association for Computational Linguistics, pages 456–461, Sofia, Bulgaria. find
  11. Habash, Nizar, Soudi, Abdelhadi, and Buckwalter, Timothy. (2007). On Arabic transliteration. In Soudi, Abdelhadi, Neumann, Guenter, and van den Bosch, Antal, editors, Arabic Computational Morphology, volume 38 of Text, Speech and Language Technology, chapter 2, pages 15–22. Springer. ↗
  12. Habash, N., Rambow, O., Diab, M., and Kanjawi-Faraj, R. (2008). Guidelines for Annotation of Arabic Dialectness. In Proceedings of the LREC Workshop on HLT & NLP within the Arabic world. find
  13. Habash, Nizar, Diab, Mona, and Rambow, Owen. (2012a). Conventional Orthography for Dialectal Arabic. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012), pages 711–718, Istanbul, Turkey. find our card
  14. Habash, Nizar, Eskander, Ramy, and Hawwari, Abdelati. (2012b). A Morphological Analyzer for Egyptian Arabic. In Proceedings of the Twelfth Meeting of the Special Interest Group on Computational Morphology and Phonology, pages 1–9, Montréal, Canada. find our card
  15. Habash, Nizar, Roth, Ryan, Rambow, Owen, Eskander, Ramy, and Tomeh, Nadi. (2013). Morphological analysis and disambiguation for dialectal arabic. In Proceedings of NAACL-HLT, pages 426–432, Atlanta, Georgia, June. find
  16. Habash, Nizar. (2006). On Arabic and its Dialects. Multilingual Magazine, 17(81). find
  17. Habash, Nizar Y. (2010). Introduction to Arabic natural language processing, volume 3. Morgan & Claypool Publishers. find
  18. Haeri, Niloofar. (1991). Sociolinguistic Variation in Cairene Arabic: Palatalization and the qaf in the Speech of Men and Women. find
  19. Pasha, Arfath, Al-Badrashiny, Mohamed, Kholy, Ahmed El, Eskander, Ramy, Diab, Mona, Habash, Nizar, Pooleery, Manoj, Rambow, Owen, and Roth, Ryan. (2014). Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of arabic. In In Proceedings of LREC, Reykjavik, Iceland. find our card
  20. Riesa, Jason and Yarowsky, David. (2006). Minimally Supervised Morphological Segmentation with Applications to Machine Translation. In Proceedings of AMTA, Cambridge, MA. find
  21. Sajjad, Hassan, Darwish, Kareem, and Belinkov, Yonatan. (2013). Translating dialectal arabic to english. In Proceedings of ACL, Sofia, Bulgaria. find
  22. Salloum, Wael and Habash, Nizar. (2013). Dialectal arabic to english machine translation: Pivoting through modern standard arabic. In Proceedings of NAACL-HLT, Atlanta, Georgia. find
  23. Sawaf, Hassan. (2010). Arabic dialect handling in hybrid machine translation. In Proceedings of AMTA, Denver, Colorado. find
  24. Watson, Janet CE. (2007). The Phonology and Morphology of Arabic. Oxford University Press. find
  25. Zaidan, Omar F. and Callison-Burch, Chris. (2011). Crowdsourcing Translation: Professional Quality from Non-Professionals. In Proceedings of ACL, Portland, Oregon, USA. find
  26. Zaidan, Omar and Callison-Burch, Chris. (2013). Arabic dialect identification. Computational Linguistics. ↗ our card
  27. Zbib, Rabih, Malchiodi, Erika, Devlin, Jacob, Stallard, David, Matsoukas, Spyros, Schwartz, Richard, Makhoul, John, Zaidan, Omar F., and Callison-Burch, Chris. (2012). Machine translation of arabic dialects. In Proceedings of NAACL-HLT, Montréal, Canada. find
  28. Zribi, I., Boujelbane, R., Masmoudi, A., Ellouze Khmekhem, M., Hadrich Belguith, L., and Habash, N. (2014). A Conventional Orthography for Tunisian Arabic. In In Proceedings of LREC, Reykjavik, Iceland. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bouamor-2014-multidialectal-parallel-corpus-arabic/