Elamur Research⌕
☰
home / papers / 2014 / Tharwa: A Large Scale Dialectal Arabic - Standard Arabic - English Lexicon

Tharwa: A Large Scale Dialectal Arabic - Standard Arabic - English Lexicon

Paper · 2014 · Mona Diab, Mohamed Al-Badrashiny, Maryam Aminian et al.

Summary

Dialectal Arabic (DA), and Egyptian Arabic (EGY) in particular, is common in online text. Most Arabic NLP tools, however, are built for Modern Standard Arabic (MSA) and perform much worse on dialects. DA and MSA differ in phonology, morphology and vocabulary. Many written forms look the same across varieties but mean different things. Before this work, no wide-coverage resource linked dialect, standard Arabic and English.

The authors built Tharwa, a lemma-based three-way EGY–MSA–English lexicon. It merges a paper dictionary (Hinds-Badawi), the LDC's ECAL, and the lexica behind the CALIMA and SAMA analyzers. Each EGY entry is written in the CODA standardized spelling and fully diacritized. Entries also list attested spelling variants, POS, gender, number, rationality, root and morphological pattern. Gaps were filled manually and semi-automatically, and several rounds of quality control were applied.

The result has over 73,000 EGY entries and is publicly available. It already helps downstream tools, namely the Elissa DA-to-MSA translation system and the AIDA dialect identifier. Tharwa is meant as a nucleus for adding other Arabic dialects.

Key points

  • Source resources:
    • BADAWI: 31,548 EGY–English entries.
    • ECAL: about 66K inflected EGY entries and about 36K lemmas, the same list CALIMA covers.
    • SAMA-LEX: about 41K MSA lemmas, reduced to about 37K once semantic id markers are dropped.
  • Lemma convention and tags:
    • Nominal lemmas use the 3rd person singular masculine form.
    • Verbal lemmas use the perfective 3rd person form.
    • The tagset has 34 POS tags, including a vbn tag for deverbal nouns.
  • EGY–MSA lexical relations fall into four types: identical, semantic cognates, homographs/homophones, and distinct words. Of the entries, 13.2% are homographs that carry extra EGY senses and 38.9% are completely distinct from MSA.
  • Elissa: Tharwa-based MSA paraphrasing raised Arabic→English BLEU from 37.2 to 38.1 on dev, and yielded an absolute improvement of 1.4 BLEU points on an unseen held-out test set.
  • AIDA: token/sentence dialect identification through MADAMIRA/CALIMA reached 76.2%. The majority baselines were 51.4% (MSA) and 45.6% (EGY).

Interesting ideas

  • I1. Each entry is keyed on one conventionalized spelling (CODA), and the naturally occurring spelling variants are stored next to it. Normalization stays explicit and reversible instead of silently erasing variation.
  • I2. Words shared between two language varieties are sorted into identical, regular cognate, false-friend homograph, or truly distinct. This shows exactly where naive transfer from one variety will mislead.
  • I3. Patterns are recorded at two levels: a deep morphological pattern and a surface morpho-phonological one. Every surface-form entry is also linked back to its lemma, which keeps each analysis traceable.
  • I4. Entries are checked from several independent angles: automatic consistency metrics, versioned edits, crowdsourcing, triangulation through parallel corpora, and native annotators.
  • I5. The standard-language equivalent is given in the same inflectional form as the dialect entry, so a translation can substitute words without re-inflecting them.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): The lemma lexicon in crate en could keep one canonical form plus attested variants (British/American, informal, misspelled), so the tagger can explain which variant it mapped.
  • I2 (tree): During UD dialect conversion between treebanks, label and relation correspondences could be classified the same way, flagging "same label, different meaning" cases for special rules.
  • I3 (tokens): English morphology could store the underlying morpheme pattern alongside its surface realization (e.g. consonant doubling before -ed), making each lemmatization step explainable.
  • I4 (learn): Lexical facts in the compiled knowledge could be accepted only when independent witnesses agree, reusing the silver-data agreement idea for the lexicon itself.
  • I5 (back): When generating or translating through the meta-world, aligned forms could be stored per inflection rather than per lemma, giving direct and explainable surface substitutions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Columbia UniversityNew York · United States254 works · 56,747 citations🎓George Washington UniversityWashington · United States51 works · 8,148 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,379 worksSemitic languages · 3,066 works

Cite

Cite as Mona Diab, Mohamed Al-Badrashiny, Maryam Aminian, Mohammed Attia, Heba Elfardy, Nizar Habash, Abdelati Hawwari, Wael Salloum, Pradeep Dasigi and Ramy Eskander. 2014. Tharwa: A Large Scale Dialectal Arabic - Standard Arabic - English Lexicon. Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{diab2014tharwa,
  title = {{Tharwa: A Large Scale Dialectal Arabic - Standard Arabic - English Lexicon}},
  author = {Mona Diab, Mohamed Al-Badrashiny, Maryam Aminian, Mohammed Attia, Heba Elfardy, Nizar Habash, Abdelati Hawwari, Wael Salloum, Pradeep Dasigi and Ramy Eskander},
  booktitle = {Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)},
  year = {2014},
  doi = {10.63317/2fmjsvgfookr},
  url = {https://doi.org/10.63317/2fmjsvgfookr}
}

References

25 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abo Bakr, H., Shaalan, K., and Ziedan, I. (2008). A Hybrid Approach for Converting Written Egyptian Colloquial Dialect into Diacritized Arabic. In The 6th International Conference on Informatics and Systems, IN-FOS2008. Cairo University. find
  2. Alkuhlani, S. and Habash, N. (2011). A Corpus for Modeling Morpho-Syntactic Agreement in Arabic: Gender, Number and Rationality. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL’11), Portland, Oregon, USA.
  3. Badawi, E.-S. and Hinds, M. (1986). A Dictionary of Egyptian Arabic. Librairie du Liban. find
  4. Brustad, K. (2000). The Syntax of Spoken Arabic: A Comparative Study of Moroccan, Egyptian, Syrian, and Kuwaiti Dialects. Georgetown University Press. find
  5. Buckwalter, T. (2004). Buckwalter Arabic Morphological Analyzer Version 2.0. LDC catalog number LDC2004L02, ISBN 1-58563-324-0. find
  6. Elfardy, H. and Diab, M. (2012). Token level identification of linguistic code switching. In Proceedings of the 24th International Conference on Computational Linguistics (COLING), IIT Mumbai, India. find
  7. Elfardy, H. and Diab, M. (2013). Sentence level dialect identification in Arabic. In Proceedings of ACL, Sofia, Bulgaria.
  8. Ferguson, C. F. (1959). Diglossia. Word, 15(2):325–340. find
  9. Gadalla, H., Kilany, H., Arram, H., Yacoub, A., El-Habashi, A., Shalaby, A., Karins, K., Rowson, E., Mac-Intyre, R., Kingsbury, P., Graff, D., and McLemore, C. (1997). CALLHOME Egyptian Arabic Transcripts. In Linguistic Data Consortium, Philadelphia. find
  10. Graff, D., Maamouri, M., Bouziri, B., Krouna, S., Kulick, S., and Buckwalter, T. (2009). Standard Arabic Morphological Analyzer (SAMA) Version 3.1. Linguistic Data Consortium LDC2009E73. find
  11. Habash, N., Soudi, A., and Buckwalter, T. (2007). On Arabic transliteration. In Soudi, A., Neumann, G., and van den Bosch, A., editors, Arabic Computational Morphology, volume 38 of Text, Speech and Language Technology, chapter 2, pages 15–22. Springer. ↗
  12. Habash, N., Eskander, R., and Hawwari, A. (2012a). A Morphological Analyzer for Egyptian Arabic. In NAACL-HLT 2012 Workshop on Computational Morphology and Phonology (SIGMORPHON2012), pages 1–9.
  13. Habash, N., Diab, M., and Rabmow, O. (2012b). Conventional Orthography for Dialectal Arabic. In Proceedings of the Language Resources and Evaluation Conference (LREC), Istanbul.
  14. Habash, N. (2010). Introduction to Arabic Natural Language Processing. Morgan & Claypool Publishers. ↗
  15. Kilany, H., Gadalla, H., Arram, H., Yacoub, A., El-Habashi, A., and McLemore, C. (2002). Egyptian Colloquial Arabic Lexicon. LDC catalog number LDC99L22. find
  16. Maamouri, M., Bies, A., Buckwalter, T., Diab, M., Habash, N., Rambow, O., and Tabessi, D. (2006). Developing and using a pilot dialectal Arabic treebank. In LREC, Genoa, Italy.
  17. Och, F. J. and Ney, H. (2000). Improved statistical alignment models. In ACL ’00: Proceedings of the 38th Annual Meeting on Association for Computational Linguistics, pages 440–447, Morristown, NJ, USA. Association for Computational Linguistics.
  18. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, PA.
  19. Pasha, A., Al-Badrashiny, M., Kholy, A. E., Eskander, R., Diab, M., Habash, N., Pooleery, M., Rambow, O., and Roth, R. (2014). Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of Arabic. In In Proceedings of LREC, Reykjavik, Iceland.
  20. Saleh, I. and Habash, N. (2009). Automatic extraction of lemma-based bilingual dictionaries for morphologically rich languages. In Third Workshop on Computational Approaches to Arabic Script-based Languages at the MT Summit XII, Ottawa, Canada. find
  21. Salloum, W. and Habash, N. (2011). Dialectal to Standard Arabic Paraphrasing to Improve Arabic-English Statistical Machine Translation. In Proceedings of the First Workshop on Algorithms and Resources for Modelling of Dialects and Language Varieties, pages 10–21, Edinburgh, Scotland. ↗
  22. Salloum, W. and Habash, N. (2013). Dialectal Arabic to English Machine Translation: Pivoting through Modern Standard Arabic. In The 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2013).
  23. Schmid, H. (1995). Treetagger, a language independent part-of-speech tagger. Technical report, Institut für Maschinelle Sprachverarbeitung, Universität Stuttgart. find
  24. Spiro, S. (1895). An Arabic-English Vocabulary of the Colloquial Arabic of Egypt. Al-Mokattam printing office. find
  25. Spiro, S. (1987). Arabic-English Dictionary of the Colloquial Arabic of Egypt. Librairie Du Liban. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/diab-2014-tharwa-large-scale-dialectal-arabic/