Elamur Research⌕
☰
home / papers / 2012 / Elissa: A Dialectal to Standard Arabic Machine Translation System

Elissa: A Dialectal to Standard Arabic Machine Translation System

Paper · 2012 · Wael Salloum and Nizar Habash

Summary

Dialectal Arabic (DA), meaning the spoken and orthographically unstandardized varieties, has very few NLP resources. Modern Standard Arabic (MSA), by contrast, is well served by analyzers, annotated data and parallel corpora. Elissa 1.0 addresses this gap by rewriting dialectal sentences as MSA, so that existing MSA tools and MT systems can handle them. The system is mainly rule-based, with some statistical components. It works in three steps:

  • Selection: it decides which words need attention, such as dialectal words, OOV words, dictionary entries or words the user has marked.
  • Translation: it produces MSA alternatives in two ways. One is a direct dictionary lookup. The other is a classic analysis, transfer and generation chain built on the ADAM dialect analyzer, expert-written transfer rules and the TOKAN generator.
  • Language modeling: all candidates are placed in a lattice, and an MSA language model ranks them.

Elissa covers Levantine, Egyptian and Iraqi Arabic, and Gulf Arabic to a lesser degree. The paper reports no formal evaluation. A worked Levantine example shows that passing Elissa's MSA output to Google Translate gives much more usable English than translating the dialect directly. The tool is publicly released as a preprocessor for dialectal input.

Key points

  • The authors cite prior work (Habash and Rambow, 2006) showing that an MSA morphological analyzer fails on more than a third of Levantine verbs. This motivates a dedicated DA→MSA bridge.
  • Transfer uses two rule types:
    • lemma-to-lemma (L2L) rules, drawn from DA-MSA dictionaries such as Tharwa and in-house lists;
    • features-to-features (F2F) rules, written by experts, which can change clitics and aspect, or split one DA word into several MSA words (for example, circumfix negation becomes a separate negation word).
  • Ranking uses the SRILM lattice-tool with a default 5-gram LM trained on about 200M untokenized Arabic words. Users can replace it with their own LM.
  • Input must be raw (untokenized) text in UTF-8 or Buckwalter. There are three output types: top-1 sentence, configurable n-best, and a map file that lists source word, transfer weight and target phrase.
  • There are five selection modes:
    • user token tags (/DIA), which allow dialect identifiers such as AIDA to plug in;
    • user word lists (OOV or INV);
    • ADAM-based detection of dialect-only or DA/MSA-ambiguous words;
    • dictionary hits;
    • all words.

Interesting ideas

  • I1. Map a non-standard variety into the standard one and then reuse the mature standard tools unchanged, instead of building a separate pipeline for every variety.
  • I2. Do transfer in a lemma-plus-features space rather than on surface strings, and allow one source word to expand into several target words. A small set of rules then generalizes across all inflected forms.
  • I3. Only rewrite the words a configurable selector flags, and let outside tools mark tokens explicitly. This limits damage to text that is already fine.
  • I4. Keep every alternative in a lattice and postpone the decision to a later scorer. Store the transfer weight separately from the LM score, as in the map file.
  • I5. Evaluate a normalizer extrinsically by comparing downstream output in three conditions: raw input, human-normalized input and system-normalized input.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): a normalization layer could rewrite informal or dialectal English forms ("gonna", "ain't", nonstandard spellings) into standard lemma+feature analyses before tagging, so later stages see familiar input.
  • I2 (back): generation from the meaning graph could build lemma+feature bundles first and leave inflection and splitting to a separate morphological generator, which keeps each choice traceable.
  • I3 (learn): self-induced repair rules could fire only on tokens flagged by disagreement among witness parsers, so the repairs stay local and auditable.
  • I4 (tokens): ambiguous morphological analyses could be passed forward as a weighted set with their origin recorded, so the parser can decide and the explanation shows both evidence sources.
  • I5 (learn): run the same three-way comparison on answer accuracy, using the gold-normalized run as an upper bound and the raw run as a negative control.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Columbia UniversityNew York · United States241 works · 54,053 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Machine translation · 1,087 worksSemitic languages · 2,908 works

Cite

Cite as Wael Salloum and Nizar Habash. 2012. Elissa: A Dialectal to Standard Arabic Machine Translation System. Proceedings of COLING 2012: Demonstration Papers.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{salloum2012elissa,
  title = {{Elissa: A Dialectal to Standard Arabic Machine Translation System}},
  author = {Wael Salloum and Nizar Habash},
  booktitle = {Proceedings of COLING 2012: Demonstration Papers},
  year = {2012},
  url = {https://aclanthology.org/C12-3048/}
}

References

23 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abo Bakr, H., Shaalan, K., and Ziedan, I. (2008). A Hybrid Approach for Converting Written Egyptian Colloquial Dialect into Diacritized Arabic. In The 6th International Conference on Informatics and Systems, INFOS2008. Cairo University. find
  2. Buckwalter, T. (2004). Buckwalter arabic morphological analyzer version 2.0. LDC catalog number LDC2004L02, ISBN 1-58563-324-0. find
  3. Chiang, D., Diab, M., Habash, N., Rambow, O., and Shareef, S. (2006). Parsing Arabic Dialects. In Proceedings of the European Chapter of ACL (EACL). ↗
  4. Diab, M., Hacioglu, K., and Jurafsky, D. (2007). Arabic Computational Morphology: Knowledge-based and Empirical Methods, chapter Automated Methods for Processing Arabic Text: From Tokenization to Base Phrase Chunking. Springer. find
  5. Diab, M., Hawwari, A., Elfardy, H., Dasigi, P., Al-Badrashiny, M., Eskander, R., and Habash, N. (Forthcoming – 2013). Tharwa: A multi-dialectal multi-lingual machine readable dictionary. find
  6. Duh, K. and Kirchhoff, K. (2005). POS tagging of dialectal Arabic: a minimally supervised approach. In Proceedings of the ACL Workshop on Computational Approaches to Semitic Languages, Semitic ’05, pages 55–62, Ann Arbor, Michigan.
  7. Elfardy, H. and Diab, M. (2012). Token level identification of linguistic code switching. In Proceedings of the 24th International Conference on Computational Linguistics (COLING), IIT Mumbai, India. find
  8. Habash, N. (2007). Arabic Morphological Representations for Machine Translation. In van den Bosch, A. and Soudi, A., editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods. Springer. ↗
  9. Habash, N. (2010). Introduction to Arabic Natural Language Processing. Morgan & Claypool Publishers. ↗
  10. Habash, N., Eskander, R., and Hawwari, A. (2012). A morphological analyzer for egyptian arabic. In Proceedings of the Twelfth Meeting of the Special Interest Group on Computational Morphology and Phonology, pages 1–9, Montréal, Canada.
  11. Habash, N. and Rambow, O. (2005). Arabic Tokenization, Part-of-Speech Tagging and Morphological Disambiguation in One Fell Swoop. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 573–580, Ann Arbor, Michigan.
  12. Habash, N. and Rambow, O. (2006). MAGEAD: A Morphological Analyzer and Generator for the Arabic Dialects. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 681–688, Sydney, Australia. ↗
  13. Habash, N., Soudi, A., and Buckwalter, T. (2007). On Arabic Transliteration. In van den Bosch, A. and Soudi, A., editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods. Springer. ↗
  14. Hajič, J., Hric, J., and Kubon, V. (2000). Machine Translation of Very Close Languages. In Proceedings of the 6th Applied Natural Language Processing Conference (ANLP’2000), pages 7–12, Seattle. find
  15. Kilany, H., Gadalla, H., Arram, H., Yacoub, A., El-Habashi, A., and McLemore, C. (2002). Egyptian Colloquial Arabic Lexicon. LDC catalog number LDC99L22. find
  16. Kirchhoff, K., Bilmes, J., Das, S., Duta, N., Egan, M., Ji, G., He, F., Henderson, J., Liu, D., Noa-many, M., Schone, P., Schwartz, R., and Vergyri, D. (2003). Novel Approaches to Arabic Speech Recognition: Report from the 2002 Johns Hopkins Summer Workshop. In IEEE International Conference on Acoustics, Speech, and Signal Processing, Hong Kong, China. find
  17. Kumar, S., Och, F. J., and Macherey, W. (2007). Improving word alignment with bridge languages. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 42–50, Prague, Czech Republic. find
  18. Salloum, W. and Habash, N. (2011). Dialectal to Standard Arabic Paraphrasing to Improve Arabic-English Statistical Machine Translation. In Proceedings of the First Workshop on Algorithms and Resources for Modelling of Dialects and Language Varieties, pages 10–21, Edinburgh, Scotland. ↗
  19. Sawaf, H. (2010). Arabic dialect handling in hybrid machine translation. In Proceedings of the Conference of the Association for Machine Translation in the Americas (AMTA), Denver, Colorado.
  20. Stolcke, S. (2002). Tokenization, morphological analysis, and part-of-speech tagging for arabic in one fell swoop. In In Proceedings of ICSLP 2002, volume 2, pages 901–904. find
  21. Utiyama, M. and Isahara, H. (2007). A comparison of pivot methods for phrase-based statistical machine translation. In HLT-NAACL, pages 484–491.
  22. Zbib, R., Malchiodi, E., Devlin, J., Stallard, D., Matsoukas, S., Schwartz, R. M., Makhoul, J., Zaidan, O., and Callison-Burch, C. (2012). Machine translation of arabic dialects. In HLT-NAACL, pages 49–59.
  23. Zhang, X. (1998). Dialect MT: a case study between Cantonese and Mandarin. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, ACL ’98, pages 1460–1464, Montreal, Canada. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/salloum-2012-elissa-dialectal-standard-arabic-machine/