Elamur Research⌕
☰
home / papers / 2015 / DIWAN: A Dialectal Word Annotation Tool for Arabic

DIWAN: A Dialectal Word Annotation Tool for Arabic

Paper · 2015 · Faisal Al-Shargi and Owen Rambow

Summary

Arabic dialects have few annotated resources. Tools built for Modern Standard Arabic (MSA) perform poorly on dialectal text, which often lacks spelling norms and mixes freely with MSA. The authors present DIWAN, a Windows desktop tool for word-level morphological annotation of dialectal Arabic. Its main design idea is that dialects share much of their morphology and lexicon with MSA and with each other, so the tool should not treat each one as an unrelated language. DIWAN preloads each token with a contextual analysis from MADAMIRA and lets the annotator browse candidate readings from SAMA (MSA) and CALIMA-EGY (Egyptian). The annotator can accept, edit, borrow or build analyses from scratch. Annotations record a conventionalized CODA spelling alongside the original orthography and keep the text order. The resulting data can therefore train both context-free morphological analyzers and in-context taggers. The tool has been used for Palestinian, Yemeni and Moroccan corpora. The authors do not report any controlled evaluation of annotation speed or quality.

Key points

  • The tool builds on three existing resources: SAMA 3.1 (MSA analyzer), CALIMA-EGY (Egyptian analyzer) and MADAMIRA (contextual tagger for MSA and Egyptian).
  • Words can be queued for annotation in three ways:
    • by frequency;
    • by text order;
    • by "coverage", which puts first the words MADAMIRA cannot analyze, since these are likely dialect-specific.
  • Each annotated token in the output records:
    • CODA spelling, lexeme, Buckwalter morpheme split and POS;
    • MSA equivalent and English gloss;
    • functional gender and number;
    • dialect subregion;
    • provenance: which resource was used, whether it was modified, and the search key.
  • Corpus sizes:
    • Palestinian: 45,000 annotated tokens; annotators became fluent with the tool after about 100 words.
    • Yemeni (Sana'ani): 32,325 tokens.
    • Moroccan: 64,171 words collected.
  • Roles are split between administrators and annotators. Data is stored in three tables (D_sentences, D_madamira, D_result), and annotators can work offline with batch upload later.

Interesting ideas

  • I1. A "coverage" queue puts tokens that existing analyzers cannot handle at the front, which steers human effort toward truly new material first.
  • I2. Analysis by substitution: when a word gets no analysis, the annotator looks up a known word with the same inflectional pattern and swaps only the stem and gloss. This turns a hard from-scratch task into a quick edit.
  • I3. Each annotation carries provenance fields: which source supplied it, whether a human changed it, and what query was used to retrieve it. This lets you later measure how often each source was right.
  • I4. A normalized spelling is stored alongside the raw orthography instead of replacing it, so one dataset serves both noisy-input and canonical-form models.
  • I5. The workflow starts with frequency-ordered annotation to cover many tokens quickly, then switches to text order to finish the long tail in context.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Rank corpus words that the en morphology cannot analyze (slang, neologisms, domain terms) as a worklist for extending the lexicon.
  • I2 (tokens): For unknown English words, copy the morphological analysis of a known analogue with the same affix pattern and swap in the new lemma, keeping the borrowing explainable.
  • I3 (learn): Tag every silver token or arc with the witness parser or LLM tier that produced it and a "human-modified" flag, so the reliability of each source can be audited.
  • I4 (tokens): Keep the raw surface token and a normalized form side by side for social-media or misspelled input, letting the tagger train on one and tests check the other.
  • I5 (learn): Schedule teacher or human correction rounds by fixing high-frequency error types first, then sweeping through full documents for the remaining contextual cases.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Columbia UniversityNew York · United States241 works · 54,053 citations🎓Leipzig UniversityLeipzig · Germany26 works · 8,622 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksMorphology · 405 works

Cite

Cite as Faisal Al-Shargi and Owen Rambow. 2015. DIWAN: A Dialectal Word Annotation Tool for Arabic. Proceedings of the Second Workshop on Arabic Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w15-3206

@inproceedings{alshargi2015diwan,
  title = {{DIWAN: A Dialectal Word Annotation Tool for Arabic}},
  author = {Faisal Al-Shargi and Owen Rambow},
  booktitle = {Proceedings of the Second Workshop on Arabic Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/w15-3206},
  url = {https://doi.org/10.18653/v1/w15-3206}
}

References

15 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Y. Benajiba and M. Diab. 2010. A web application for dialectal Arabic text annotation. In Proceedings of the LREC Workshop for Language Resources (LRs) and Human Language Technologies (HLT) for Semitic Languages: Status, Updates, and Prospects. find
  2. T. Buckwalter. 2004. Buckwalter Arabic Morphological Analyzer Version 2.0. Linguistic Data Consortium, University of Pennsylvania. LDC Catalog No.: LDC2004L02, ISBN 1-58563-324-0 find
  3. D. Chiang, M. Diab, N. Habash, O. Rambow, and S. Shareef. 2006. Parsing Arabic Dialect. In Proceedings of the European chapter of the Association of Computational Linguistics (EACL). find
  4. M. Diab, N. Habash, O. Rambow, M. AlTantawy, and Y. Benajiba. 2010a. COLABA: Arabic Dialect Annotation and Processing. In Proceedings of the Language Resources (LRs) and Human Language Technologies (HLT) for Semitic Languages at LREC. find
  5. M. Diab, K. Hacioglu, and D. Jurafsky. 2007. Automated Methods for Processing Arabic Text: From Tokenization to Base Phrase Chunking. In van den Bosch, A. and Soudi, A., editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods. Kluwer/Springer. find
  6. D. Graff, M. Maamouri, B. Bouziri, S. Krouna, S. Kulick, and T. Buckwalter. 2009. Standard Arabic Morphological Analyzer (SAMA) Version 3.1. Linguistic Data Consortium LDC2009E73. find
  7. N. Habash and O. Rambow. 2005. Arabic Tokenization, Part-of-Speech Tagging and Morphological Disambiguation in One Fell Swoop. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 573–580, Ann Arbor, Michigan.
  8. N. Habash and O. Rambow. 2006. MAGEAD: A Morphological Analyzer and Generator for the Arabic Dialects. In Proceedings of ACL, Sydney, Australia. ↗
  9. N. Habash, M. Diab, and O. Rabmow. 2012a. Conventional Orthography for Dialectal Arabic. In Proceedings of the Language Resources and Evaluation Conference (LREC), Istanbul.
  10. N. Habash, R. Eskander, and A. Hawwari. 2012b. A Morphological Analyzer for Egyptian Arabic. In Proc. of the Special Interest Group on Computational Morphology and Phonology, Montréal, Canada.
  11. N. Habash, O. Rambow, and R. Roth. 2009. MADA+TOKAN: A toolkit for Arabic tokenization, diacritization, morphological disambiguation, POS tagging, stemming and lemmatization. In Choukri, K. and Maegaard, B., editors, Proceedings of the Second International Conference on Arabic Language Resources and Tools. The MEDAR Consortium, April. find
  12. N. Habash, R. Roth, O. Rambow, R. Eskander, and N. Tomeh. 2013. Morphological Analysis and Disambiguation for Dialectal Arabic. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), Atlanta, GA.
  13. M. Jarrar, N. Habash, D. Akra, and N. Zalmout 2014. "Building a Corpus for Palestinian Arabic: a Preliminary Study." In Proceedings of the Workshop on Arabic Natural Language Processing (ANLP 2014).
  14. M. Maamouri, A. Bies, S. Kulick, M. Ciul, N. Habash and R. Eskander. 2014. Developing a dialectal Egyptian Arabic Treebank: Impact of Morphology and Syntax on Annotation and Tool Development. In Proceedings of the Lan- guage Resources and Evaluation Conference (LREC), Reykjavik, Iceland. find
  15. A. Pasha, M. Al-Badrashiny, M. Diab, A. El Kholy, R. Eskander, N. Habash, M. Pooleery, O. Rambow and R. M. Roth. MADAMIRA: A Fast, Comprehensive Tool for Morphological Analysis and Disambiguation of Arabic. In Proc. of LREC, Reykjavik, Iceland, 2014.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shargi-2015-diwan-dialectal-word-annotation-tool/