Elamur Research⌕
☰
home / papers / 2022 / Lisan: Yemeni, Iraqi, Libyan, and Sudanese Arabic Dialect Copora with Morphological Annotations

Lisan: Yemeni, Iraqi, Libyan, and Sudanese Arabic Dialect Copora with Morphological Annotations

Summary

Arabic dialects are widely used online but have few annotated resources and no standard spelling, which makes them hard for NLP systems. This paper presents Lisan, a collection of four dialect corpora (Yemeni, Iraqi, Libyan, Sudanese) totalling about 1.25 million tokens. Every token is manually annotated with morphology. The Yemeni portion was gathered automatically through the Twitter API and then filtered so that each tweet contains a distinctive colloquial word. The other three corpora were hand-picked by native speakers from Facebook and YouTube posts and comments. Each token is split into prefixes, stem and suffixes and receives a POS tag, a lemma and an English gloss. The tags follow SAMA/BAMA so the data stays compatible with Curras and other Arabic corpora. Thirty-five trained native speakers did the work over two years using ADAT, a new collaborative annotation tool that proposes candidate analyses. The paper reports corpus statistics but no inter-annotator agreement yet. The corpora and the tool are released for academic research.

Key points

  • Corpus sizes are Yemeni 1,098,222 tokens (38,819 tweets), Iraqi 45,881, Libyan 51,686 and Sudanese 52,616, for a total of 1,248,405 tokens across about 48K documents.
  • The manually collected items had to be 10–30 words long and contain at least one word from the target dialect. No sub-dialect was favoured.
  • Each token is annotated as the tuple ⟨rawToken, Token, Prefixes, Stem, Suffixes, POS, Lemma, Gloss⟩ using LDC SAMA/BAMA tags.
  • Annotator training had several parts:
    • a roughly 15-hour workshop, including a guided 200-token exercise;
    • a 1,000-word quiz used to screen candidates;
    • pay of 5–10 USD per hour.
  • Lemmas came from SAMA, Birzeit's lexicographic database or the Arabic Ontology. When none fit, a new lemma was created and linked to its MSA equivalent and given senses.

Interesting ideas

  • I1. Applying CODA-style spelling standards proved too hard to teach a large annotator team, so the authors used three light rules instead: fix typos, collapse letters repeated more than twice, and map rare spellings to common ones. The raw form is kept alongside the normalized one. — For Mova (tokens): Keep both a raw and a normalized token for noisy English input such as "soooo" or typos, so the tagger sees clean forms while explanations can still point to the original text.
  • I2. In ADAT, an annotator labels a word once and then ticks the other sentences where it carries the same meaning, so one decision is reused across many contexts without forcing a single sense on all of them. — For Mova (learn): When reviewing silver data, group occurrences of an ambiguous word and let one verified analysis propagate only to the contexts a reviewer confirms, cutting review cost while avoiding blind sense merging.
  • I3. The tool offers candidate analyses drawn from existing lexicons, an older corpus and earlier annotations, which the authors found reduces typing mistakes and improves consistency. — For Mova (tokens): Pre-fill lemma and morphology choices from the en lexicon and prior agreed annotations, so manual corrections become selections rather than free-form edits.
  • I4. Each annotation carries a self-reported confidence level (High, Normal or Low), and uncertain cases can be referred to a more experienced annotator. — For Mova (learn): Attach confidence to silver labels and send low-confidence items to a higher LLM teacher tier or a human, instead of treating all training data as equally reliable.
  • I5. Dialect lemmas are linked to their standard-language equivalents, turning variant vocabulary into synonym links rather than isolated entries. — For Mova (world): Map regional or informal English lemmas (e.g., British vs. American terms) to a canonical WordNet/Wikidata concept, so the world model treats them as the same entity type.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

3 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 2 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2212.06468 · arXiv 2212.06468

@misc{jarrar2022lisan,
  title = {{Lisan: Yemeni, Iraqi, Libyan, and Sudanese Arabic Dialect Copora with Morphological Annotations}},
  author = {Jarrar, Mustafa and Zaraket, Fadi A and Hammouda, Tymaa and Alavi, Daanish Masood and Waahlisch, Martin},
  year = {2022},
  doi = {10.48550/arxiv.2212.06468},
  eprint = {2212.06468},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2212.06468}
}

References

24 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020. NADI 2020: The first nuanced Arabic dialect identification shared task. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 97–110, Barcelona, Spain (Online). Association for Computational Linguistics. find
  2. Muhammad Abdul-Mageed, Chiyu Zhang, Abdel-Rahim Elmadany, Houda Bouamor, and Nizar Habash. 2021. NADI 2021: The second nuanced Arabic dialect identification shared task. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 244–259, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. find
  3. Moustafa Al-Hajj and Mustafa Jarrar. 2021. Lu-bzu at semeval-2021 task 2: Word2vec and lemma2vec performance in arabic word-in-context disambiguation. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 748–755, Online. Association for Computational Linguistics. find
  4. Faisal Alshargi, Shahd Dibas, Sakhar Alkhereyf, Reem Faraj, Basmah Abdulkareem, Sane Yagi, Ouafaa Kacha, Nizar Habash, and Owen Rambow. 2019. Morphologically annotated corpora for seven Arabic dialects: Taizi, sanaani, najdi, jordanian, syrian, iraqi and Moroccan. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 137–147, Florence, Italy. Association for Computational Linguistics. find our card
  5. Houda Bouamor, Nizar Habash, and Kemal Oflazer. 2014. A multidialectal parallel corpus of Arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 1240–1245, Reykjavik, Iceland. European Language Resources Association (ELRA). find our card
  6. Alexandra Canavan, George Zipperlen, and David Graff. 1997. Callhome egyptian arabic speech. LDC97S45. find
  7. Kareem Darwish, Nizar Habash, Mourad Abbas, Hend Al-Khalifa, Huseein T. Al-Natsheh, Houda Bouamor, Karim Bouzoubaa, Violetta Cavalli-Sforza, Samhaa R. El-Beltagy, Wassim El-Hajj, Mustafa Jarrar, and Hamdy Mubarak. 2021. A panoramic survey of natural language processing in the arab worlds. Commun. ACM, 64(4):72–81. find
  8. Mona Diab, Nizar Habash, Owen Rambow, Mohamed Altantawy, and Yassine Benajiba. 2010. Colaba: Arabic dialect annotation and processing. LREC Workshop on Semitic Language Processing, pages 66–74. find
  9. Sana Ghanem, Mustafa Jarrar, Radi Jarrar, and Ibrahim Bounhas. 2023. A benchmark and scoring algorithm for enriching arabic synonyms. In Proceedings of the 12th International Global Wordnet Conference (GWC2023), pages 215–222. Global Wordnet Association. find
  10. Elisa Gugliotta and Marco Dinarelli. 2022. TArC: Tunisian Arabish corpus, first complete release. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1125–1136, Marseille, France. European Language Resources Association. find our card
  11. Nizar Habash, Ramy Eskander, and Abdelati Hawwari. 2012. A morphological analyzer for Egyptian Arabic. In Proceedings of the Twelfth Meeting of the Special Interest Group on Computational Morphology and Phonology, pages 1–9, Montréal, Canada. Association for Computational Linguistics. find our card
  12. Karim El Haff, Mustafa Jarrar, Tymaa Hammouda, and Fadi Zaraket. 2022. Curras + baladi: Towards a levantine corpus. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2022), Marseille, France. find
  13. Mustafa Jarrar. 2021. The arabic ontology - an arabic wordnet with ontologically clean content. Applied Ontology Journal, 16(1):1–26. find our card
  14. Mustafa Jarrar and Hamzeh Amayreh. 2019. An arabic-multilingual database with a lexicographic search engine. In The 24th International Conference on Applications of Natural Language to Information Systems (NLDB 2019), volume 11608 of LNCS, pages 234– 246. Springer. find
  15. Mustafa Jarrar, Nizar Habash, Diyam Akra, and Nasser Zalmout. 2014. Building a corpus for palestinian arabic: a preliminary study. In Proceedings of the EMNLP 2014, Workshop on Arabic Natural Language, pages 18–27. Association For Computational Linguistics. find
  16. Mustafa Jarrar, Nizar Habash, Faeq Alrimawi, Diyam Akra, and Nasser Zalmout. 2017. Curras: An annotated corpus for the palestinian arabic dialect. Journal Language Resources and Evaluation, 51(3):745– 775. find
  17. Mustafa Jarrar, Mohammed Khalilia, and Sana Ghanem. 2022. Wojood: Nested arabic named entity corpus and recognition using bert. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2022), Marseille, France. find our card
  18. Kyo Kageura and Genichiro Kikui. 2006. A self-referring quantitative evaluation of the ATR basic travel expression corpus (BTEC). In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06), Genoa, Italy. European Language Resources Association (ELRA). find
  19. Salam Khalifa, Nizar Habash, Fadhl Eryani, Ossama Obeid, Dana Abdulrahim, and Meera Al Kaabi. 2018. A morphologically annotated corpus of emirati Arabic. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  20. Hanaa Kilany, H Gadalla, Howaida Arram, A Yacoub, Alaa El-Habashi, and C McLemore. 2002. Egyptian colloquial arabic lexicon. LDC99L22. find
  21. Mohamed Maamouri, Ann Bies, Tim Buckwalter, Mona Diab, Nizar Habash, Owen Rambow, and Dalila Tabessi. 2006. Developing and using a pilot dialectal Arabic treebank. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06), Genoa, Italy. European Language Resources Association (ELRA). find
  22. Mohamed Maamouri, Ann Bies, Seth Kulick, Michael Ciul, Nizar Habash, and Ramy Eskander. 2014. Developing an Egyptian Arabic treebank: Impact of dialectal morphology on annotation and tool development. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 2348–2354, Reykjavik, Iceland. European Language Resources Association (ELRA). find
  23. Mohamed Maamouri, David Graff, Basma Bouziri, Sondos Krouna, Ann Bies, and Seth Kulick. 2010. Ldc standard arabic morphological analyzer (sama) version 3.1. LDC2010L01. find
  24. Mohammad Salameh, Houda Bouamor, and Nizar Habash. 2018. Fine-grained Arabic dialect identification. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1332–1344, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jarrar-2022-lisan-yemeni-iraqi-libyan-sudanese/