Elamur Research⌕
☰
home / papers / 2024 / Qabas: An Open-Source Arabic Lexicographic Database

Qabas: An Open-Source Arabic Lexicographic Database

Paper · 2024 · Mustafa Jarrar and Tymaa Hasanain Hammouda

Summary

Arabic has many traditional dictionaries, but few lexical resources are built for NLP and openly licensed. Most digitized lexicons stay closed because of owner copyright, and LDC's SAMA is restricted to members and covers only morphology. Qabas addresses this with an open lexicon of about 58K lemmas. Its central design choice is that each lemma serves as a hub: it links to matching entries in 110 existing lexicons and to the lemma annotations of 12 morphologically annotated corpora. Together these links form a lexicographic graph.

The resource was built semi-automatically over two years:

  • Lemmas from the Modern lexicon were imported first.
  • Three lexicographers revised them and added eight morphological features.
  • Rule-based heuristics proposed candidate links for the lexicographers to confirm.
  • Uncovered lemmas were then queued by how often they appear across all resources.

Qabas is nearly twice the size of Modern and about 40% larger than SAMA. It includes every Modern lemma and 99% of SAMA's. Because the links let separate resources be combined, Qabas enables tasks such as merging morphology from one source with senses from another, or computing per-lemma corpus statistics.

Key points

  • Size: about 58K lemmas in total: roughly 45K nominal, 12.5K verbal and 473 functional-word lemmas. The tagset has 41 POS tags.
  • Lexicon links: the 110 source lexicons hold about 297K single-word lemmas, of which about 255.5K (84%) are mapped. Sources include SAMA, Modern, Ghani, Al-Waseet, Arabic WordNet, 40 ALECSO Unified dictionaries, 16 Arab Academy lexicons and Wikidata. There are about 256K mapping correspondences, labelled with six relation types.
  • Corpus links: the 12 corpora contain 2.4M tokens and about 144.5K lemmas, and about 2M tokens are linked. They cover MSA (Arabic Treebank, SALMA), Classical (Quran) and dialects (Curras, Baladi, Lisan, Gummar, Nabra, Egyptian Treebank). Most corpus links came from swapping SAMA lemma IDs for Qabas IDs.
  • Mapping quality: heuristics proposed 179K candidate mappings for human checking. On a random 5% sample (2,850 lemmas), agreement between the three lexicographers reached Kappa of 85–88%.
  • Licensing: only the Qabas lemmas with their features, plus the mapping links, are released. The copyrighted lexicon contents are not shared.

Interesting ideas

  • I1. Treat a curated lemma inventory as a pivot that every external resource points to, so the resources can be combined through shared IDs rather than merged into one copy.
  • I2. Generate mapping candidates by checking whether sets of inflected forms overlap, using POS-specific conditions; for verbs, perfective, imperfective, imperative and root must be compatible wherever both sides provide them.
  • I3. Expand coverage by ranking missing items according to how many resources and corpora they occur in, so annotation effort goes first to the most widely attested gaps.
  • I4. Tag a lemma as a proper noun only when all of its meanings are proper names; otherwise keep one lemma, with naming treated as one of its senses.
  • I5. When source data cannot be redistributed, publish only your own entries plus the links into the licensed resources, which leaves users to obtain those resources themselves.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): A single Mova lemma/concept ID could anchor WordNet synsets, VerbNet classes, FrameNet frames and Wikidata items, so each knowledge lookup becomes an explainable path through explicit links.
  • I2 (world): Alignments between VerbNet members, WordNet lemmas and FrameNet lexical units could be proposed the same way, by comparing inflection and subcategorization, with an LLM teacher or human approving them.
  • I3 (learn): Repair-rule induction and silver-data review could prioritize lemmas or constructions that are frequent across several treebanks but currently fail, giving more improvement per hour of work.
  • I4 (tokens): Capitalized common words such as "Rose" or "Bush" could keep a single lemma, with proper-name use resolved later, avoiding duplicate lemmas that break downstream frames.
  • I5 (world): Mova's compiled knowledge could ship link tables into restricted lexicons or corpora without redistributing them, keeping the graph shareable and legally clean.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksLiterature and Literary Theory · 519 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksMorphology · 405 works

Cite

Cite as Mustafa Jarrar and Tymaa Hasanain Hammouda. 2024. Qabas: An Open-Source Arabic Lexicographic Database. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{jarrar2024qabas,
  title = {{Qabas: An Open-Source Arabic Lexicographic Database}},
  author = {Mustafa Jarrar and Tymaa Hasanain Hammouda},
  booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
  year = {2024},
  doi = {10.63317/37k66zmippxm},
  url = {https://doi.org/10.63317/37k66zmippxm}
}

References

39 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdul-Ghani Abul-Azm. 2014. Al-Ghani Al-Zaher Dictionary. Rabat: Al-Ghani Publishing Institution. find
  2. Moustafa Al-Hajj and Mustafa Jarrar. 2021. Arabglossbert: Fine-tuning bert on context-gloss pairs for wsd. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 40–48, Online. INCOMA Ltd. Arabization Coordination Bureau (Rabat) ALECSO. http://www.arabization.org.ma/. ↗
  3. Diana Alhafi, Anton Deik, and Mustafa Jarrar. 2019. Usability evaluation of lexicographic e-services. In The 2019 IEEE/ACS 16th International Conference on Computer Systems and Applications (AICCSA), pages 1–7. IEE. find
  4. Hamzeh Amayreh, Mohammad Dwaikat, and Mustafa Jarrar. 2019. Lexicon digitization-a framework for structuring, normalizing and cleaning lexical entries. Technical Report, Birzeit University. ↗
  5. William Black, Sabri Elkateb, Horacio Rodriguez, Musa Alkhalifa, Piek Vossen, Adam Pease, Christiane Fellbaum, et al. 2006. Introducing the arabic wordnet project. In Proceedings of the third international WordNet conference, pages 295–300. Jeju Korea. find
  6. Tim Buckwalter. 2004. Buckwalter arabic morphological analyzer (bama) version 2.0. linguistic data consortium (ldc) catalogue number ldc2004l02. Technical report, ISBN1-58563-324-0. find
  7. Arabic Language Academy Cairo. 2004. Al-Waseet Dictionary. Shorouk International Bookshop. Arabic Language Academy in Cairo. https://www.arabicacademy.gov.eg/. Arabic Language Academy in Damascus. http://arabacademy.gov.sy/. ↗
  8. Kareem Darwish, Nizar Habash, Mourad Abbas, Hend Al-Khalifa, Huseein T. Al-Natsheh, Houda Bouamor, Karim Bouzoubaa, Violetta Cavalli-Sforza, Samhaa R. El-Beltagy, Wassim El-Hajj, Mustafa Jarrar, and Hamdy Mubarak. 2021. A panoramic survey of natural language processing in the arab worlds. Commun. ACM, 64(4):72–81. find
  9. Kais Dukes and Nizar Habash. 2010. Morphological annotation of quranic arabic. In Lrec, pages 2530–2536.
  10. Gil Francopoulo, Monte George, Nicoletta Calzolari, Monica Monachini, Nuria Bel, Mandy Pet, and Claudia Soria. 2006. Lexical markup framework (LMF). In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06), Genoa, Italy. European Language Resources Association (ELRA).
  11. Sana Ghanem, Mustafa Jarrar, Radi Jarrar, and Ibrahim Bounhas. 2023. A benchmark and scoring algorithm for enriching arabic synonyms. In Proceedings of the 12th International Global Wordnet Conference (GWC2023), pages 215– 222. Global Wordnet Association.
  12. Karim El Haff, Mustafa Jarrar, Tymaa Hammouda, and Fadi Zaraket. 2022. Curras + baladi: Towards a levantine corpus. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2022), Marseille, France. ↗
  13. Mustafa Jarrar. 2011. Building a formal arabic ontology (invited paper). In Proceedings of the Experts Meeting on Arabic Ontologies and Semantic Networks. ALECSO, Arab League. find
  14. Mustafa Jarrar. 2020. Digitization of Arabic Lexicons, pages 214–217. UAE Ministry of Culture and Youth. find
  15. Mustafa Jarrar. 2021. The arabic ontology - an arabic wordnet with ontologically clean content. Applied Ontology Journal, 16(1):1–26. ↗
  16. Mustafa Jarrar and Hamzeh Amayreh. 2019. An arabic-multilingual database with a lexicographic search engine. In The 24th International Conference on Applications of Natural Language to Information Systems (NLDB 2019), volume 11608 of LNCS, pages 234–246. Springer. find
  17. Mustafa Jarrar, Hamzeh Amayreh, and John P. Mc-Crae. 2019. Representing arabic lexicons in lemon - a preliminary study. In The 2nd Conference on Language, Data and Knowledge (LDK 2019), volume 2402, pages 29–33. CEUR Workshop Proceedings. ↗
  18. Mustafa Jarrar, Nizar Habash, Diyam Akra, and Nasser Zalmout. 2014. Building a corpus for palestinian arabic: a preliminary study. In Proceedings of the EMNLP 2014, Workshop on Arabic Natural Language, pages 18–27. Association For Computational Linguistics.
  19. Mustafa Jarrar, Nizar Habash, Faeq Alrimawi, Diyam Akra, and Nasser Zalmout. 2017. Curras: An annotated corpus for the palestinian arabic dialect. Journal Language Resources and Evaluation, 51(3):745–775. ↗
  20. Mustafa Jarrar, Mohammed Khalilia, and Sana Ghanem. 2022. Wojood: Nested arabic named entity corpus and recognition using bert. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2022), Marseille, France.
  21. Mustafa Jarrar, Sanad Malaysha, Tymaa Hammouda, and Mohammad Khalilia. 2023a. Salma: Arabic sense-annotated corpus and wsd benchmarks. In Proceedings of the 1st Arabic Natural Language Processing Conference (ArabicNLP), Part of the EMNLP 2023. ACL.
  22. Mustafa Jarrar, Fadi Zaraket, Rami Asia, and Hamzeh Amayreh. 2018. Diacritic-based matching of arabic words. ACM Asian and Low-Resource Language Information Processing, 18(2):10:1–10:21. ↗
  23. Mustafa Jarrar, Fadi Zaraket, Tymaa Hammouda, Daanish Masood Alavi, and Martin Waahlisch. 2023b. Lisan: Yemeni, irqi, libyan, and sudanese arabic dialect copora with morphological annotations. In The 20th IEEE/ACS International Conference on Computer Systems and Applications (AICCSA). IEEE. find
  24. Mustapha Khalfi, Ouafae Nahli, and Arsalane Zarghili. 2016. Classical dictionary al-qamus in lemon. In 2016 4th IEEE International Colloquium on Information Science and Technology (CiSt), pages 325–330. IEEE. find
  25. Salam Khalifa, Nizar Habash, Fadhl Eryani, Ossama Obeid, Dana Abdulrahim, and Meera Al Kaabi. 2018. A morphologically annotated corpus of emirati Arabic. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
  26. Nouran Khallaf, Elin Arfon, Mo El-Haj, Jon Morris, Dawn Knight, Paul Rayson, and Tymaa Ham-moudaand Mustafa Jarrar. 2023. Open-source thesaurus development for under-resourced languages: a welsh case study. find
  27. Aïda Khemakhem, Bilel Gargouri, Abdelmajid Ben Hamadou, and Gil Francopoulo. 2016. Iso standard modeling of a large arabic dictionary. Natural Language Engineering, 22(6):849–879. ↗
  28. Mohamed Maamouri, Ann Bies, Seth Kulick, Sondos Krouna, Dalila Tabassi, and Michael Ciul. 2021. Bolt egyptian arabic treebank - sms/chat. Mohamed Maamouri, Ann Bies, Sondos Krouna Seth Kulick, Fatma Gaddeche, and Wajdi Zaghouani. Arabic treebank: Part 3 v 3.2. find
  29. Mohamed Maamouri, David Graff, Basma Bouziri, Sondos Krouna, Ann Bies, and Seth Kulick. 2010. Ldc standard arabic morphological analyzer (sama) version 3.1. LDC2010L01. ↗
  30. Isa Maks, Carole Tiberius, and Remco van Veenendaal. 2009. Standardising bilingual lexical resources according to the lexicon markup framework. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), Marrakech, Morocco. European Language Resources Association (ELRA). find
  31. Sanad Malaysha, Mustafa Jarrar, and Mohammad Khalilia. 2023. Context-gloss augmentation for improving arabic target sense verification. In Proceedings of the 12th International Global Wordnet Conference (GWC2023). Global Wordnet Association.
  32. John P McCrae, Christian Chiarcos, Francis Bond, Philipp Cimiano, Thierry Declerck, Gerard De Melo, Jorge Gracia, Sebastian Hellmann, Bettina Klimek, Steven Moran, et al. 2016. The open linguistics working group: Developing the linguistic linked open data cloud. find
  33. Navigli, Roberto, Ponzetto, and Simone Paolo. 2012. Babelnet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artificial intelligence, 193:217–250. find
  34. Amal Nayouf, Mustafa Jarrar, Fadi zaraket, Tymaa Hammouda, and Mohamad-Bassam Kurdy. 2023. Nâbra: Syrian arabic dialects with morphological annotations. In Proceedings of the 1st Arabic Natural Language Processing Conference (ArabicNLP), Part of the EMNLP 2023. ACL.
  35. Ahmed Mukhtar Omar. 2008. Contemporary Arabic Dictionary., volume 14. World of Books, Cairo, Egypt. find
  36. Paul Buitelaar Philipp Cimiano, John P. McCrae. 2016. Lexicon model for ontologies. final community group report, 10 may 2016. find
  37. Susanne Salmon-Alt, Amine Akrout, and Laurent Romary. 2005. Proposals for a normalized representation of standard arabic full form lexica. In International Conference on Machine Intelligence. find
  38. Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the AAAI conference on artificial intelligence, volume 31. ↗
  39. Anthony Viera and Joanne Garrett. 2005. Understanding interobserver agreement: the kappa statistic. Fam med, 37(5):360–363. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jarrar-2024-qabas-open-source-arabic-lexicographic/