Elamur Research⌕
☰
home / papers / 2023 / SALMA: Arabic Sense-Annotated Corpus and WSD Benchmarks

SALMA: Arabic Sense-Annotated Corpus and WSD Benchmarks

Paper · 2023 · Mustafa Jarrar, Sanad Malaysha, Tymaa Hammouda et al.

Summary

Arabic word sense disambiguation (WSD) has lagged because the language had no sense-annotated corpus and no ready sense inventory. The authors present SALMA, a ~34K-token corpus of Modern Standard Arabic in which every token carries sense annotations. Two digitized lexicons serve as inventories: the Contemporary Arabic Dictionary ("Modern") and Al-Ghani Al-Zaher ("Ghani"). Rather than choosing a single correct sense, annotators grade every candidate sense of a word on an ordinal relatedness scale, and they do this separately for each lexicon. A dedicated web tool supports this workflow. The corpus also marks six named-entity types. Agreement among annotators, measured with several categorical and ordinal metrics, is reported as very high. To set baselines, the authors built an end-to-end WSD pipeline based on Target Sense Verification (TSV), which scores context–gloss pairs and picks the highest-scoring gloss. They tested three existing Arabic TSV models with it. ArabGlossBERT reached 84.2% accuracy with Modern, and Aug-ArabGlossBERT(D9) reached 78.7% with Ghani. Both the corpus and the tool are openly released.

Key points

  • SALMA contains 34,253 tokens in 1,439 sentences, averaging 23.8 tokens per sentence. It is drawn from 33 MSA online media sources and is a subset of the Wojood corpus. It covers 19,030 nouns, 2,763 verbs and 7,116 function words, and uses 4,151 unique senses from each inventory.
  • The grading scale has six levels: Explicate 100%, General 80%, Referral 60%, Related 40%, Root semantics 20% and Different 1%. A sense counts as correct if it scores at least 60%.
  • The named-entity labels are PERS, ORG, GPE, LOC, FAC and CURR. Three linguists annotated the corpus, and IAA was checked on 2.6% of the words using Kappa, LWK, QWK, MAE and RMSE.
  • Accuracy for the three TSV models (Modern / Ghani), using an 11-token context window:
    • Razzaz: 66.0% / 68.4%
    • ArabGlossBERT: 84.2% / 77.6%
    • Aug-ArabGlossBERT(D9): 82.6% / 78.7%
  • Further results for ArabGlossBERT:
    • Top-3 accuracy rises to about 98% with Modern.
    • Nouns score about 8.5 points above verbs with Modern.
    • Function words stay near 32–42%.
    • Changing the context window size alters accuracy by at most 1.4 points.

Interesting ideas

  • I1. Each candidate sense gets a graded relatedness score instead of a single gold label, which handles vague or overlapping glosses and lets the same data evaluate ranking-based disambiguators.
  • I2. The same text is annotated against two independent inventories, so the corpus also measures how well each dictionary covers the text and how good its glosses are.
  • I3. Disambiguation is recast as verifying context–gloss pairs and taking the best-scoring gloss, so any lexicon with definitions can plug in without retraining for a fixed label set.
  • I4. Ordinal labels are evaluated with weighted Kappa, MAE and RMSE, so a near-miss is penalized less than a total miss.
  • I5. Results are broken down by top-k rank and by part of speech, showing that function words need separate handling because their glosses describe grammatical roles rather than meaning.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): WordNet/FrameNet sense links in the ontology could store graded fits rather than one pick, so later steps can explain why an alternative sense was rejected.
  • I2 (world): running the same text against both WordNet and FrameNet/VerbNet would expose coverage gaps in our compiled knowledge before those gaps cause answer errors.
  • I3 (frames): when choosing a VerbNet class or frame for a predicate, each candidate's definition could be checked against the clause, and the verification score kept as an explainable justification.
  • I4 (learn): when witness parsers or LLM teachers give graded judgments, weighted agreement would separate mild disagreement from real conflict when filtering silver data.
  • I5 (answer): evaluation reports could split errors by word class and by rank, keeping function-word handling out of sense evaluation so it does not drag down the scores.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Birzeit UniversityBirzeit · Palestine38 works · 798 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 3Research subjects and languages the work is about.Semitic languages · 3,066 worksEvaluation & benchmarks · 1,341 worksWord sense disambiguation · 87 works

Cite

Cite as Mustafa Jarrar, Sanad Malaysha, Tymaa Hammouda and Mohammed Khalilia. 2023. SALMA: Arabic Sense-Annotated Corpus and WSD Benchmarks. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.29

@inproceedings{jarrar2023salma,
  title = {{SALMA: Arabic Sense-Annotated Corpus and WSD Benchmarks}},
  author = {Mustafa Jarrar, Sanad Malaysha, Tymaa Hammouda and Mohammed Khalilia},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.29},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.29}
}

References

52 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdul-Ghani Abul-Azm. 2014. Al-ghani al-zaher dictionary. Rabat: Al-Ghani Publishing Institution. find
  2. Moustafa Al-Hajj and Mustafa Jarrar. 2021. Arabgloss-bert: Fine-tuning bert on context-gloss pairs for wsd. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 40–48, Online. INCOMA Ltd. find
  3. Diana Alhafi, Anton Deik, and Mustafa Jarrar. 2019. Usability evaluation of lexicographic e-services. In The 2019 IEEE/ACS 16th International Conference on Computer Systems and Applications (AICCSA), pages 1–7. IEE. find
  4. Hamzeh Amayreh, Mohammad Dwaikat, and Mustafa Jarrar. 2019. Lexicon digitization-a framework for structuring, normalizing and cleaning lexical entries. Technical Report, Birzeit University. ↗
  5. William Black, Sabri Elkateb, Horacio Rodriguez, Musa Alkhalifa, Piek Vossen, Adam Pease, Christiane Fellbaum, et al. 2006. Introducing the arabic wordnet project. In Proceedings of the third international WordNet conference, pages 295–300. Jeju Korea. find
  6. Terra Blevins and Luke Zettlemoyer. 2020. Moving down the long tail of word sense disambiguation with gloss-informed biencoders. arXiv preprint arXiv:2005.02590. ↗
  7. Sami Boudelaa and William D Marslen-Wilson. 2004. Abstract morphemes and lexical representation: The cv-skeleton in arabic. Cognition, 92(3):271–303. find
  8. Sami Boudelaa, Friedemann Pulvermüller, Olaf Hauk, Yury Shtyrov, and William Marslen-Wilson. 2010. Arabic morphology in the neural language system. Journal of cognitive neuroscience, 22(5):998–1010. ↗
  9. Anna Breit, Artem Revenko, Kiamehr Rezaee, Mohammad Taher Pilehvar, and Jose Camacho-Collados. 2020. Wic-tsv: An evaluation benchmark for target sense verification of words in context. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 - 23, 2021, pages 1635–1645. find
  10. Bartosz Broda, Michał Marcińczuk, Marek Maziarz, Adam Radziszewski, and Adam Wardyński. 2012. KPWr: Towards a free corpus of Polish. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 3218–3222, Istanbul, Turkey. European Language Resources Association (ELRA). find
  11. Simone Conia and Roberto Navigli. 2021. Framing word sense disambiguation as a multi-label problem for model-agnostic knowledge integration. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 - 23, 2021, pages 3269–3275. Association for Computational Linguistics. ↗
  12. Kareem Darwish, Nizar Habash, Mourad Abbas, Hend Al-Khalifa, Huseein T. Al-Natsheh, Houda Bouamor, Karim Bouzoubaa, Violetta Cavalli-Sforza, Samhaa R. El-Beltagy, Wassim El-Hajj, Mustafa Jarrar, and Hamdy Mubarak. 2021. A panoramic survey of natural language processing in the arab worlds. Commun. ACM, 64(4):72–81. find
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171–4186. Association for Computational Linguistics.
  14. Mohammed El-Razzaz, Mohamed Waleed Fakhr, and Fahima A Maghraby. 2021. Arabic gloss wsd using bert. Applied Sciences, 11(6):2567. ↗
  15. Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 3507–3512. Association for Computational Linguistics. find
  16. Mustafa Jarrar. 2005. Towards Methodological Principles for Ontology Engineering. Ph.D. thesis, Vrije Universiteit Brussel. find
  17. Mustafa Jarrar. 2006. Towards the notion of gloss, and the adoption of linguistic resources in formal ontology engineering. In Proceedings of the 15th international conference on World Wide Web (WWW2006), pages 497–503. ACM Press, New York, NY. find
  18. Mustafa Jarrar. 2011. Building a formal arabic ontology (invited paper). In Proceedings of the Experts Meeting on Arabic Ontologies and Semantic Networks. ALECSO, Arab League. find
  19. Mustafa Jarrar. 2021. The arabic ontology - an arabic wordnet with ontologically clean content. Applied Ontology Journal, 16(1):1–26. ↗
  20. Mustafa Jarrar and Hamzeh Amayreh. 2019. An arabic-multilingual database with a lexicographic search engine. In The 24th International Conference on Applications of Natural Language to Information Systems (NLDB 2019), volume 11608 of LNCS, pages 234–246. Springer. find
  21. Mustafa Jarrar, Hamzeh Amayreh, and John P. McCrae. 2019. Representing arabic lexicons in lemon - a preliminary study. In The 2nd Conference on Language, Data and Knowledge (LDK 2019), volume 2402, pages 29–33. CEUR Workshop Proceedings. ↗
  22. Mustafa Jarrar, Nizar Habash, Faeq Alrimawi, Diyam Akra, and Nasser Zalmout. 2017. Curras: An annotated corpus for the palestinian arabic dialect. Journal Language Resources and Evaluation, 51(3):745– 775. ↗
  23. Bilel Elayeb. 2019. Arabic word sense disambiguation: a review. Artif. Intell. Rev., 52(4):2475–2532. ↗
  24. Barbara Di Eugenio and Michael Glass. 2004. The kappa statistic: A second look. Computational linguistics, 30(1):95–101. ↗
  25. Sana Ghanem, Mustafa Jarrar, Radi Jarrar, and Ibrahim Bounhas. 2023. A benchmark and scoring algorithm for enriching arabic synonyms. In Proceedings of the 12th International Global Wordnet Conference (GWC2023), pages 215–222. Global Wordnet Association.
  26. Karim El Haff, Mustafa Jarrar, Tymaa Hammouda, and Fadi Zaraket. 2022. Curras + baladi: Towards a levantine corpus. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2022), Marseille, France. ↗
  27. Mustafa Jarrar, Eman Karajah, Muhammad Khalifa, and Khaled Shaalan. 2021. Extracting synonyms from bilingual dictionaries. In Proceedings of the 11th International Global Wordnet Conference (GWC2021), pages 215–222. Global Wordnet Association. find
  28. Mustafa Jarrar, Mohammed Khalilia, and Sana Ghanem. 2022. Wojood: Nested arabic named entity corpus and recognition using bert. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2022), Marseille, France.
  29. Mustafa Jarrar, Fadi Zaraket, Rami Asia, and Hamzeh Amayreh. 2018. Diacritic-based matching of arabic words. ACM Asian and Low-Resource Language Information Processing, 18(2):10:1–10:21. ↗
  30. Luyao Huang, Chi Sun, Xipeng Qiu, and Xuanjing Huang. 2019. Glossbert: BERT for word sense disambiguation with gloss knowledge. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Mustafa Jarrar, Fadi Zaraket, Tymaa Hammouda, Daanish Masood Alavi, and Martin Waahlisch. 2023. Lisan: Yemeni, irqi, libyan, and sudanese arabic dialect copora with morphological annotations. In The 20th IEEE/ACS International Conference on Computer Systems and Applications (AICCSA). IEEE. ↗
  31. Sanad Malaysha, Mustafa Jarrar, and Mohammad Khalilia. 2023. Context-gloss augmentation for improving arabic target sense verification. In Proceedings of the 12th International Global Wordnet Conference (GWC2023). Global Wordnet Association.
  32. Federico Martelli, Najla Kalach, Gabriele Tola, and Roberto Navigli. 2021. Semeval-2021 task 2: Multilingual and cross-lingual word-in-context disambiguation (mcl-wic). In Proceedings of the 15th International Workshop on Semantic Evaluation, SemEval@ACL/IJCNLP 2021, Virtual Event / Bangkok, Thailand, August 5-6, 2021, pages 24–36. Association for Computational Linguistics. find
  33. Marek Maziarz, Maciej Piasecki, and Stan Szpakowicz. 2012. Approaching plwordnet 2.0. In Proceedings of 6th International Global Wordnet Conference, The Global WordNet Association, pages 189–196. find
  34. George A Miller, Richard Beckwith, Christiane Fellbaum, Derek Gross, and Katherine J Miller. 1990. Introduction to wordnet: An on-line lexical database. International journal of lexicography, 3(4):235–244. ↗
  35. George A Miller, Claudia Leacock, Randee Tengi, and Ross T Bunker. 1993. A semantic concordance. In Human Language Technology: Proceedings of a Workshop Held at Plainsboro, New Jersey, March 21-24, 1993.
  36. Simonetta Montemagni and Guglielmo Venturi. 2003. Building sense-tagged corpora for all: The itec project. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04). find
  37. Eman Naser-Karajah, Nabil Arman, and Mustafa Jarrar. 2021. Current trends and approaches in synonyms extraction: Potential adaptation to arabic. In Proceedings of the 2021 International Conference on Information Technology (ICIT), pages 428–434, Amman, Jordan. IEEE.
  38. Amal Nayouf, Mustafa Jarrar, Fadi zaraket, Tymaa Hammouda, and Mohamad-Bassam Kurdy. 2023. Nâbra: Syrian arabic dialects with morphological annotations. In Proceedings of the 1st Arabic Natural Language Processing Conference (ArabicNLP), Part of the EMNLP 2023. ACL.
  39. Ahmed Mukhtar Omar. 2008. Contemporary arabic dictionary.(i1). World of Books, Cairo, Egypt. Retrieval Date, 14(8):2020. find
  40. Tommaso Petrolito and Francis Bond. 2014. A survey of wordnet annotated corpora. In Proceedings of the Seventh Global Wordnet Conference, GWC 2014, Tartu, Estonia, January 25-29, 2014, pages 236–245. University of Tartu Press. find
  41. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. OpenAI Technical Report, 1(8). find
  42. Karin C Ryding. 2014. Arabic: A linguistic introduction. Cambridge University Press. find
  43. Erik F. Tjong Kim Sang and Jorn Veenstra. 1999. Representing text chunks. In EACL 1999, 9th Conference of the European Chapter of the Association for Computational Linguistics, June 8-12, 1999, University of Bergen, Bergen, Norway, pages 173–179. The Association for Computer Linguistics.
  44. Nathan Schneider, Behrang Mohit, Kemal Oflazer, and Noah A. Smith. 2012. Coarse lexical semantic annotation with supersenses: An arabic case study. In The 50th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, July 8-14, 2012, Jeju Island, Korea - Volume 2: Short Papers, pages 253–258. The Association for Computer Linguistics.
  45. Liling Tan and Francis Bond. 2012. Building and annotating the linguistically diverse NTU-MC (NTU - multilingual corpus). Int. J. Asian Lang. Process., 22(4):161–174. find
  46. Mariona Taulé, M. Antònia Martí, and Marta Recasens. 2008. AnCora: Multilevel annotated corpora for Catalan and Spanish. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), Marrakech, Morocco. European Language Resources Association (ELRA). find
  47. Heike Telljohann, Erhard Hinrichs, and Ra Ubler. 2004. The tüba-d/z treebank: Annotating german with a context-free backbone. Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC). find
  48. Sophie Vanbelle. 2016. A new interpretation of the weighted kappa coefficients. Psychometrika, 81(2):399–410. find
  49. Warren Weaver. 1949/1955. Translation. In William N. Locke and A. Donald Boothe, editors, Machine Translation of Languages, pages 15–23. MIT Press, Cambridge, MA. Reprinted from a memorandum written by Weaver in 1949. find
  50. Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Mohammed El-Bachouti, Robert Belvin, and Ann Houston. 2013. Ontonotes release 5.0 ldc2013t19. Philadelphia: Linguistic Data Consortium. find
  51. Ralph Weischedel, Sameer Pradhan, Lance Ramshaw, Martha Palmer, Nianwen Xue, Mitchell Marcus, Ann Taylor, Craig Greenberg, Eduard Hovy, Robert Belvin, et al. 2011. Ontonotes release 4.0. LDC2011T03, Philadelphia, Penn.: Linguistic Data Consortium. ↗
  52. Boon Peng Yap, Andrew Koh, and Eng Siong Chng. 2020. Adapting bert for word sense disambiguation with gloss selection objective and example sentences. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 41–46. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jarrar-2023-salma-arabic-sense-annotated-corpus/