Elamur Research⌕
☰
home / papers / 2024 / KSAA-CAD Shared Task: Contemporary Arabic Dictionary for Reverse Dictionary and Word Sense Disambiguation

KSAA-CAD Shared Task: Contemporary Arabic Dictionary for Reverse Dictionary and Word Sense Disambiguation

Paper · 2024 · Waad Alshammari, Amal Almazrua, Asma Al Wazrah et al.

Summary

KSAA-CAD is the second edition of the King Salman Global Academy's shared task on Contemporary Arabic lexical semantics. It continues the earlier KSAA-RD by keeping the Reverse Dictionary (RD) task and adding Word Sense Disambiguation (WSD). In RD, a system reads a gloss and must regress a vector for the defined lemma. Three contextualized target spaces are used: AraELECTRA, AraBERTv2 and camelBERT-MSA. In WSD, a system decides which dictionary sense a word has in a given context, framed as gloss-based sentence-pair binary classification.

The RD data now merges three lemma-based dictionaries, all stored in ISO LMF format. The WSD data pairs contexts with gloss IDs from the Omar dictionary.

Four teams submitted papers, mostly using fine-tuned AraT5 V2 or AraBERTv2. ASOS ranked best among participants and also led on cosine and MSE. Even so, the organizers' CamelBERT and MARBERT baselines still beat every team on the rank metric. WSD results lagged far behind the baseline.

The task matters because it supplies open Arabic resources and shows that Arabic RD and WSD remain largely unsolved.

Key points

  • RD dictionary sources: Alriyadh Dictionary (10,626 entries), Ahmed Mokhtar Omar's Contemporary Arabic Language Dictionary (3,341) and Al Wassit LMF (25,248), totalling 39,215 entries.
    • The RD split is 31,372 / 3,921 / 3,922.
  • WSD data: 22,404 / 2,801 / 2,801 context–gloss items, plus a WSD dictionary of 15,865 glosses.
  • Target embeddings: AraELECTRA ("Electra"), AraBERTv2 ("bertSEG") and camelBERT-MSA ("bertMSA").
    • Metrics are cosine similarity, MSE and rank, where a lower rank is better.
  • RD results on test:
    • ASOS ranked best among teams at 0.0644 with Electra, and reached cosine 0.83 and MSE 0.059 with bertSEG.
    • The baselines sat around 0.033 rank.
    • Baleegh (0.1781) and MISSION (0.2482) used AraT5 V2.
    • Baleegh found that enriching glosses with Wikipedia example sentences did not help.
  • Ambiguity statistics and WSD outcome:
    • The cited figures are roughly 43% of diacritized Arabic words and 72% of undiacritized words being polysemous.
    • The participating WSD system scored 64% below the baseline.

Interesting ideas

  • I1. Reverse dictionary as regression: map a definition onto a word's embedding, then score it by where the true word ranks among all candidates, so it doubles as definition-based retrieval and a test of gloss understanding.
  • I2. Score the same task against several target embedding spaces at once; the paper shows the best space differs by metric (Electra for rank, bertSEG for cosine).
  • I3. Cosine, MSE and rank can disagree, and a simple organizer baseline still won on rank, which shows why a strong, plain reference system is needed in any leaderboard.
  • I4. Gloss-based WSD as sentence-pair binary classification: pair the context with each candidate gloss, label each pair as matching or not, and pick the best-scoring sense; this turns sense inventories into supervised data without a sense-specific classifier.
  • I5. Merge several dictionaries into one lemma-based lexicon under a shared standard format (ISO LMF), keeping only lemmas attested across sources.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): WordNet glosses could be encoded and matched to lemma vectors so that a paraphrase in a text ("a person who repairs shoes") resolves to an ontology node, with the ranked candidates shown as the explanation.
  • I2 (learn): when evaluating lexical components, report results across several target representations and metrics so that a gain tied to one space is not mistaken for real progress.
  • I3 (learn): keep a minimal baseline and negative controls next to every new learned module, and prefer the retrieval-style rank metric when the downstream use is lookup.
  • I4 (frames): VerbNet/WordNet sense choice for a predicate could be scored as context–gloss pairs, giving an inspectable reason (the winning gloss) for the frame chosen.
  • I5 (world): unifying WordNet, FrameNet and Wikidata entries under one lemma-keyed schema, with cross-source agreement as a filter, could yield cleaner compiled knowledge.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksWord sense disambiguation · 83 works

Cite

Cite as Waad Alshammari, Amal Almazrua, Asma Al Wazrah, Rawan Almatham, Muneera Alhoshan and Abdulrahman Alosaimy. 2024. KSAA-CAD Shared Task: Contemporary Arabic Dictionary for Reverse Dictionary and Word Sense Disambiguation. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.74

@inproceedings{alshammari2024ksaa,
  title = {{KSAA-CAD Shared Task: Contemporary Arabic Dictionary for Reverse Dictionary and Word Sense Disambiguation}},
  author = {Waad Alshammari, Amal Almazrua, Asma {Al Wazrah}, Rawan Almatham, Muneera Alhoshan and Abdulrahman Alosaimy},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.74},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.74}
}

References

35 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim A. Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic. CoRR, abs/2101.01785. arXiv: 2101.01785. ↗
  2. Abdul-Ghani Abul-Azm. 2014. Al-ghani al-zaher dictionary. find
  3. Aarchi Agrawal, Athulkumar R, K S Ashin Shanly, Kavita Vaishnaw, and Mayank Singh. 2021. Reverse Dictionary Using an Improved CBoW Model. In Proceedings of the 3rd ACM India Joint International Conference on Data Science & Management of Data (8th ACM IKDD CODS & 26th COMAD), page 420, New York, NY, USA. Association for Computing Machinery. find
  4. Moustafa Al-Hajj and Mustafa Jarrar. 2021. ArabGlossBERT: Fine-Tuning BERT on Context-Gloss Pairs for WSD. In Ruslan Mitkov and Galia Angelova, editors, Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 35–43, Held Online. INCOMA Ltd.
  5. THAMER MASEER Alharbi. 2024. MISSION at KSAA-CAD 2024: AraT5 with Arabic Reverse Dictionary. In Proceedings of the 2nd Arabic Natural Language Processing Conference (Arabic-NLP), Part of the ACL 2024. ↗
  6. Mais Alheraki and Souham Meshoul. 2024. Baleegh at KSAA-CAD 2024: Towards Enhancing Arabic Reverse Dictionaries. In Proceedings of the 2nd Arabic Natural Language Processing Conference (Arabic-NLP), Part of the ACL 2024. Association for Computational Linguistics. ↗
  7. Hawra Aljasim, Sultanah Alghurabi, Halah Alharbi, Abdulrahman Alosaimy, Afrah Altamimi, Asyah Almutawa, and Abdullah Alfifi. 2022. Toward an Automatic Semantic Order of Word Meanings Based on Lexically Tagged Corpus. Manuscript submitted for publication. find
  8. Rawan Al-Matham, Waad Alshammari, Abdulrahman AlOsaimy, Sarah Alhumoud, Asma Wazrah, Afrah Altamimi, Halah Alharbi, and Abdullah Alaifi. 2023. KSAA-RD Shared Task: Arabic Reverse Dictionary. In Hassan Sawaf, Samhaa El-Beltagy, Wajdi Zaghouani, Walid Magdy, Ahmed Abdelali, Nadi Tomeh, Ibrahim Abu Farha, Nizar Habash, Salam Khalifa, Amr Keleg, Hatem Haddad, Imed Zitouni, Khalil Mrini, and Rawan Almatham, editors, Proceedings of ArabicNLP 2023, pages 450–460, Singapore (Hybrid).
  9. Sawsan Alqahtani, Hanan Aldarmaki, and Mona Diab. 2019. Homograph Disambiguation through Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based Model for Arabic Language Understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association.
  10. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. AraELECTRA: Pre-Training Text Discriminators for Arabic Language Understanding. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 191–195, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. ↗
  11. Pinzhen Chen, Zheng Zhao, and Shun Shao. 2024. Cher at KSAA-CAD 2024: Compressing Words and Definitions into the Same Space for Arabic Reverse Dictionary. In Proceedings of the 2nd Arabic Natural Language Processing Conference (Arabic-NLP), Part of the ACL 2024. Association for Computational Linguistics.
  12. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. CoRR, abs/2003.10555. arXiv: 2003.10555. ↗
  13. Fathi Debili, Hadhemi Achour, and Emna Souissi. 2002. La langue arabe et l’ordinateur de l’étiquetage gramatical à la voyellation automatique. In Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North, pages 4171–4186, Minneapolis, Minnesota.
  14. Mohammed El-Razzaz, Mohamed Waleed Fakhr, and Fahima A. Maghraby. 2021. Arabic Gloss WSD Using BERT. Applied Sciences, 11(6). ↗
  15. Michael A. Hedderich, Andrew Yates, Dietrich Klakow, and Gerard de Melo. 2019. Using Multi-Sense Vector Embeddings for Reverse Dictionaries. In Proceedings of the 13th International Conference on Computational Semantics - Long Papers, pages 247–258, Gothenburg, Sweden. ↗
  16. Felix Hill, Kyunghyun Cho, Anna Korhonen, and Yoshua Bengio. 2016. Learning to Understand Phrases by Embedding the Dictionary. Transactions of the Association for Computational Linguistics, 4:17–30. ↗
  17. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92–104, Kyiv, Ukraine (Virtual).
  18. Mustafa Jarrar, Sanad Malaysha, Tymaa Hammouda, and Mohammed Khalilia. 2023. SALMA: Arabic Sense-Annotated Corpus and WSD Benchmarks. In Hassan Sawaf, Samhaa El-Beltagy, Wajdi Zaghouani, Walid Magdy, Ahmed Abdelali, Nadi Tomeh, Ibrahim Abu Farha, Nizar Habash, Salam Khalifa, Amr Keleg, Hatem Haddad, Imed Zitouni, Khalil Mrini, and Rawan Almatham, editors, Proceedings of ArabicNLP 2023, pages 359–369, Singapore (Hybrid). king Salman global academy for Arabic language KSAA. 2023. Alriyadh Dictionary.
  19. Timothee Mickus, Kees Van Deemter, Mathieu Constant, and Denis Paperno. 2022. Semeval-2022 Task 1: CODWOE – Comparing Dictionaries and Word Embeddings. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1–14, Seattle, United States. Association for Computational Linguistics.
  20. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781 [cs]. ↗
  21. Yuya Morinaga and Kazunori Yamaguchi. 2018. Improvement of Reverse Dictionary by Tuning Word Vectors and Category Inference. In Robertas Damaševičius and Giedrė Vasiljevienė, editors, Information and Software Technologies, pages 533– 545, Cham. Springer International Publishing. find
  22. Yuya Morinaga and Kazunori Yamaguchi. 2020. Improvement of Neural Reverse Dictionary by Using Cascade Forward Neural Network. Journal of Information Processing, 28:715–723. ↗
  23. El Moatez Billah Nagoudi, AbdelRahim A. Elmadany, and Muhammad Abdul-Mageed. 2021. AraT5: Text-to-Text Transformers for Arabic Language Understanding and Generation. CoRR, abs/2109.12068. arXiv: 2109.12068. ↗
  24. Driss Namly. 2015. “Al wassit” LMF Arabic dictionary. find
  25. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2151–2156, Minneapolis, Minnesota. find
  26. Fanchao Qi, Lei Zhang, Yanhui Yang, Zhiyuan Liu, and Maosong Sun. 2020. WantWords: An Open-source Online Reverse Dictionary System. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 175–181, Online. find
  27. Serry Sibaee, Abdullah I. Alharbi, Samar Ahmad, OMER NACAR, and Anis Koubaa. 2024. ASOS at KSAA-CAD 2024: One Embedding is All You Need for Your Dictionary. In Proceedings of the 2nd Arabic Natural Language Processing Conference (Arabic-NLP), Part of the ACL 2024.
  28. Abu Bakr Soliman, Kareem Eissa, and Samhaa R. El-Beltagy. 2017. AraVec: A set of Arabic Word Embedding Models for use in Arabic NLP. Procedia Computer Science, 117:256–265. find
  29. Sicheng Tian, Shaobin Huang, Rongsheng Li, and Chi Wei. 2024a. A prompt construction method for the reverse dictionary task of large-scale language models. Engineering Applications of Artificial Intelligence, 133:108596. ↗
  30. Sicheng Tian, Shaobin Huang, Rongsheng Li, Chi Wei, and Ye Liu. 2024b. RDMTL: Reverse dictionary model based on multitask learning. Knowledge-Based Systems, 296:111869. ↗
  31. Hang Yan, Xiaonan Li, Xipeng Qiu, and Bocao Deng. 2020. BERT for Monolingual and Cross-Lingual Reverse Dictionary. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4329–4338, Online. find
  32. Fabio Massimo Zanzotto, Ioannis Korkontzelos, Francesca Fallucchi, and Suresh Manandhar. 2010. Estimating Linear Models for Compositional Distributional Semantics. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 1263–1271, Beijing, China. Coling 2010 Organizing Committee.
  33. Lei Zhang, Fanchao Qi, Zhiyuan Liu, Yasheng Wang, Qun Liu, and Maosong Sun. 2020. Multi-Channel Reverse Dictionary Model. Proceedings of the AAAI Conference on Artificial Intelligence, 34(01):312–319. find
  34. Ahmed Mukhtar Omar. 2008. Dictionary of Contemporary Arabic Language. Volume One, 1st Edition, Cairo, Egypt:2109. find
  35. Mohammad Taher Pilehvar. 2019. On the Importance of Distinguishing Word Meaning Representations: A Case Study on Reverse Dictionary Mapping. In find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alshammari-2024-ksaa-cad-shared-task-contemporary/