Elamur Research⌕
☰
home / papers / 2023 / KSAA-RD Shared Task: Arabic Reverse Dictionary

KSAA-RD Shared Task: Arabic Reverse Dictionary

Paper · 2023 · Rawan Al-Matham, Waad Alshammari, Abdulrahman AlOsaimy et al.

Summary

The paper reports on KSAA-RD, the first shared task on Arabic reverse dictionaries. A reverse dictionary takes a description of a meaning and returns the matching word, which helps with tip-of-the-tongue cases. There were two tracks. In the monolingual RD track, systems read an Arabic gloss. In the cross-lingual CLRD track, they read an English gloss. In both tracks the system must output a vector close to the Arabic target word's embedding.

The organisers built the data from existing Arabic and English dictionaries. They also manually linked Arabic and English entries. Each entry comes with two kinds of target vectors, SGNS and ELECTRA, and systems were scored by cosine similarity, MSE and a rank measure.

Four teams wrote system papers:

  • Three teams used BERT-style encoders.
  • One team used a rule-based dictionary method.

The best system was an ensemble of fine-tuned Arabic BERT models from Rosetta Stone. It reached rank 24.20 on RD and 12.70 on CLRD. The task gives Arabic its first public benchmark for meaning-to-word lookup, and all datasets are released openly.

Key points

  • Participation: 31 teams registered, producing 39 valid submissions and 4 accepted system papers. The data is on GitHub (Waadtss/ArReverseDictionary).
  • Metrics: systems were scored by cosine, MSE and rank, with rank as the main metric (lower is better).
    • On the test set, the ELECTRA baseline scored rank 31.87 for RD and 40.72 for CLRD.
  • Winning system (Rosetta Stone):
    • The team fine-tuned camelBERT-MSA, camelBERT-Mix, MARBERTv2 and AraBERTv2.
    • For RD, averaging the outputs of camelBERT-MSA and MARBERTv2 gave rank 24.20 with ELECTRA targets.
    • For CLRD, the team first machine-translated the English glosses into Arabic and reached rank 12.70.
  • Other teams:
    • Qamosy encoded glosses with SBERT and then trained a Simi-Decoder (RD rank 28.10).
    • Abed used a modified multilingual BERT together with synonym replacement, insertion, deletion and swap augmentation (RD rank 28.50).
  • Rule-based system (UWB): UWB built a dictionary from the dataset. It trailed the baseline on RD (43.80) but slightly beat the SGNS baseline on CLRD (48.87 vs 49.95).

Interesting ideas

  • I1. The task asks systems to regress the target word's embedding and then scores how highly the true word ranks, instead of training a closed-vocabulary classifier, so unseen words remain reachable.
  • I2. The benchmark provides two target spaces, static SGNS and contextual ELECTRA, and team rankings shift between them, so the choice of target representation is itself a variable being tested.
  • I3. Rank and MSE disagree in the tables: ELECTRA targets show much larger MSE yet better rank, which warns against relying on a single distance number.
  • I4. Rosetta Stone handled the cross-lingual track by translating the English gloss into Arabic first, then reusing the monolingual model, which avoided training a separate bilingual mapper.
  • I5. Averaging embeddings from diverse encoders won, and a plain dictionary-based method still beat a transformer baseline on one track, showing that model diversity and simple strong baselines both matter.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): a gloss→vector→lemma lookup could map descriptive phrases in a text ("the organ you see with") onto WordNet entries in the ontology, with the ranked neighbours kept as an inspectable explanation.
  • I2 (learn): evaluations of lexical components should be run against more than one target representation, so a gain that holds in only one space is flagged rather than trusted.
  • I3 (learn): pair every vector-distance score with a retrieval-style rank check, which acts as a sanity control on whether the meaning is actually recovered.
  • I4 (back): when a new language direction is needed, normalising the input into the language where a strong pipeline already exists may beat building a dedicated cross-language component.
  • I5 (learn): this supports combining several witness parsers for silver data and always keeping a rule-based baseline as a negative control in evaluations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Rawan Al-Matham, Waad Alshammari, Abdulrahman AlOsaimy, Sarah Alhumoud, Asma Wazrah, Afrah Altamimi, Halah Alharbi and Abdullah Alaifi. 2023. KSAA-RD Shared Task: Arabic Reverse Dictionary. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.39

@inproceedings{almatham2023ksaa,
  title = {{KSAA-RD Shared Task: Arabic Reverse Dictionary}},
  author = {Rawan Al-Matham, Waad Alshammari, Abdulrahman AlOsaimy, Sarah Alhumoud, Asma Wazrah, Afrah Altamimi, Halah Alharbi and Abdullah Alaifi},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.39},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.39}
}

References

37 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim A. Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic. CoRR, abs/2101.01785. arXiv: 2101.01785. ↗
  2. Aarchi Agrawal, Athulkumar R, K S Ashin Shanly, Kavita Vaishnaw, and Mayank Singh. 2021. Reverse Dictionary Using an Improved CBoW Model. In Proceedings of the 3rd ACM India Joint International Conference on Data Science & Management of Data (8th ACM IKDD CODS & 26th COMAD), page 420, New York, NY, USA. Association for Computing Machinery. find
  3. Hawra Aljasim, Sultanah Alghurabi, Halah Alharbi, Abdulrahman Alosaimy, Afrah Altamimi, Asyah Almutawa, and Abdullah Alfifi. 2022. Toward an Automatic Semantic Order of Word Meanings Based on Lexically Tagged Corpus. Manuscript submitted for publication. find
  4. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based Model for Arabic Language Understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North, pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  6. Dominique Dutoit and Pierre Nugues. 2002. A lexical database and an algorithm to find words from definitions | Proceedings of the 15th European Conference on Artificial Intelligence. In ECAI’02: Proceedings of the 15th European Conference on Artificial Intelligence. find
  7. Ahmed ElBakry, Mohamed Gabr, Muhammad ElNokrashy, and Badr AlKhamissi. 2023. Rosetta Stone at KSAA-RD Shared Task: A Hop From Language Modeling To Word — Definition Alignment. In ˙Ilknur El-Kahlout and Kemal Oflazer. 2004. Use of Wordnet for Retrieving Words from Their Meanings. In The Global Wordnet confrence, pages 118–123.
  8. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. AraELECTRA: Pre-Training Text Discriminators for Arabic Language Understanding. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 191–195, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. ↗
  9. Roger Brown and David McNeill. 1966. The “tip of the tongue” phenomenon. Journal of Verbal Learning and Verbal Behavior, 5(4):325–337. find
  10. Hiram Calvo, Oscar Méndez, and Marco A. Moreno-Armendáriz. 2016. Integrated concept blending with vector space models. Computer Speech & Language, 40:79–96. ↗
  11. Kawin Ethayarajh. 2019. How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 55–65, Hong Kong, China. Association for Computational Linguistics. find
  12. Michael A. Hedderich, Andrew Yates, Dietrich Klakow, and Gerard de Melo. 2019. Using Multi-Sense Vector Embeddings for Reverse Dictionaries. In Proceedings of the 13th International Conference on Computational Semantics - Long Papers, pages 247–258, Gothenburg, Sweden. Association for Computational Linguistics. ↗
  13. Felix Hill, Kyunghyun Cho, Anna Korhonen, and Yoshua Bengio. 2016. Learning to Understand Phrases by Embedding the Dictionary. Transactions of the Association for Computational Linguistics, 4:17–30. ↗
  14. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. CoRR, abs/2003.10555. arXiv: 2003.10555. ↗
  15. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models. In Proceedings of the Sixth Arabic Natural Language Processing H. Vance Crawford and Joel Crawford. 1997. Reverse electronic dictionary using synonyms to expand search capabilities. Workshop, pages 92–104, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. Papers), pages 2151–2156, Minneapolis, Minnesota. Association for Computational Linguistics. king Salman global academy for Arabic language KSAA. 2023. Alriyadh Dictionary.
  16. Abed Qaddoumi. 2023. Abed at KSAA-RD Shared Task: Enhancing Arabic Word Embedding with Modified BERT Multilingual. In Oscar Méndez, Hiram Calvo, and Marco A. Moreno-Armendáriz. 2013. A Reverse Dictionary Based on Semantic Analysis Using WordNet. In Félix Castro, Alexander Gelbukh, and Miguel González, editors, Advances in Artificial Intelligence and Its Applications, pages 275–285, Berlin, Heidelberg. Springer.
  17. Timothee Mickus, Kees Van Deemter, Mathieu Constant, and Denis Paperno. 2022. Semeval-2022 Task 1: CODWOE – Comparing Dictionaries and Word Embeddings. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1–14, Seattle, United States. Association for Computational Linguistics.
  18. Fanchao Qi, Lei Zhang, Yanhui Yang, Zhiyuan Liu, and Maosong Sun. 2020. WantWords: An Open-source Online Reverse Dictionary System. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 175–181, Online. Association for Computational Linguistics. find
  19. Jorge C. Reyes Magaña, Gemma Bel Enguix, Gerardo Sierra, and Helena Gómez-Adorno. 2019. Designing an Electronic Reverse Dictionary Based on Two Word Association Norms of English Language. In Electronic lexicography in the 21st century. Proceedings of the eLex 2019 conference. 1-3 October 2019, Sintra, Portugal, 2019, págs. 865-880, pages 865–880. Lexical Computing. find
  20. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781 [cs]. ↗
  21. Ryan Shaw, Anindya Datta, Debra VanderMeer, and Kaushik Dutta. 2013. Building a Scalable Database-Driven Reverse Dictionary. IEEE Transactions on Knowledge and Data Engineering, 25(3):528–540. find
  22. Yuya Morinaga and Kazunori Yamaguchi. 2018. Improvement of Reverse Dictionary by Tuning Word Vectors and Category Inference. In Robertas Damaševičius and Giedrė Vasiljevienė, editors, Information and Software Technologies, pages 533– 545, Cham. Springer International Publishing. find
  23. Serry Sibaee, Samar Ahmad, Ibrahim Khurfan, Vian Sabeeh, Ahmed Bahaaulddin, Hanan M. Belhaj, and Abdullah I. Alharbi. 2023. Qamosy at KSAA-RD shared task: Semi Decoder Architecture for Reverse Dictionary with SBERT Encoder. In Yuya Morinaga and Kazunori Yamaguchi. 2020. Improvement of Neural Reverse Dictionary by Using Cascade Forward Neural Network. Journal of Information Processing, 28:715–723. find
  24. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL Tools: An Open Source Python Toolkit for Arabic Natural Language Processing. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 7022– 7032, Marseille, France. European Language Resources Association.
  25. Bushra Siddique and Mirza Mohd Sufyan Beg. 2019. A Review of Reverse Dictionary: Finding Words from Concept Description. In Manish Prateek, Durgansh Sharma, Rajeev Tiwari, Rashmi Sharma, Kamal Kumar, and Neeraj Kumar, editors, Next Generation Computing Technologies on Computational Intelligence, pages 128–139, Singapore. Springer. find
  26. Bilac Slaven, Wataru Watanabe, Taiichi Hashimoto, Takenobu Tokunaga, and Hozumi Tanaka. 2004. Dictionary search based on the target word description. In pages 556–559. find
  27. Ahmed Mukhtar Omar. 2008. Dictionary of Contemporary Arabic Language. Volume One, 1st Edition, Cairo, Egypt:2109. find
  28. Abu Bakr Soliman, Kareem Eissa, and Samhaa R. El-Beltagy. 2017. AraVec: A set of Arabic Word Embedding Models for use in Arabic NLP. Procedia Computer Science, 117:256–265. find
  29. Mohammad Taher Pilehvar. 2019. On the Importance of Distinguishing Word Meaning Representations: A Case Study on Reverse Dictionary Mapping. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Stephen Taylor. 2023. UWB at KSAA-RD shared task: Computing the meaning of a gloss. In Sushrut Thorat and Varad Choudhari. 2016. Implementing a Reverse Dictionary, based on word definitions, using a Node-Graph Architecture. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2797–2806, Osaka, Japan. find
  30. The COLING 2016 Organizing Committee. find
  31. William Timkey and Marten van Schijndel. 2021. All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4527–4546, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  32. Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. ArXiv, abs/1706.03762. ↗
  33. Hang Yan, Xiaonan Li, Xipeng Qiu, and Bocao Deng. 2020. BERT for Monolingual and Cross-Lingual Reverse Dictionary. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4329–4338, Online. Association for Computational Linguistics. find
  34. Fabio Massimo Zanzotto, Ioannis Korkontzelos, Francesca Fallucchi, and Suresh Manandhar. 2010. Estimating Linear Models for Compositional Distributional Semantics. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 1263–1271, Beijing, China. Coling 2010 Organizing Committee.
  35. Lei Zhang, Fanchao Qi, Zhiyuan Liu, Yasheng Wang, Qun Liu, and Maosong Sun. 2020. Multi-Channel Reverse Dictionary Model. Proceedings of the AAAI Conference on Artificial Intelligence, 34(01):312–319. find
  36. Michael Zock and Slaven Bilac. 2004. Word Lookup on the Basis of Associations : from an Idea to a Roadmap. In Proceedings of the Workshop on Enhancing and Using Electronic Dictionaries, pages 29–35, Geneva, Switzerland. COLING. find
  37. Michael Zock and Didier Schwab. 2008. Lexical access based on underspecified input. In Coling 2008: Proceedings of the Workshop on Cognitive Aspects of the Lexicon (COGALEX 2008), pages 9– 17, Manchester, United Kingdom. Coling 2008 Organizing Committee. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/matham-2023-ksaa-rd-shared-task-arabic/