Elamur Research⌕
☰
home / papers / 2024 / ASOS at KSAA-CAD 2024: One Embedding is All You Need for Your Dictionary

ASOS at KSAA-CAD 2024: One Embedding is All You Need for Your Dictionary

Paper · 2024 · Serry Sibaee, Abdullah Alharbi, Samar Ahmad et al.

Summary

The paper addresses two Arabic semantic tasks from the KSAA-CAD 2024 shared task. The first is the Reverse Dictionary (RD), which maps a definition to the word it describes. The second is Word Sense Disambiguation (WSD), which picks the correct gloss for a word in a given context. The authors reverse the strategy of their earlier system. They keep a single feed-forward network design fixed and vary the text encoder, testing five sentence transformers trained on many languages alongside ArBERTv2, an Arabic BERT model whose token vectors are mean-pooled. For RD, the encoded gloss is regressed into the provided target embedding spaces (Electra, camelBERT-MSA, AraBERTv2). The training data was enlarged with roughly 84,000 Electra-embedded entries from the previous year. For WSD, the word and its context are encoded separately, then merged to predict a gloss embedding. Both systems placed first. The RD rank score of 0.06 is about twice as good as the previous year's winner, and WSD accuracy is 0.268. A manual audit of the dataset shows inconsistent definitions. The authors argue that dataset quality, more than model design, now limits progress.

Key points

  • The RD data has 31,372 train, 3,921 dev and 3,922 test samples. Train and dev were merged, so no dev results are reported.
  • The fixed RD network has four hidden layers. Their width starts at 8× the input size and halves at each layer down to the input size. It uses GELU, dropout of 0.2–0.4, AdamW and a learning rate of 1e-4.
  • The compared encoders are LaBSE, MiniLM-L12-v2, mpnet-base-v2, and distiluse-base-multilingual-cased v1 and v2. ArBERTv2 was also tested, and this monolingual MSA model gave the strongest embeddings.
  • The WSD data has 22,404, 2,801 and 2,801 entries, plus a dictionary of 15,865 glosses. The best run used lr 1e-3 for 75 epochs and reached accuracy 0.268 and mrr@2 0.297.
  • In a sample of 100 random entries per split, the audit found three problems:
    • definitions that only describe morphological forms;
    • narrow or field-specific senses given without naming the field;
    • definitions that contain the word being defined.

Interesting ideas

  • I1. Hold the downstream network constant and swap only the encoder, so any change in score can be credited to the representation alone.
  • I2. WSD is framed as regression: the model predicts a vector in the gloss embedding space and then retrieves the closest gloss, so no per-word classifier is needed.
  • I3. A reverse dictionary maps a description of a meaning to the word that best expresses it.
  • I4. The authors found that a clean monolingual encoder beat multilingual sentence transformers. They attribute this to noise from mixing many languages in the multilingual models.
  • I5. A small random audit of each data split exposed systematic flaws in how definitions were written. The authors suggest following classical definition rules and naming the domain of each sense.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluations of gparse could keep the parser head and training schedule fixed while changing the encoder, giving a clean comparison of encoders.
  • I2 (world): Sense choice against WordNet glosses could use the same nearest-gloss lookup, and the retrieved gloss itself would show why that sense was chosen.
  • I3 (back): During generation from the meaning graph, a node's description could be matched against lexicon entries to choose the surface word.
  • I4 (tree): When choosing the pretrained backbone for gparse, an English-only model is worth testing before defaulting to a multilingual one.
  • I5 (world): Compiled glosses from WordNet/Wikidata could be spot-checked the same way, with entries flagged when they are circular or only describe word forms.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓King Abdulaziz UniversityJeddah · Saudi Arabia31 works · 508 citations🎓Prince Sultan UniversityRiyadh · Saudi Arabia17 works · 120 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Serry Sibaee, Abdullah Alharbi, Samar Ahmad, Omer Nacar, Anis Koubaa and Lahouari Ghouti. 2024. ASOS at KSAA-CAD 2024: One Embedding is All You Need for Your Dictionary. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.77

@inproceedings{sibaee2024asos,
  title = {{ASOS at KSAA-CAD 2024: One Embedding is All You Need for Your Dictionary}},
  author = {Serry Sibaee, Abdullah Alharbi, Samar Ahmad, Omer Nacar, Anis Koubaa and Lahouari Ghouti},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.77},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.77}
}

References

21 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  2. Rawan Al-Matham, Waad Alshammari, Abdulrahman AlOsaimy, Sarah Alhumoud, Asma Wazrah, Afrah Altamimi, Halah Alharbi, and Abdullah Alaifi. 2023. KSAA-RD shared task: Arabic reverse dictionary. In Proceedings of ArabicNLP 2023, pages 450–460, Singapore (Hybrid). Association for Computational Linguistics.
  3. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. ↗
  4. Wissam Antoun, Fady Baly, and Hazem Hajj. 2021. AraELECTRA: Pre-training text discriminators for Arabic language understanding. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 191–195, Kyiv, Ukraine (Virtual). Association for Computational Linguistics. ↗
  5. Edoardo Barba, Luigi Procopio, and Roberto Navigli. 2021. Consec: Word sense disambiguation as continuous sense comprehension. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1492–1503. find
  6. Nihed Bendahman, Julien Breton, Lina Nicolaieff, Mokhtar Boumedyen Billami, Christophe Bortolaso, and Youssef Miloudi. 2022. Bl. research at semeval-2022 task 1: Deep networks for reverse dictionary using embeddings and lstm autoencoders. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 94–100. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. Preprint, arXiv:1810.04805. ↗
  8. Ahmed Elbakry, Mohamed Gabr, Muhammad El-Nokrashy, and Badr AlKhamissi. 2023. Rosetta stone at KSAA-RD shared task: A hop from language modeling to word–definition alignment. In Proceedings of ArabicNLP 2023, pages 477–482, Singapore (Hybrid). Association for Computational Linguistics.
  9. AbdelRahim Elmadany, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2022. Orca: A challenging benchmark for arabic language understanding. arXiv preprint arXiv:2212.10758. of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2151–2156. ↗
  10. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  11. Rakia Saidi and Fethi Jarray. 2023. Sentence transformers and distilbert for arabic word sense induction. In ICAART (3), pages 1020–1027. find
  12. Serry Sibaee, Samar Ahmad, Ibrahim Khurfan, Vian Sabeeh, Ahmed Bahaaulddin, Hanan Belhaj, and Abdullah Alharbi. 2023. Qamosy at Arabic reverse dictionary shared task: Semi decoder architecture for reverse dictionary with SBERT encoder. In Proceedings of ArabicNLP 2023, pages 467–471, Singapore (Hybrid). Association for Computational Linguistics.
  13. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. Language-agnostic bert sentence embedding. Preprint, arXiv:2007.01852. ↗
  14. Sushrut Thorat and Varad Choudhari. 2016. Implementing a reverse dictionary, based on word definitions, using a node-graph architecture. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2797–2806. ↗
  15. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine (Online). Association for Computational Linguistics.
  16. Jiajia Wang, Jimmy X. Huang, Xinhui Tu, Junmei Wang, Angela J. Huang, Md Tahmid Rahman Laskar, and Amran Bhuiyan. 2024. Utilizing bert for information retrieval: Survey, applications, resources, and challenges. Preprint, arXiv:2403.00784. ↗
  17. A.M. Jurjānı̄. 1969. Kitab al-tarifat: a book of definitions. Librairie du Liban. find
  18. Sanaa Kaddoura and Reem Nassar. 2024. Enhancedbert: A feature-rich ensemble model for arabic word sense disambiguation with statistical analysis and optimized data collection. Journal of King Saud University-Computer and Information Sciences, 36(1):101911. ↗
  19. Oscar Méndez, Hiram Calvo, and Marco A Moreno-Armendáriz. 2013. A reverse dictionary based on semantic analysis using wordnet. In Advances in Artificial Intelligence and Its Applications: 12th Mexican International Conference on Artificial Intelligence, MICAI 2013, Mexico City, Mexico, November 24-30, 2013, Proceedings, Part I 12, pages 275–285. Springer. find
  20. Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. 2024. A comprehensive overview of large language models. Preprint, arXiv:2307.06435. ↗
  21. Mohammad Taher Pilehvar. 2019. On the importance of distinguishing word meaning representations: A case study on reverse dictionary mapping. In Proceedings find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sibaee-2024-asos-ksaa-cad-2024-one/