Elamur Research⌕
☰
home / papers / 2024 / ArabicNLU 2024: The First Arabic Natural Language Understanding Shared Task

ArabicNLU 2024: The First Arabic Natural Language Understanding Shared Task

Paper · 2024 · Mohammed Khalilia, Sanad Malaysha, Reem Suwaileh et al.

Summary

ArabicNLU 2024 was the first shared task on lexical disambiguation in Arabic. The organizers argue that Arabic's rich morphology makes many words ambiguous, and that progress has been slow mainly because annotated data is scarce. The task had two subtasks. In Word Sense Disambiguation (WSD), systems choose the right sense of a word in context. In Location Mention Disambiguation (LMD), systems link a place name in a post to the correct toponym in a gazetteer.

For WSD, the organizers used SALMA, a corpus of about 34k tokens. Its senses come from two lexicons, and each token's candidate senses carry relatedness scores. For LMD, they used IDRISI-DA, a set of disaster-related posts from X. Participants had to use fixed sense and location inventories but could bring in any external resources, including LLMs. Scoring ran on CodaLab, and LMD was ranked by MRR@k.

Of 38 registered teams, only three reached the final evaluation. The best WSD result (77.8%) stayed below the supervised baseline. In LMD, an LLM-plus-GeoPy pipeline reached 95% MRR@1. The overview concludes that general generative models trail supervised classifiers on Arabic, and that tools centred on English often need a translation step to work well.

Key points

  • SALMA has about 34k sense-annotated MSA tokens. Its senses come from Al-Ghani Al-Zaher and the Contemporary Arabic Dictionary, and every candidate sense is scored for relatedness to the word in context.
  • IDRISI-DA is the only public Arabic LMD dataset. It holds 3,893 annotations covering 763 unique location mentions, drawn from X posts about events such as floods, bombings and COVID-19.
  • Of 38 registered teams, 4 submitted 40 test entries and 3 took part in the final phase. The top WSD accuracy was 77.8%, while Figure 1 reports a baseline of about 84%.
  • LMD results by MRR@1 were:
    • OSM baseline: 0.5724.
    • UPAYA (OSM candidate retrieval followed by Cohere rerank-multilingual-v2.0): 0.5994.
    • REMATCHKA (Llama3 renders the mention or its likely country in English, then GeoPy resolves it): 0.9497.
  • The authors list several limitations:
    • SALMA covers MSA only, has no dialects and handles single-word lemmas only.
    • IDRISI-DA under-represents fine-grained and temporary locations.
    • X API restrictions limit how useful IDRISI-DA is for social-media research.

Interesting ideas

  • I1. The task treats disambiguation as ranking and scores it with MRR@k, avoiding plain accuracy or distance-based scores that need granularity-specific thresholds; this rewards systems that put the right candidate near the top.
  • I2. SALMA marks every candidate sense with a degree of relatedness instead of a single gold label, so partly-right choices can be told apart from fully wrong ones.
  • I3. One LMD team used a two-stage pipeline: cheap retrieval of gazetteer candidates first, then a stronger model reranks the shortlist.
  • I4. The inventories were frozen at test time while external resources stayed allowed, so systems remained comparable without banning outside knowledge.
  • I5. A plain supervised baseline outperformed the LLM-based WSD entries, a reminder that a strong simple reference is essential.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): Linking entities to Wikidata or WordNet synsets could be scored with MRR@k over the candidate list, which also shows how close a near-miss came.
  • I2 (frames): Choosing a VerbNet class could be stored and evaluated with graded relatedness, so neighbouring classes earn partial credit and the explanation can show which alternatives were close.
  • I3 (world): Grounding entities could first fetch a shortlist from the knowledge graph by name, then rerank it with context features from the frames, keeping each stage inspectable.
  • I4 (learn): Evaluation can lock the frame and sense inventories while letting teacher LLMs and witnesses vary, so score changes reflect the method rather than label drift.
  • I5 (learn): Every evaluation should include a trained non-generative baseline next to the negative controls, so LLM-teacher gains are measured against a real bar.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States430 works · 424,363 citations🎓Hamad bin Khalifa UniversityDoha · Qatar141 works · 14,175 citations🎓Qatar UniversityDoha · Qatar48 works · 1,026 citations🎓Birzeit UniversityBirzeit · Palestine36 works · 798 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Mohammed Khalilia, Sanad Malaysha, Reem Suwaileh, Mustafa Jarrar, Alaa Aljabari, Tamer Elsayed and Imed Zitouni. 2024. ArabicNLU 2024: The First Arabic Natural Language Understanding Shared Task. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.30

@inproceedings{khalilia2024arabicnlu,
  title = {{ArabicNLU 2024: The First Arabic Natural Language Understanding Shared Task}},
  author = {Mohammed Khalilia, Sanad Malaysha, Reem Suwaileh, Mustafa Jarrar, Alaa Aljabari, Tamer Elsayed and Imed Zitouni},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.30},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.30}
}

References

60 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Reem Abdel-Salam. 2024. Rematchka at arabicnlu shared task: Evaluating large language models for arabicword sense and location sense disambiguation. In The Second Arabic Natural Language Processing Conference (ArabicNLP 2024) Part of ACL 2024. find
  2. Mohammed Alaeddine Abderrahim and Mohammed El Amine Abderrahim. 2022. Arabic word sense disambiguation for information retrieval. ACM Trans. Asian Low Resour. Lang. Inf. Process., 21(4):69:1– 69:19. ↗
  3. Muhammad Abdul-Mageed, Hassan Alhuzali, and Mohamed Elaraby. 2018. You tweet what you speak: A city-level dataset of Arabic dialects. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
  4. Muhammad Abdul-Mageed, AbdelRahim Elmadany, Chiyu Zhang, El Moatez Billah Nagoudi, Houda Bouamor, and Nizar Habash. 2023. NADI 2023: The fourth nuanced Arabic dialect identification shared task. In Proceedings of ArabicNLP 2023, pages 600– 613, Singapore (Hybrid). Association for Computational Linguistics.
  5. Muhammad Abdul-Mageed, AbdelRahim A. Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: deep bidirectional transformers for arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 7088–7105. Association for Computational Linguistics. ↗
  6. Miuru Abeysiriwardana and Deshan Sumanathilaka. 2024. A survey on lexical ambiguity detection and word sense disambiguation. CoRR, abs/2403.16129. ↗
  7. Abdul-Ghani Abul-Azm. 2014. Al-ghani al-zaher dictionary. Rabat: Al-Ghani Publishing Institution. find
  8. AI@Meta. 2024. Llama 3 model card. find
  9. Moustafa Al-Hajj and Mustafa Jarrar. 2021a. Arabgloss-bert: Fine-tuning bert on context-gloss pairs for wsd. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 40–48, Online. INCOMA Ltd. find
  10. Moustafa Al-Hajj and Mustafa Jarrar. 2021b. Lu-bzu at semeval-2021 task 2: Word2vec and lemma2vec performance in arabic word-in-context disambiguation. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 748– 755, Online. Association for Computational Linguistics.
  11. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15. ↗
  12. Vedanth Baiju. 2022. Word sense disambiguation in the domain of sentiment analysis through deep learning. find
  13. Wided Bakari, Mabrouka Ben-Sghaier, and Mahmoud Neji. 2021. Implementing an arabic question answering system using conceptual graphs. In Hybrid Intelligent Systems - 21st International Conference on Hybrid Intelligent Systems (HIS 2021), December 14-16, 2021, volume 420 of Lecture Notes in Networks and Systems, pages 295–304. Springer. find
  14. Sylvio Barbon Junior, Paolo Ceravolo, Sven Groppe, Mustafa Jarrar, Samira Maghool, Florence Sèdes, Soror Sahri, and Maurice Van Keulen. 2024. Are Large Language Models the New Interface for Data Pipelines? In Proceedings of the International Workshop on Big Data in Emergent Distributed Environments, BiDEDE ’24, New York, NY, USA. Association for Computing Machinery. ↗
  15. Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: on meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 5185–5198. Association for Computational Linguistics. ↗
  16. M Soha Eid, Almoataz B Al-Said, Nayer M Wanas, Mohsen A Rashwan, and Nadia H Hegazy. 2010. Comparative study of rocchio classifier applied to supervised wsd using arabic lexical samples. In Proceedings of the tenth conference of language engeneering (SEOLEC’2010), Cairo, Egypt. find
  17. Mohammed El-Razzaz, Mohamed Waleed Fakhr, and Fahima A Maghraby. 2021. Arabic gloss wsd using bert. Applied Sciences, 11(6):2567. ↗
  18. Bilel Elayeb. 2019. Arabic word sense disambiguation: a review. Artif. Intell. Rev., 52(4):2475–2532. ↗
  19. Karim El Haff, Mustafa Jarrar, Tymaa Hammouda, and Fadi Zaraket. 2022. Curras + baladi: Towards a levantine corpus. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2022), Marseille, France. ↗
  20. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Ziche Liu, Zhiyi Zhang, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, and Jinchao Xu. 2023. Acegpt, localizing large language models in arabic. CoRR, abs/2309.12053. ↗
  21. Mustafa Jarrar. 2021. The arabic ontology - an arabic wordnet with ontologically clean content. Applied Ontology Journal, 16(1):1–26. ↗
  22. Mustafa Jarrar, Muhammad Abdul-Mageed, Mohammed Khalilia, Bashar Talafha, AbdelRahim Elmadany, Nagham Hamad, and Alaa’ Omar. 2023a. WojoodNER 2023: The First Arabic Named Entity Recognition Shared Task. In Proceedings of the 1st Arabic Natural Language Processing Conference (ArabicNLP), Part of the EMNLP 2023, pages 748– 758. ACL.
  23. Mustafa Jarrar, Ahmet Birim, Mohammed Khalilia, Mustafa Erden, and Sana Ghanem. 2023b. ArBanking77: Intent Detection Neural Model and a New Dataset in Modern and Dialectical arabic. In Proceedings of the 1st Arabic Natural Language Processing Conference (ArabicNLP), Part of the EMNLP 2023, pages 276–287. ACL.
  24. Mustafa Jarrar, Nagham Hamad, Mohammed Khalilia, Bashar Talafha, and Muhammad Elmadany, Abdel-Rahim Abdul-Mageed. 2024. WojoodNER 2024: The Second Arabic Named Entity Recognition Shared Task. In Proceedings of the Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok, Thailand. Association for Computational Linguistics.
  25. Mustafa Jarrar and Tymaa Hasanain Hammouda. 2024. Qabas: An Open-Source Arabic Lexicographic Database. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 13363–13370, Torino, Italy. ELRA and ICCL.
  26. Mustafa Jarrar, Mohammed Khalilia, and Sana Ghanem. 2022. Wojood: Nested Arabic Named Entity Corpus and Recognition using BERT. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2022), Marseille, France.
  27. Mustafa Jarrar, Sanad Malaysha, Tymaa Hammouda, and Mohammad Khalilia. 2023c. SALMA: Arabic Sense-annotated Corpus and WSD Benchmarks. In Proceedings of the 1st Arabic Natural Language Processing Conference (ArabicNLP), Part of the EMNLP 2023, pages 359–369. ACL.
  28. Mustafa Jarrar, Fadi Zaraket, Tymaa Hammouda, Daanish Masood Alavi, and Martin Waahlisch. 2023d. Lisan: Yemeni, irqi, libyan, and sudanese arabic dialect copora with morphological annotations. In The 20th IEEE/ACS International Conference on Computer Systems and Applications (AICCSA). IEEE. find
  29. Zongcheng Ji, Aixin Sun, Gao Cong, and Jialong Han. 2016. Joint recognition and linking of fine-grained locations from tweets. In Proceedings of the 25th International Conference on World Wide Web, WWW ’16, page 1271–1281, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee. find
  30. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171–4186.
  31. Panagiotis Kouris, Georgios Alexandridis, and Andreas Stafylopatis. 2021. Abstractive text summarization: Enhancing sequence-to-sequence models using word sense disambiguation and semantic content generalization. Comput. Linguistics, 47(4):813–859. find
  32. Anastasia Kritharoula, Maria Lymperaiou, and Giorgos Stamou. 2023. Large language models and multimodal retrieval for visual word sense disambiguation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 13053– 13077. Association for Computational Linguistics. ↗
  33. Guoliang Li, Jun Hu, Jianhua Feng, and Kian-lee Tan. 2014. Effective location identification from microblogs. In 2014 IEEE 30th International Conference on Data Engineering, pages 880–891, Chicago, IL, USA. Institute of Electrical and Electronics Engineers (IEEE). find
  34. Haonan Li, Minghan Wang, Timothy Baldwin, Martin Tomko, and Maria Vasardani. 2019. UniMelb at SemEval-2019 task 12: Multi-model combination for toponym resolution. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 1313–1318, Minneapolis, Minnesota, USA. Association for Computational Linguistics. find
  35. Sanad Malaysha, Mo El-Haj, Saad Ezzini, Mohammad Khalilia, Mustafa Jarrar, Sultan Nasser, Ismail Berrada, and Houda Bouamor. 2024. AraFinNLP 2024: The First Arabic Financial NLP Shared Task. In Proceedings of the Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok, Thailand. Association for Computational Linguistics. ↗
  36. Sanad Malaysha, Mustafa Jarrar, and Mohammad Khalilia. 2023. Context-gloss augmentation for improving arabic target sense verification. In Proceedings of the 12th International Global Wordnet Conference (GWC2023). Global Wordnet Association.
  37. Dastan Hussen Maulud, Subhi RM Zeebaree, Karwan Jacksi, Mohammed A Mohammed Sadeeq, and Karzan Hussein Sharif. 2021. State of art for semantic analysis of natural language processing. Qubahan academic journal, 1(2):21–28. ↗
  38. Stuart E. Middleton, Giorgos Kordopatis-Zilos, Symeon Papadopoulos, and Yiannis Kompatsiaris. 2018. Location extraction from social media: Geoparsing, location disambiguation, and geotagging. ACM Transactions on Information Systems, 36(4):1–27. ↗
  39. George A Miller, Richard Beckwith, Christiane Fellbaum, Derek Gross, and Katherine J Miller. 1990. Introduction to wordnet: An on-line lexical database. International journal of lexicography, 3(4):235–244. ↗
  40. Sangita S Modi and Sudhir B Jagtap. 2018. Web page classification using wsd and yago and ontology. In 2018 3rd International Conference on Communication and Electronics Systems (ICCES), pages 887– 891. IEEE. find
  41. Ahmed Mukhtar Omar. 2008. Contemporary arabic dictionary.(i1). World of Books, Cairo, Egypt. Retrieval Date, 14(8):2020. find
  42. OpenAI. 2023. GPT-4 technical report. abs/2303.08774. ↗
  43. CoRR, Lizhen Ou, Yiping Yao, Xueshan Luo, Xinmeng Li, and Kai Chen. 2023. Contextad: Context-aware acronym disambiguation with siamese BERT network. Int. J. Intell. Syst., 2023:1–14. ↗
  44. Kathiravan Pannerselvam, Saranya Rajiakodi, Sajeetha Thavareesan, Sathiyaraj Thangasamy, and Kishore Ponnusamy. 2024. Setfit: A robust approach for offensive content detection in tamil-english code-mixed conversations using sentence transfer fine-tuning. In Proceedings of the Fourth Workshop on Speech, Vision, and Language Technologies for Dravidian Languages, pages 35–42. find
  45. Gayatri Potey, Rucha Jadhav, Kushagra Shroff, Anish Gore, D Phalke, and J Shimpi. 2020. Fake news detection and sentiment analysis in twitter. International Journal for Research in Applied Science & Engineering Technology (IJRASET), 8:72–75. find
  46. Alessandro Raganato, Yves Scherrer, and Jörg Tiedemann. 2020. An evaluation benchmark for testing the word sense disambiguation capabilities of machine translation systems. In Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020, pages 3668–3675. European Language Resources Association. find
  47. Pawan Kumar Rajpoot, Ashvini Kumar Jindal, and Ankur Parikh. 2024. Upaya at arabicnlu shared-task: Arabic lexical disambiguation using large language models. In The Second Arabic Natural Language Processing Conference (ArabicNLP 2024) Part of ACL 2024.
  48. Nathan Schneider, Behrang Mohit, Kemal Oflazer, and Noah A. Smith. 2012. Coarse lexical semantic annotation with supersenses: An arabic case study. In The 50th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, July 8-14, 2012, Jeju Island, Korea - Volume 2: Short Papers, pages 253–258. The Association for Computer Linguistics.
  49. Yujia Sun and Jan Platoš. 2023. Attention-based stacked bidirectional long short-term memory model for word sense disambiguation. ACM Transactions on Asian and Low-Resource Language Information Processing. ↗
  50. Reem Suwaileh, Tamer Elsayed, and Muhammad Imran. 2023a. IDRISI-D: arabic and english datasets and benchmarks for location mention disambiguation over disaster microblogs. In Proceedings of Arabic-NLP 2023, Singapore (Hybrid), December 7, 2023, pages 158–169. Association for Computational Linguistics.
  51. Reem Suwaileh, Muhammad Imran, and Tamer Elsayed. 2023b. IDRISI-RA: The first Arabic location mention recognition dataset of disaster tweets. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16298–16317, Toronto, Canada. Association for Computational Linguistics. deep learning pipeline for fine-grained location recognition and linking in tweets. In The World Wide Web Conference, WWW 2019, pages 3391–3397, San Francisco, CA, USA. ACM.
  52. Vikas Yadav, Egoitz Laparra, Ti-Tai Wang, Mihai Surdeanu, and Steven Bethard. 2019. University of Arizona at SemEval-2019 task 12: Deep-affix named entity recognition of geolocation entities. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 1319–1323, Minneapolis, Minnesota, USA. Association for Computational Linguistics. find
  53. Wei Zhang and Judith Gelernter. 2014. Geocoding location expressions in Twitter messages: A preference learning method. Journal of Spatial Information Science, 2014(9):37–70. ↗
  54. Tasneem Wael, Eman Elrefai, Mohamed Makram, Sahar Selim, and Ghada Khoriba. 2024. Pirates at arabicnlu2024: Enhancing arabic word sense disambiguation using transformer-based approaches. In The Second Arabic Natural Language Processing Conference (ArabicNLP 2024) Part of ACL 2024.
  55. Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. 2023. Openchat: Advancing open-source language models with mixed-quality data. CoRR, abs/2309.11235. ↗
  56. Jimin Wang and Yingjie Hu. 2019. Are we there yet? evaluating state-of-the-art neural network based geoparsers using eupeg as a benchmarking platform. In Proceedings of the 3rd ACM SIGSPATIAL International Workshop on Geospatial Humanities, GeoHumanities ’19, New York, NY, USA. Association for Computing Machinery. ↗
  57. Xiaobin Wang, Chunping Ma, Huafei Zheng, Chu Liu, Pengjun Xie, Linlin Li, and Luo Si. 2019. DM_NLP at SemEval-2018 task 12: A pipeline system for toponym resolution. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 917–923, Minneapolis, Minnesota, USA. Association for Computational Linguistics. find
  58. Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Mohammed El-Bachouti, Robert Belvin, and Ann Houston. 2013. Ontonotes release 5.0 ldc2013t19. Philadelphia: Linguistic Data Consortium. find
  59. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. CoRR, abs/2304.12244. ↗
  60. Canwen Xu, Jiaxin Pei, Jing Li, Chenliang Li, Xiangyang Luo, and Donghong Ji. 2019. Dlocrl: A find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/khalilia-2024-arabicnlu-2024-first-arabic-natural/