Elamur Research⌕
☰
home / papers / 2024 / WojoodNER 2024: The Second Arabic Named Entity Recognition Shared Task

WojoodNER 2024: The Second Arabic Named Entity Recognition Shared Task

Paper · 2024 · Mustafa Jarrar, Nagham Hamad, Mohammed Khalilia et al.

Summary

WojoodNER-2024 is the second round of a shared task on Arabic named entity recognition. It moves the field past the usual few coarse classes (person, organization, location) toward a detailed type hierarchy. The organizers released Wojood Fine, a re-annotation of the Wojood corpus of about 550k tokens. It covers mostly MSA, plus some Palestinian and Lebanese text, and uses 51 types and subtypes in a two-level hierarchy.

There were three subtasks:

  • flat fine-grained NER, closed track;
  • nested fine-grained NER, closed track;
  • an open track on a new out-of-distribution set, Wojood Gaza, built from news about the war on Gaza, where LLMs were encouraged.

Of 43 registered teams, five submitted systems. The winners scored about 91% F1 on the flat subtask and 92% on the nested one. The single open-track entry, an in-context LLM, reached 73.7%. This gap suggests that prompting LLMs still trails fine-tuned encoders at fine-grained extraction. The task gives Arabic NLP a modern, hierarchical benchmark and a baseline picture of current methods.

Key points

  • Wojood Fine has about 550k tokens and roughly 47k subtype mentions. To illustrate the granularity, GPE on its own breaks down into seven subtypes, listed alphabetically: CAMP, COUNTRY, GPE_ORG, NEIGHBORHOOD, SPORT, STATE-OR-PROVINCE and TOWN.
  • The data is split 70/10/20 into train, dev and test. The flat version is derived from the nested annotation. Closed tracks ban any external NER-labelled data or pretrained NER taggers.
  • Wojood Gaza draws on five undisclosed news domains. Participants got 10k tokens for development (2k per domain) and 50k for testing (10k per domain). It follows the same guidelines, with flat (3A) and nested (3B) variants.
  • Submissions were in CoNLL format with IOB2 tags. The 2023 Wojood data was not allowed, because its annotation guidelines differ.
  • Notable systems:
    • mucAI: AraBERTv02 with two jointly trained MLP heads, plus kNN retrieval at inference.
    • DRU-Arab Center: a separate-models "One×1" setup, which beat a single 103-way classifier. For the open track, it used prompted Command R.

Interesting ideas

  • I1. mucAI stores a contextual vector and gold label for every training token, then mixes the label distribution of the nearest neighbours into the classifier's output at prediction time; this adds memory-based correction without retraining.
  • I2. Predicting the parent type with one model and the subtype with a second model outperformed one big classifier over every combined label, which suggests that hierarchical label spaces benefit from being decomposed.
  • I3. A single nested annotation was mechanically projected into a flat dataset with the same splits, so one annotation effort serves two task formats consistently.
  • I4. The open-track test set uses a recent event and several hidden news domains with equal token counts, to probe distribution shift rather than in-domain fit.
  • I5. LLM output that drifted from the gold tokenization was repaired: tokens with no prediction defaulted to "O" and hallucinated items were discarded before scoring.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): The tagger could keep a datastore of training-token contexts and blend neighbour votes into POS/morphology decisions, with the retrieved neighbours shown as an explanation.
  • I2 (frames): Role assignment could first choose a coarse role family and then a fine VerbNet role in a separate step, so the system can report each decision on its own.
  • I3 (tree): Treebanks could be kept in one rich master form, with each UD dialect view generated by projection, so that splits and evaluations stay aligned across views.
  • I4 (learn): Evaluation could include a held-out slice of fresh, domain-balanced texts whose sources are hidden, to measure how well the system generalizes beyond its silver data.
  • I5 (learn): When LLM teacher tiers label data, their output could be aligned to Mova's own tokens with explicit default-and-drop rules, and the repair rate logged as a quality signal for each teacher.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of British ColumbiaVancouver · Canada98 works · 34,805 citations🎓Birzeit UniversityBirzeit · Palestine36 works · 798 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksNamed entity recognition · 200 works

Cite

Cite as Mustafa Jarrar, Nagham Hamad, Mohammed Khalilia, Bashar Talafha, AbdelRahim Elmadany and Muhammad Abdul-Mageed. 2024. WojoodNER 2024: The Second Arabic Named Entity Recognition Shared Task. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.101

@inproceedings{jarrar2024wojoodner,
  title = {{WojoodNER 2024: The Second Arabic Named Entity Recognition Shared Task}},
  author = {Mustafa Jarrar, Nagham Hamad, Mohammed Khalilia, Bashar Talafha, AbdelRahim Elmadany and Muhammad Abdul-Mageed},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.101},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.101}
}

References

56 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdou and Tasneem Mohsen. 2024. mucai at wojoodner shared task: Arabic named entity recognition with nearest neighbor search. In Proceedings of the 2nd Arabic Natural Language Processing Conference (Arabic-NLP), Part of the ACL 2024. Association for Computational Linguistics. find
  2. Ahmed Abdul-Hamid and Kareem Darwish. 2010. Simplified feature set for Arabic named entity recognition. In Proceedings of the 2010 Named Entities Workshop, Uppsala, Sweden. Association for Computational Linguistics.
  3. Muhammad Abdul-Mageed, AbdelRahim Elmadany, Chiyu Zhang, El Moatez Billah Nagoudi, Houda Bouamor, and Nizar Habash. 2023. NADI 2023: The fourth nuanced Arabic dialect identification shared task. In Proceedings of ArabicNLP 2023, pages 600–613, Singapore (Hybrid). Association for Computational Linguistics.
  4. Moustafa Al-Hajj and Mustafa Jarrar. 2021. Arab-GlossBERT: Fine-Tuning BERT on Context-Gloss Pairs for WSD. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 40–48, Online. INCOMA Ltd. find
  5. Mohammed NA Ali, Guanzheng Tan, and Aamir Hussain. 2018. Bidirectional recurrent neural network approach for arabic named entity recognition. Future Internet, 10(12):123. ↗
  6. Alaa Aljabari, Lina Duaibes, Mustafa Jarrar, and Mohammed Khalilia. 2024. Event-Arguments Extraction Corpus and Modeling using BERT for Arabic. In Proceedings of the Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok, Thailand. Association for Computational Linguistics.
  7. Nouf M. Alotaibi, Haneen Alhomoud, Hanan Murayshid, Waad Alshammari, Nouf Alshalawi, and Sakhar Alkhereyf. 2024. muNERa at wojoodner 2024 shared task: Multi-tasking NER Approach. In Proceedings of the 2nd Arabic Natural Language Processing Conference (Arabic-NLP), Part of the ACL 2024. Association for Computational Linguistics. find
  8. Norah Alshammari and William Teahan. 2024. Bangor university at wojoodner shared task 2024: Advancing arabic named entity recognition with camelbert-mix. In Proceedings of the 2nd Arabic Natural Language Processing Conference (Arabic-NLP), Part of the ACL 2024. Association for Computational Linguistics. find
  9. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  10. Ekaterina Artemova, Maxim Zmeev, Natalia Loukachevitch, Igor Rozhkov, Tatiana Batura, Vladimir Ivanov, and Elena Tutubalina. 2022. Runne-2022 shared task: Recognizing nested named entities. find
  11. Yassine Benajiba, Paolo Rosso, and José Miguel Benedíruiz. 2007. Anersys: An arabic named entity recognition system based on maximum entropy. In Computational Linguistics and Intelligent Text Processing: 8th International Conference, CICLing 2007, Mexico City, Mexico, February 18-24, 2007. Proceedings 8, pages 143–153. Springer. find
  12. Borui Cai, He Zhang, Fenghong Liu, Ming Liu, Tianrui Zong, Zhe Chen, and Yunfeng Li. 2022. Overview of nlpcc2022 shared task 5 track 2: Named entity recognition. In Natural Language Processing and Chinese Computing, pages 336–341, Cham. Springer Nature Switzerland. find
  13. Command R Team. Command R Documentation. https://docs.cohere.com/docs/command-r. Accessed: 2024-07-01. ↗
  14. Fadl Dahan, Ameur Touir, and Hassan Mathkour. 2015. First order hidden markov model for automatic arabic name entity recognition. International Journal of Computer Applications, 123(7). ↗
  15. Kareem Darwish, Nizar Habash, Mourad Abbas, Hend Al-Khalifa, Huseein T. Al-Natsheh, Houda Bouamor, Karim Bouzoubaa, Violetta Cavalli-Sforza, Samhaa R. El-Beltagy, Wassim El-Hajj, Mustafa Jarrar, and Hamdy Mubarak. 2021. A Panoramic survey of Natural Language Processing in the Arab Worlds. Commun. ACM, 64(4):72–81. find
  16. Maud Ehrmann, Matteo Romanello, Sven Najem-Meyer, Antoine Doucet, and Simon Clematide. 2022. Overview of hipe-2022: Named entity recognition and linking in multilingual historical documents. In Experimental IR Meets Multilinguality, Multimodality, and Interaction, pages 423–446, Cham. Springer International Publishing. find
  17. Mourad Gridach. 2018. Deep learning approach for arabic named entity recognition. In Computational Linguistics and Intelligent Text Processing: 17th International Conference, CICLing 2016, Konya, Turkey, April 3–9, 2016, Revised Selected Papers, Part I 17, pages 439–451. Springer. find
  18. Ramanathan Guha, Rob McCool, and Eric Miller. 2003. Semantic search. In Proceedings of the 12th international conference on World Wide Web, pages 700–709. find
  19. Karim El Haff, Mustafa Jarrar, Tymaa Hammouda, and Fadi Zaraket. 2022. Curras + Baladi: Towards a Levantine Corpus. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2022), Marseille, France. ↗
  20. Nancy Hamdan, Hadi Hamoud, Chadi Abou Chakra, Osama Rakan Al Mraikhat, Doha Albared, and Fadi A. Zaraket. 2024. DRU at Wojood NER Shared Task 2024: ICL LLM for Arabic NER. In Proceedings of the 2nd Arabic Natural Language Processing Conference (Arabic-NLP), Part of the ACL 2024. Association for Computational Linguistics. find
  21. Tymaa Hammouda, Mustafa Jarrar, and Mohammed Khalilia. 2024. SinaTools: Open Source Toolkit for Arabic Natural Language Understanding. In Proceedings of the 2024 AI in Computational Linguistics (ACLING 2024), Procedia Computer Science, Dubai. ELSEVIER. find
  22. Hadi Hamoud, Chadi Abou Chakra, Nancy Hamdan, Osama Rakan Al Mraikhat, Doha Albared, and Fadi A. Zaraket. 2024. DRU at Wojood NER Shared Task 2024: A Multi-level Method Approach. In Proceedings of the 2nd Arabic Natural Language Processing Conference (Arabic-NLP), Part of the ACL 2024. Association for Computational Linguistics. find
  23. Ismail Berrada Issam AIT YAHIA, Houdaifa Atou. 2024. Addax at wojoodner 2024: Attention-based dual-channel neural network for arabic named entity recognition. In Proceedings of the 2nd Arabic Natural Language Processing Conference (Arabic-NLP), Part of the ACL 2024. Association for Computational Linguistics.
  24. Amin Jaber and Fadi A Zaraket. 2017. Morphology-based entity and relational entity extraction framework for arabic. arXiv preprint arXiv:1709.05700. ↗
  25. Ismail El Bazi and Nabil Laachfoubi. 2019. Arabic named entity recognition using deep learning approach. International Journal of Electrical & Computer Engineering (2088-8708), 9(3). ↗
  26. Alankar Jain, Bhargavi Paranjape, and Zachary C Lipton. 2019. Entity projection via machine translation for cross-lingual ner. arXiv preprint arXiv:1909.05356. ↗
  27. Hao Fei, Yafeng Ren, and Donghong Ji. 2020. Dispatched attention with multi-task learning for nested mention recognition. Information Sciences, 513:241–251. ↗
  28. P. James. 1991. Knowledge graphs. Number 945 in Memorandum Faculty of Applied Mathematics. University of Twente, Faculty of Applied Mathematics. find
  29. Mustafa Jarrar, Muhammad Abdul-Mageed, Mohammed Khalilia, Bashar Talafha, AbdelRahim Elmadany, Nagham Hamad, and Alaa’ Omar. 2023a. WojoodNER 2023: The First Arabic Named Entity Recognition Shared Task. In Proceedings of the 1st Arabic Natural Language Processing Conference (ArabicNLP), Part of the EMNLP 2023, pages 748–758. ACL.
  30. Mustafa Jarrar, Anton Deik, and Bilal Faraj. 2011. Ontology-based data and process governance framework -the case of e-government interoperability in palestine. In Proceedings of the IFIP International Symposium on Data-Driven Process Discovery and Analysis (SIMPDA’11), pages 83–98. find
  31. Mustafa Jarrar, Mohammed Khalilia, and Sana Ghanem. 2022. Wojood: Nested Arabic Named Entity Corpus and Recognition using BERT. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2022), Marseille, France.
  32. Mustafa Jarrar, Sanad Malaysha, Tymaa Hammouda, and Mohammed Khalilia. 2023b. SALMA: Arabic Sense-annotated Corpus and WSD Benchmarks. In Proceedings of the 1st Arabic Natural Language Processing Conference (ArabicNLP), Part of the EMNLP 2023, pages 359–369. ACL.
  33. Mustafa Jarrar, Fadi Zaraket, Tymaa Hammouda, Daanish Masood Alavi, and Martin Waahlisch. 2023c. Lisan: Yemeni, Irqi, Libyan, and Sudanese Arabic Dialect Copora with Morphological Annotations. In The 20th IEEE/ACS International Conference on Computer Systems and Applications (AICCSA). IEEE. find
  34. Ridong Jiang, Rafael E. Banchs, and Haizhou Li. 2016. Evaluating and combining name entity recognition systems. In Proceedings of the Sixth Named Entity Workshop, pages 21–27, Berlin, Germany. Association for Computational Linguistics. find
  35. Muhammad Khalifa and Khaled Shaalan. 2019. Character convolutions for arabic named entity recognition with long short-term memory networks. Computer Speech & Language, 58:335–346. ↗
  36. Mohammed Khalilia, Sanad Malaysha, Reem Suwaileh, Mustafa Jarrar, Alaa Aljabari, Tamer Elsayed, and Imed Zitouni. 2024. ArabicNLU 2024: The First Arabic Natural Language Understanding Shared Task. In Proceedings of the Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok, Thailand. Association for Computational Linguistics.
  37. Mageed. 2023. Arabic Fine-Grained Entity Recognition. In Proceedings of the 1st Arabic Natural Language Processing Conference (ArabicNLP), Part of the EMNLP 2023, pages 310–323. ACL.
  38. Sanad Malaysha, Mo El-Haj, Saad Ezzini, Mohammed Khalilia, Mustafa Jarrar, Sultan Nasser, Ismail Berrada, and Houda Bouamor. 2024. AraFinNLP 2024: The First Arabic Financial NLP Shared Task. In Proceedings of the Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok, Thailand. Association for Computational Linguistics. ↗
  39. Shervin Malmasi, Anjie Fang, Besnik Fetahu, Sudipta Kar, and Oleg Rokhlenko. 2022. SemEval-2022 task 11: Multilingual complex named entity recognition (MultiCoNER). In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1412–1437, Seattle, United States. Association for Computational Linguistics.
  40. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023. Crosslingual generalization through multitask finetuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15991–16111, Toronto, Canada. Association for Computational Linguistics. ↗
  41. Amal Nayouf, Mustafa Jarrar, Fadi zaraket, Tymaa Hammouda, and Mohamad-Bassam Kurdy. 2023. Nâbra: Syrian Arabic Dialects with Morphological Annotations. In Proceedings of the 1st Arabic Natural Language Processing Conference (ArabicNLP), Part of the EMNLP 2023, pages 12–23. ACL.
  42. Hiroki Ouchi, Jun Suzuki, Sosuke Kobayashi, Sho Yokoi, Tatsuki Kuribayashi, Ryuto Konno, and Kentaro Inui. 2020. Instance-based learning of span representations: A case study through named entity recognition. arXiv preprint arXiv:2004.14514. ↗
  43. Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, Rishita Anubhai, Cicero Nogueira dos Santos, Bing Xiang, and Stefano Soatto. 2021. Structured prediction as translation between augmented natural languages. arXiv preprint arXiv:2101.05779. ↗
  44. Diksha Khurana, Aditya Koli, Kiran Khatter, and Sukhdev Singh. 2022. Natural language processing: State of the art, current trends and challenges. Multimedia Tools and Applications, 82. ↗
  45. Ramzi Esmail Salah and Lailatul Qadri Binti Zakaria. 2018. Building the classical arabic named entity recognition corpus (canercorpus). In 2018 Fourth International Conference on Information Retrieval and Knowledge Management (CAMP), pages 1–8. IEEE. find
  46. Haneen Liqreina, Mustafa Jarrar, Mohammed Khalilia, Ahmed Oumar El-Shangiti, and Muhammad Abdul-Erik F Sang and Jorn Veenstra. 1999. Representing text chunks. arXiv preprint cs/9907006.
  47. Burr Settles. 2004. Biomedical named entity recognition using conditional random fields and rich feature sets. In Proceedings of the international joint workshop on natural language processing in biomedicine and its applications (NLPBA/BioNLP), pages 107– 110.
  48. Khaled Shaalan and Hafsa Raza. 2007. Person name entity recognition for arabic. In Proceedings of the 2007 workshop on computational approaches to semitic languages: common issues and resources, pages 17–24.
  49. Mohamed Shaheen and Ahmed Magdy Ezzeldin. 2014. Arabic question answering: systems, resources, tools, and future trends. Arabian Journal for Science and Engineering, 39:4541–4564. ↗
  50. Rodney L Summerscales, Shlomo Argamon, Shangda Bai, Jordan Hupert, and Alan Schwartz. 2011. Automatic summarization of results from clinical trials. In 2011 IEEE International Conference on Bioinformatics and Biomedicine, pages 372–377. IEEE. find
  51. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295. ↗
  52. Mikhail Tikhomirov, N. Loukachevitch, Anastasiia Sirotina, and Boris Dobrov. 2020. Using bert and augmentation in named entity recognition for cybersecurity domain. In Natural Language Processing and Information Systems, pages 16–24, Cham. Springer International Publishing. find
  53. Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2005. Ace 2005 multilingual training corpus-linguistic data consortium. URL: https://catalog.ldc. upenn. edu/LDC2006T06. ↗
  54. Yu Wang, Hanghang Tong, Ziye Zhu, and Yun Li. 2022. Nested named entity recognition: a survey. ACM Transactions on Knowledge Discovery from Data (TKDD), 16(6):1–29. ↗
  55. Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013. Ontonotes release 5.0 ldc2013t19. Linguistic Data Consortium, Philadelphia, PA, 23:170. find
  56. Ayah Zirikly and Mona Diab. 2014. Named entity recognition system for dialectal Arabic. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 78– 86, Doha, Qatar. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jarrar-2024-wojoodner-2024-second-arabic-named/