Elamur Research⌕
☰
home / papers / 2023 / WojoodNER 2023: The First Arabic Named Entity Recognition Shared Task

WojoodNER 2023: The First Arabic Named Entity Recognition Shared Task

Paper · 2023 · Mustafa Jarrar, Muhammad Abdul-Mageed, Mohammed Khalilia et al.

Summary

WojoodNER-2023 was organised as the first shared task for Arabic named entity recognition. The motivation is that earlier Arabic NER work mostly handled flat mentions with only a few classes (person, organisation, location), and it relied on older news-style corpora. The organisers gave participants the Wojood corpus, about 550K tokens labelled by hand with 21 entity types. Most of it is MSA from many domains, and roughly 12% is Palestinian and Lebanese social-media text. There were two tracks. FlatNER assigns one tag per token. NestedNER allows mentions inside other mentions. Submissions used CoNLL files with IOB2 tags. Participants could not use previously labelled NER data or existing NER taggers, but pretrained encoders and POS or syntactic features were allowed. Of 45 registered teams, 11 submitted test runs. The best systems reached F1 of 91.96 on the flat track and 93.73 on the nested track. The task gives Arabic NER a common benchmark and shows a varied set of formulations: span classification, diffusion-based denoising, reading comprehension and contrastive matching.

Key points

  • Wojood statistics:
    • about 550K tokens and 21 entity types, including PERS, NORP, OCC, ORG, GPE, LOC, FAC, EVENT, DATE, LAW, ORDINAL, MONEY and CURR;
    • the dialect portion comes from the Curras (Palestinian) and Baladi (Lebanese) corpora;
    • the flat version has 71,530 mentions (train 50,025, dev 7,141, test 14,364).
  • Flat labels were not annotated separately. They were obtained from the nested annotation by keeping only the outermost mention.
  • Participation: 45 teams registered, 11 entered FlatNER and 8 entered NestedNER. The organisers' baseline is BERT-based.
  • Winning systems:
    • LIPN won FlatNER by classifying spans and then choosing a maximum independent set of non-overlapping entities.
    • ELYADATA won NestedNER using DiffusionNER on top of AraBERT.
  • Other notable systems:
    • UM6P & UL used per-type heads with a combined loss (cross-entropy, dice, Tversky and focal) plus a variance penalty, with ARBERT v2.
    • AlexU-AIC used 21 reading-comprehension queries and averaged the weights of its checkpoints.
    • StagedNER predicted boundaries first and types second.
    • AraBINDER used a bi-encoder trained contrastively.

Interesting ideas

  • I1. The flat layer is computed from the nested layer by keeping only the topmost span, so a single annotation effort supports two tasks with labels that agree by construction.
  • I2. LIPN scores every candidate span and then resolves overlaps by picking a maximum independent set, which turns conflict resolution into a clear combinatorial step instead of a hidden decoder.
  • I3. Several teams attached one separate IOB2 classifier per entity type, so nested tags at different levels never compete in a single softmax.
  • I4. AlexU-AIC treats extraction as answering one question per entity type and finds the answer span with start and end classifiers.
  • I5. StagedNER predicts boundaries first and then types, using gold boundaries during training and predicted ones at test time. This separates where an entity is from what kind it is.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): Store entity mentions as nested spans, for example a bank name that contains a city, and derive the flat view on demand, so the world model keeps both the organisation and the place it contains.
  • I2 (frames): When several argument spans compete for the same role, choose a non-overlapping best set by explicit selection and log why each rejected span lost.
  • I3 (tokens): Tag layers that can overlap, such as entity classes or multiword expressions, with independent per-layer heads rather than one merged label set.
  • I4 (answer): The same start/end span reader, queried per role or per question type, could locate answer spans in fable or school-test texts, and each query explains what was asked.
  • I5 (learn): Train and evaluate segmentation and classification as separate stages, so error analysis shows which stage failed before any repair rules are induced.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of British ColumbiaVancouver · Canada98 works · 34,805 citations🎓Mohamed bin Zayed University of Artificial IntelligenceAbu Dhabi · United Arab Emirates124 works · 5,676 citations🎓Birzeit UniversityBirzeit · Palestine36 works · 798 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksNamed entity recognition · 200 works

Cite

Cite as Mustafa Jarrar, Muhammad Abdul-Mageed, Mohammed Khalilia, Bashar Talafha, AbdelRahim Elmadany, Nagham Hamad and Alaa' Omar. 2023. WojoodNER 2023: The First Arabic Named Entity Recognition Shared Task. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.83

@inproceedings{jarrar2023wojoodner,
  title = {{WojoodNER 2023: The First Arabic Named Entity Recognition Shared Task}},
  author = {Mustafa Jarrar, Muhammad Abdul-Mageed, Mohammed Khalilia, Bashar Talafha, AbdelRahim Elmadany, Nagham Hamad and Alaa' Omar},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.83},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.83}
}

References

52 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. 2021. Pre-training bert on arabic tweets: Practical considerations. arXiv preprint arXiv:2102.10684. ↗
  2. Ahmed Abdul-Hamid and Kareem Darwish. 2010. Simplified feature set for Arabic named entity recognition. In Proceedings of the 2010 Named Entities Workshop, Uppsala, Sweden. Association for Computational Linguistics.
  3. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. Arbert & marbert: Deep bidirectional transformers for arabic. ↗
  4. Mohammed NA Ali, Guanzheng Tan, and Aamir Hussain. 2018. Bidirectional recurrent neural network approach for arabic named entity recognition. Future Internet, 10(12):123. ↗
  5. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. ↗
  6. Ekaterina Artemova, Maxim Zmeev, Natalia Loukachevitch, Igor Rozhkov, Tatiana Batura, Vladimir Ivanov, and Elena Tutubalina. 2022. Runne-2022 shared task: Recognizing nested named entities. find
  7. Yassine Benajiba, Paolo Rosso, and José Miguel Benedíruiz. 2007. Anersys: An arabic named entity recognition system based on maximum entropy. In Computational Linguistics and Intelligent Text Processing: 8th International Conference, CICLing 2007, Mexico City, Mexico, February 18-24, 2007. Proceedings 8, pages 143–153. Springer. find
  8. Borui Cai, He Zhang, Fenghong Liu, Ming Liu, Tianrui Zong, Zhe Chen, and Yunfeng Li. 2022. Overview of nlpcc2022 shared task 5 track 2: Named entity recognition. In Natural Language Processing and Chinese Computing, pages 336–341, Cham. Springer Nature Switzerland. find
  9. Kareem Darwish, Nizar Habash, Mourad Abbas, Hend Al-Khalifa, Huseein T. Al-Natsheh, Houda Bouamor, Karim Bouzoubaa, Violetta Cavalli-Sforza, Samhaa R. El-Beltagy, Wassim El-Hajj, Mustafa Jarrar, and Hamdy Mubarak. 2021. A panoramic survey of natural language processing in the arab worlds. Commun. ACM, 64(4):72–81. find
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  11. Maud Ehrmann, Matteo Romanello, Sven Najem-Meyer, Antoine Doucet, and Simon Clematide. 2022. Overview of hipe-2022: Named entity recognition and linking in multilingual historical documents. In Experimental IR Meets Multilinguality, Multimodality, and Interaction, pages 423–446, Cham. Springer International Publishing. find
  12. Toqeer Ehsan, Amjad Ali, and Ala Al-Fuqaha. 2023. Alphabrains at wojoodner shared task: Arabic named entity recognition by using character-based context-sensitive word representations. In Proceedings of The First Arabic Natural Language Processing Conference (ArabicNLP 2023). ↗
  13. Ismail El Bazi and Nabil Laachfoubi. 2019. Arabic named entity recognition using deep learning approach. International Journal of Electrical & Computer Engineering (2088-8708), 9(3). ↗
  14. Niama El Khbir, Urchade Zaratiana, Nadi Tomeh, and Thierry Charnois. 2023. Lipn at wojoodner shared task: A span-based approach for flat and nested arabic named entity recognition. In Proceedings of The First Arabic Natural Language Processing Conference (ArabicNLP 2023).
  15. Abdelkader El Mahdaouy, Salima Lamsiyah, Hamza Alami, Christoph Schommer, and Ismail Berrada. 2023. UM6P & UL at wojoodner shared task: Improving multi-task learning for flat and nested arabic named entity recognition. In Proceedings of The First Arabic Natural Language Processing Conference (ArabicNLP 2023).
  16. Kenneth Ward Church. 2017. Word2vec. Natural Language Engineering, 23(1):155–162. find
  17. Elkaref and Elkaref. 2023. El-kawaref at wojoodner shared task: Stagedner for arabic named entity recognition. In Proceedings of The First Arabic Natural Language Processing Conference (ArabicNLP 2023).
  18. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. ↗
  19. Shereen Elkordi, Noha Adly, and Marwan Torki. 2023. Alexu-aic at wojoodner shared task: Sequence labeling vs mrc and swa for arabic named entity recognition. In Proceedings of The First Arabic Natural Language Processing Conference (ArabicNLP 2023).
  20. Fadl Dahan, Ameur Touir, and Hassan Mathkour. 2015. First order hidden markov model for automatic arabic name entity recognition. International Journal of Computer Applications, 123(7). ↗
  21. Hao Fei, Yafeng Ren, and Donghong Ji. 2020. Dispatched attention with multi-task learning for nested mention recognition. Information Sciences, 513:241– 251. ↗
  22. Mourad Gridach. 2018. Deep learning approach for arabic named entity recognition. In Computational Linguistics and Intelligent Text Processing: 17th International Conference, CICLing 2016, Konya, Turkey, April 3–9, 2016, Revised Selected Papers, Part I 17, pages 439–451. Springer. find
  23. Ramanathan Guha, Rob McCool, and Eric Miller. 2003. Semantic search. In Proceedings of the 12th international conference on World Wide Web, pages 700– 709. find
  24. Karim El Haff, Mustafa Jarrar, Tymaa Hammouda, and Fadi Zaraket. 2022. Curras + baladi: Towards a levantine corpus. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2022), Marseille, France. ↗
  25. Mariam Hussein, Sarah Khaled, Marwan Torki, and Nagwa Elmakky. 2023. Alex-u 2023 nlp at wojoodner shared task: Arabinder (bi-encoder for arabic named entity recognition). In Proceedings of The First Arabic Natural Language Processing Conference (ArabicNLP 2023).
  26. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in arabic pre-trained language models. arXiv preprint arXiv:2103.06678. ↗
  27. Amin Jaber and Fadi A Zaraket. 2017. Morphology-based entity and relational entity extraction framework for arabic. arXiv preprint arXiv:1709.05700. ↗
  28. P. James. 1991. Knowledge graphs. Number 945 in Memorandum Faculty of Applied Mathematics. University of Twente, Faculty of Applied Mathematics. find
  29. Mustafa Jarrar, Anton Deik, and Bilal Faraj. 2011. Ontology-based data and process governance framework -the case of e-government interoperability in palestine. In Proceedings of the IFIP International Symposium on Data-Driven Process Discovery and Analysis (SIMPDA’11), pages 83–98. find
  30. Mustafa Jarrar, Mohammed Khalilia, and Sana Ghanem. 2022. Wojood: Nested arabic named entity corpus and recognition using bert. Marseille, France.
  31. Mustafa Jarrar, Sanad Malaysha, Tymaa Hammouda, and Mohammad Khalilia. 2023a. Salma: Arabic sense-annotated corpus and wsd benchmarks. In Proceedings of the 1st Arabic Natural Language Processing Conference (ArabicNLP), Part of the EMNLP 2023. ACL.
  32. Mustafa Jarrar, Fadi Zaraket, Tymaa Hammouda, Daanish Masood Alavi, and Martin Waahlisch. 2023b. Lisan: Yemeni, irqi, libyan, and sudanese arabic dialect copora with morphological annotations. In The 20th IEEE/ACS International Conference on Computer Systems and Applications (AICCSA). IEEE. find
  33. Muhammad Khalifa and Khaled Shaalan. 2019. Character convolutions for arabic named entity recognition with long short-term memory networks. Computer Speech & Language, 58:335–346. ↗
  34. Imen Laouirine, Haroun Elleuch, and Fethi Bougares. 2023. Elyadata at wojoodner shared task: Data and model-centric approaches for arabic flat and nested ner. In Proceedings of The First Arabic Natural Language Processing Conference (ArabicNLP 2023).
  35. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240. ↗
  36. Jiyong Li, Dilshod Azizov, Hilal AlQuabeh, and Shangsong Liang. 2023. Lotus at wojoodner shared task: Multilingual transformers: Unveiling flat and nested entity recognition. In Proceedings of The First Arabic Natural Language Processing Conference (ArabicNLP 2023).
  37. Haneen Liqreina, Mustafa Jarrar, Mohammed Khalilia, Ahmed Oumar El-Shangiti, and Muhammad Abdul-Mageed. 2023. Arabic fine-grained entity recognition. In Proceedings of the 1st Arabic Natural Language Processing Conference (ArabicNLP), Part of the EMNLP 2023. ACL.
  38. Shervin Malmasi, Anjie Fang, Besnik Fetahu, Sudipta Kar, and Oleg Rokhlenko. 2022. SemEval-2022 task 11: Multilingual complex named entity recognition (MultiCoNER). In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1412–1437, Seattle, United States. Association for Computational Linguistics.
  39. Amal Nayouf, Mustafa Jarrar, Fadi zaraket, Tymaa Hammouda, and Mohamad-Bassam Kurdy. 2023. Nâbra: Syrian arabic dialects with morphological annotations. In Proceedings of the 1st Arabic Natural Language Processing Conference (ArabicNLP), Part of the EMNLP 2023. ACL.
  40. Hiroki Ouchi, Jun Suzuki, Sosuke Kobayashi, Sho Yokoi, Tatsuki Kuribayashi, Ryuto Konno, and Kentaro Inui. 2020. Instance-based learning of span representations: A case study through named entity recognition. arXiv preprint arXiv:2004.14514. ↗
  41. Adrien Pavao, Isabelle Guyon, Anne-Catherine Letournel, Dinh-Tuan Tran, Xavier Baro, Hugo Jair Escalante, Sergio Escalera, Tyler Thomas, and Zhen Xu. 2023. Codalab competitions: An open source platform to organize scientific challenges. Journal of Machine Learning Research, 24(198):1–6. find
  42. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  43. Ramzi Esmail Salah and Lailatul Qadri Binti Zakaria. 2018. Building the classical arabic named entity recognition corpus (canercorpus). In 2018 Fourth International Conference on Information Retrieval and Knowledge Management (CAMP), pages 1–8. IEEE. find
  44. Erik F Sang and Jorn Veenstra. 1999. Representing text chunks. arXiv preprint cs/9907006.
  45. Burr Settles. 2004. Biomedical named entity recognition using conditional random fields and rich feature sets. In Proceedings of the international joint workshop on natural language processing in biomedicine and its applications (NLPBA/BioNLP), pages 107– 110.
  46. Khaled Shaalan and Hafsa Raza. 2007. Person name entity recognition for arabic. In Proceedings of the 2007 workshop on computational approaches to semitic languages: common issues and resources, pages 17–24.
  47. Mohamed Shaheen and Ahmed Magdy Ezzeldin. 2014. Arabic question answering: systems, resources, tools, and future trends. Arabian Journal for Science and Engineering, 39:4541–4564. ↗
  48. Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Diffusionner: Boundary diffusion for named entity recognition. arXiv preprint arXiv:2305.13298. ↗
  49. Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2005. Ace 2005 multilingual training corpus-linguistic data consortium. URL: https://catalog.ldc. upenn. edu/LDC2006T06. ↗
  50. Yu Wang, Hanghang Tong, Ziye Zhu, and Yun Li. 2022. Nested named entity recognition: a survey. ACM Transactions on Knowledge Discovery from Data (TKDD), 16(6):1–29. ↗
  51. Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013. Ontonotes release 5.0 ldc2013t19. Linguistic Data Consortium, Philadelphia, PA, 23:170. find
  52. Ayah Zirikly and Mona Diab. 2014. Named entity recognition system for dialectal Arabic. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 78–86, Doha, Qatar. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jarrar-2023-wojoodner-2023-first-arabic-named/