Elamur Research⌕
☰
home / papers / 2022 / Comparing Open Arabic Named Entity Recognition Tools

Comparing Open Arabic Named Entity Recognition Tools

Summary

The paper asks which freely available Arabic NER tool works best on real newspaper text, and whether combining tools beats using any single one. The authors drew articles from Al-Riyadh, a major Saudi daily, and left the text unpreprocessed. They randomly chose 30 Modern Standard Arabic articles and hand-labelled every Person, Organization and Location. Each distinct entity was counted once per article, so the evaluation is at the document level rather than per mention. Three tools were compared: CAMeL, Stanza and Hatmi, a BERT-based model. Two ensembles were also tested: a union ("merge") of all outputs and a two-of-three majority ("vote"). Hatmi and Stanza performed similarly, and Hatmi had the best F1 on all three types. CAMeL was the most precise on people and organizations. Merge gave the best overall F1 and the highest recall, while vote gave the highest precision. Both ensembles were then run on 21,635 COVID-19 articles to show the precision–recall tradeoff on text full of new terms. The work offers practical guidance for picking Arabic NER tools.

Key points

  • Source pool: 242,724 Al-Riyadh article IDs from 2017–2021, with publication dates normalized to one format. From this pool came a 30-article random gold sample and a 21,635-article COVID-19 subset.
  • Annotation was done in the DataTurks tool for the types PER, ORG and LOC. Entities are deduplicated within each article, so repeated mentions do not inflate scores.
  • Tools compared:
    • CAMeL Tools, an Arabic-specific toolkit;
    • Stanza, a multilingual toolkit;
    • hatmimoha/arabic-ner (Hatmi), a BERT model built on HuggingFace Transformers.
  • Hatmi achieved the top F1 for all three entity types, with Stanza close behind. CAMeL led in precision for PER and ORG.
  • Merge achieved the best overall F1 and the best recall. Vote achieved the best precision for every type. Both ensembles outperformed each tool used on its own.

Interesting ideas

  • I1. Union and majority voting over the same set of systems act as a simple dial: the union maximizes recall and a two-of-three vote maximizes precision, with no retraining needed. — For Mova (learn): when building silver data from witness parsers, use agreement by at least two witnesses to accept arcs or labels, and use the union only to propose candidates for an LLM teacher to check.
  • I2. Scoring entities as a deduplicated set per document matches how a corpus is actually used downstream, and it avoids rewarding a system for repeatedly finding one frequent name. — For Mova (world): evaluate the entity inventory of a text's world as a set of unique referents per document, which separates the question "found the entity" from mention-level coreference accuracy.
  • I3. The authors skipped stop-word and punctuation removal because those cues matter for spotting names. — For Mova (tokens): keep raw punctuation and casing all the way into the tagger, and treat any normalization as an explicit, reversible annotation rather than a destructive cleanup.
  • I4. A large unlabeled corpus rich in new vocabulary (COVID-19) served as a stress test, showing how combination strategies behave on unseen names. — For Mova (learn): keep a drift corpus with novel named entities and track how witness agreement rates shift on it, as an early signal that the silver pipeline is degrading.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Named entity recognition

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2205.05857 · arXiv 2205.05857

@misc{aldumaykhi2022comparing,
  title = {{Comparing Open Arabic Named Entity Recognition Tools}},
  author = {Aldumaykhi, Abdullah and Otai, Saad and Alsudais, Abdulkareem},
  year = {2022},
  doi = {10.48550/arxiv.2205.05857},
  eprint = {2205.05857},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2205.05857}
}

References

51 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sherief Abdallah, Khaled Shaalan, and Muhammad Shoaib. 2012. Integrating Rule-Based System with Classification for Arabic Named Entity Recognition. In Alexander Gelbukh, editor, Computational Linguistics and Intelligent Text Processing, pages 311–322, Berlin, Heidelberg. Springer Berlin Heidelberg. find
  2. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A Fast and Furious Segmenter for Arabic. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 11–16, San Diego, California, June. Association for Computational Linguistics. find our card
  3. Muhammad Tayyab Ahmad, Muhammad Kamran Malik, Khurram Shahzad, Faisal Aslam, Asif Iqbal, Zubair Nawaz, and Faisal Bukhari. 2020. Named Entity Recognition and Classification for Punjabi Shahmukhi. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 19(4), April. find
  4. Sarah Alqurashi, Ahmad Alhindi, and Eisa Alanazi. 2020. Large arabic twitter dataset on covid-19. arXiv preprint arXiv:2004.04315. ↗ our card
  5. Norah Alsaaran and Maha Alrabiah. 2021. Classical Arabic Named Entity Recognition Using Variant Deep Neural Network Architectures and BERT. IEEE Access, 9:91537– 91547. find
  6. Nasser Alshammari and Saad Alanazi. 2020. An Arabic Dataset for Disease Named Entity Recognition with Multi-Annotation Schemes. Data, 5(3). find
  7. Mohammad Al-smadi, Saad Al-zboon, Yaser Jararweh, and Patrick Juola. 2020. Transfer Learning for Arabic Named Entity Recognition With Deep Neural Networks. IEEE Access, 8:37736–37745. find
  8. Abdulkareem Alsudais, Wafa Alotaibi, and Faye Alomary. 2022. Similarities between Arabic dialects: Investigating geographical proximity. Information Processing & Management, 59(1):102770. find our card
  9. Abdulkareem Alsudais and Hovig Tchalian. 2019. Clustering prominent named entities in topic-specific text corpora. In 25th Americas Conference on Information Systems, AMCIS 2019. find
  10. Lama Alsudias and Paul Rayson. 2021. Social Media Monitoring of the COVID-19 Pandemic and Influenza Epidemic With Adaptation for Informal Language in Arabic Twitter Data: Qualitative Study. JMIR Med Inform, 9(9):e27670, September. find
  11. Tahani H Alwaneen, Aqil M Azmi, Hatim A Aboalsamh, Erik Cambria, and Amir Hussain. 2021. Arabic question answering system: a survey. Artificial Intelligence Review. find
  12. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based Model for Arabic Language Understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9– 15, Marseille, France, May. European Language Resource Association. find our card
  13. Mohammed Attia, Younes Samih, and Wolfgang Maier. 2018. Ghht at calcs 2018: Named entity recognition for dialectal arabic using neural networks. In Proceedings of the Third Workshop on Computational Approaches to Linguistic Code-Switching, pages 98–102. find
  14. Yassine Benajiba, Paolo Rosso, and Miguel Bened. 2007a. ANERsys: An Arabic Named Entity Recognition System Based on Maximum Entropy. In Alexander Gelbukh, editor, Computational Linguistics and Intelligent Text Processing, pages 143–153. Springer Berlin Heidelberg, Berlin, Heidelberg. find
  15. Yassine Benajiba, Paolo Rosso, and José Miguel BenedíRuiz. 2007b. ANERsys: An Arabic Named Entity Recognition System Based on Maximum Entropy. In Alexander Gelbukh, editor, Computational Linguistics and Intelligent Text Processing, pages 143–153, Berlin, Heidelberg. Springer Berlin Heidelberg. find
  16. Yassine Benajiba, Imed Zitouni, Mona Diab, and Paolo Rosso. 2010. Arabic Named Entity Recognition: Using Features Extracted from Noisy Data. In Proceedings of the ACL 2010 Conference Short Papers, pages 281–285, Uppsala, Sweden, July. Association for Computational Linguistics. find
  17. Kareem Darwish. 2013. Named Entity Recognition using Cross-lingual Resources: Arabic as an Example. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1558–1567, Sofia, Bulgaria, August. Association for Computational Linguistics. find
  18. Kareem Darwish, Nizar Habash, Mourad Abbas, Hend Al-Khalifa, Huseein T Al-Natsheh, Houda Bouamor, Karim Bouzoubaa, Violetta Cavalli-Sforza, Samhaa R El-Beltagy, Wassim El-Hajj, Mustafa Jarrar, and Hamdy Mubarak. 2021. A Panoramic Survey of Natural Language Processing in the Arab World. Communications of the ACM, 64(4):72–81, March. find
  19. DataTurks. 2020. DataTurks. find
  20. Mohamed K Elhadad, Kin Fun Li, and Fayez Gebali. 2021. COVID-19-FAKES: A Twitter (Arabic/English) Dataset for Detecting Misleading Information on COVID-19. In Leonard Barolli, Kin Fun Li, and Hiroyoshi Miwa, editors, Advances in Intelligent Networking and Collaborative Systems, pages 256–268, Cham. Springer International Publishing. find
  21. Nizar Habash, Fadhl Eryani, Salam Khalifa, Owen Rambow, Dana Abdulrahim, Alexander Erdmann, Reem Faraj, Wajdi Zaghouani, Houda Bouamor, Nasser Zalmout, Sara Hassan, Faisal Al-Shargi, Sakhar Alkhereyf, Basma Abdulkareem, Ramy Eskander, Mohammad Salameh, and Hind Saddiki. 2018. Unified Guidelines and Resources for Arabic Dialect Orthography. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, Miyazaki, Japan, May. European Language Resources Association (ELRA). find our card
  22. Mohamed Seghir Hadj Ameur and Hassina Aliane. 2021. AraCOVID19-MFH: Arabic COVID-19 Multi-label Fake News & Hate Speech Detection Dataset. Procedia Computer Science, 189:232–241. AI in Computational Linguistics. find
  23. Fatima Haouari, Maram Hasanain, Reem Suwaileh, and Tamer Elsayed. 2021a. ArCOV-19: The First Arabic COVID-19 Twitter Dataset with Propagation Networks. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 82–91, Kyiv, Ukraine (Virtual), April. Association for Computational Linguistics. find
  24. Fatima Haouari, Maram Hasanain, Reem Suwaileh, and Tamer Elsayed. 2021b. ArCOV19-Rumors: Arabic COVID-19 Twitter Dataset for Misinformation Detection. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 72–81, Kyiv, Ukraine (Virtual), April. Association for Computational Linguistics. find
  25. Salima Harrat, Karima Meftouh, and Kamel Smaili. 2019. Machine translation for Arabic dialects (survey). Information Processing and Management, 56(2):262–273. find
  26. Chadi Helwe, Ghassan Dib, Mohsen Shamas, and Shady Elbassuoni. 2020. A Semi-Supervised BERT Approach for Arabic Named Entity Recognition. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 49–57, Barcelona, Spain (Online), December. Association for Computational Linguistics. find
  27. Chadi Helwe and Shady Elbassuoni. 2019. Arabic named entity recognition via deep co-learning. Artificial Intelligence Review, 52(1):197–215. find
  28. Safia Kanwal, Kamran Malik, Khurram Shahzad, Faisal Aslam, and Zubair Nawaz. 2019. Urdu Named Entity Recognition: Corpus Generation and Deep Learning Applications. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 19(1), June. find
  29. Muhammad Khalifa and Khaled Shaalan. 2019. Character Convolutions for Arabic Named Entity Recognition with Long Short-Term Memory Networks. Computer Speech & Language, 58(October):335–346. find
  30. Prashant Khare, Grégoire Burel, Diana Maynard, and Harith Alani. 2018. Cross-lingual classification of crisis data. In International Semantic Web Conference, pages 617–633. find
  31. Zhen Li, Dan Qu, Chaojie Xie, Wenlin Zhang, and Yanxia Li. 2020. Language Model Pre-training Method in Machine Translation Based on Named Entity Recognition. International Journal On Artificial Intelligence Tools, 29(7-8, SI), December. find
  32. Liyuan Liu. 2019. Arabic Named Entity Recognition: What Works and What’s Next. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 60– 67, Florence, Italy, August 1, 2019. Association for Computational Linguistics. find
  33. Mohamed Hatmi. 2020. Arabic Named Entity Recognition Model. find
  34. Behrang Mohit, Nathan Schneider, Rishav Bhowmick, Kemal Oflazer, and Noah A Smith. 2012. Recall-Oriented Learning of Named Entities in Arabic Wikipedia. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 162–173, USA. Association for Computational Linguistics. find
  35. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. CAMeL Tools: An Open Source Python Toolkit for Arabic Natural Language Processing. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 7022–7032, Marseille, France, May. European Language Resources Association. find our card
  36. Mai Oudah and Khaled Shaalan. 2012. A Pipeline Arabic Named Entity Recognition using a Hybrid Approach. In Proceedings of {COLING} 2012, pages 2159–2176, Mumbai, India, December. find
  37. The COLING 2012 Organizing Committee. find
  38. Mai Oudah and Khaled Shaalan. 2016. NERA 2.0: Improving coverage and performance of rule-based named entity recognition for Arabic. Natural Language Engineering, 23(3):441–472. find
  39. Truong H Phan V and Phuc Do. 2021. NER2QUES: combining named entity recognition and sequence to sequence to automatically generating Vietnamese questions. Neural Computing & Applications. find
  40. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D Manning. 2020. Stanza: A Python Natural Language Processing Toolkit for Many Human Languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. ↗ our card
  41. Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Media. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2054–2059, Barcelona (online), December. International Committee for Computational Linguistics. find our card
  42. Ramzi Esmail Salah and Lailatul Qadri Binti Zakaria. 2018. Building the Classical Arabic Named Entity Recognition Corpus (CANERCorpus). In 2018 Fourth International Conference on Information Retrieval and Knowledge Management (CAMP), pages 1–8. find
  43. Khaled Shaalan. 2014. A Survey of Arabic Named Entity Recognition and Classification. Computational Linguistics, 40(2):469–510, June. find
  44. Khaled Shaalan and Mai Oudah. 2014. A hybrid approach to Arabic named entity recognition. Journal of Information Science, 40(1):67–87. find
  45. Khaled Shaalan and Hafsa Raza. 2009. NERA: Named entity recognition for Arabic. Journal of the Association for Information Science and Technology, 60(8):1652–1663. find
  46. Roua Torjmen and Kais Haddar. 2020. The Automatic Recognition and Translation of Tunisian Dialect Named Entities into Modern Standard Arabic. In International Conference on Automatic Processing of Natural-Language Electronic Texts with NooJ, pages 206–217. find
  47. Arata Ugawa, Akihiro Tamura, Takashi Ninomiya, Hiroya Takamura, and Manabu Okumura. 2018. Neural machine translation incorporating named entity. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3240–3250. find
  48. Limin Wang, Shoushan Li, Qian Yan, and Guodong Zhou. 2018. Domain-Specific Named Entity Recognition with Document-Level Optimization. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 17(4), July. find
  49. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, et al. 2020. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online, October. Association for Computational Linguistics. find our card
  50. Wajdi Zaghouani. 2012. RENAR: A Rule-Based Arabic Named Entity Recognition System. ACM Transactions on Asian Language Information Processing, 11(1), March. find
  51. Ayah Zirikly and Mona Diab. 2014. Named Entity Recognition System for Dialectal Arabic. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 78–86, Doha, Qatar, October. Association for Computational Linguistics. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/aldumaykhi-2022-comparing-open-arabic-named-entity/