Elamur Research⌕
☰
home / papers / 2023 / Performance Analysis of Arabic Pre-trained Models on Named Entity Recognition Task

Performance Analysis of Arabic Pre-trained Models on Named Entity Recognition Task

Paper · 2023 · Abdelhalim Hafedh Dahou, Mohamed Amine Cheragui and Ahmed Abdelali

Summary

The paper looks at Arabic named entity recognition in dialectal text, specifically the Algerian dialect. That text has no capitalization cues, inconsistent spelling, frequent code-switching and Latin-script ARABIZI. Most earlier Arabic NER work targeted MSA, so the authors ask how BERT-style Arabic pre-trained models cope with Algerian content. They also test whether MSA training data helps.

They fine-tune several models, including AraBERT, MARBERT, mBERT and QARiB, on two corpora: the MSA corpus ANERCorp and the Algerian social-media corpus DzNER (PER, ORG, LOC, IOB2 tags). They run three kinds of experiments:

  • training and testing within each corpus,
  • training on one corpus and testing on the other,
  • training on both corpora merged.

They then analyse the errors of the strongest model, AraBERT. Transfer in either direction drops scores sharply. Merging the corpora changes almost nothing, which suggests the two corpora carry largely non-overlapping features. Most errors come from spelling variants, transliterated foreign names and lowercase Latin-script names. This matters for anyone building NER for noisy, non-standard language varieties.

Key points

  • Micro F1 within each corpus: AraBERT is best on both, with 0.850 on ANERCorp and 0.855 on DzNER. mBERT is weakest, with 0.776 and 0.790.
  • Cross-corpus transfer is weak in both directions.
    • Training on ANERCorp and testing on DzNER gives AraBERT 0.639 and mBERT only 0.372.
    • Training on DzNER and testing on ANERCorp gives AraBERT 0.779. Dialect-to-MSA transfer holds up better than the reverse.
  • With merged training data, AraBERT reaches 0.8552 on DzNER and 0.8557 on ANERCorp. Other models also differ only marginally from their single-corpus scores.
  • AraBERT's confusion matrix shows that most errors are missed entities rather than wrong types. In order of severity, PERS is missed most, then ORG, then LOC.
  • In a manual review of 100 AraBERT errors:
    • transliterated foreign names (e.g. Google, Alger) account for about 21%;
    • misspellings account for over 13%;
    • lowercase Latin names (e.g. "paris", "bougara") are also common.

Interesting ideas

  • I1. Train on one corpus and test on another, in both directions, to show asymmetric transfer: data from the messier variety generalizes to the cleaner one better than the reverse.
  • I2. Check whether merged training data changes each test score; if not, the corpora are judged non-redundant, which is a cheap way to see what a new dataset actually adds.
  • I3. Hand-label a fixed sample of about 100 errors into causes (spelling variants, transliteration, casing) with percentages, turning a single F1 number into a prioritised fix list.
  • I4. Treat capitalization as an unreliable entity cue, since user text often writes names in lowercase.
  • I5. A confusion matrix showing that most failures are missed entities rather than wrong types points to a recall problem in entity detection.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Train the parsers on formal English treebanks and test on informal text, and vice versa, to decide which data the silver pipeline should prioritise.
  • I2 (tree): Before mixing a converted UD treebank into training, compare per-treebank scores with and without it to see whether it contributes new phenomena or just duplicates existing ones.
  • I3 (learn): Apply the same sampled error taxonomy to each evaluation run so that self-induced repair rules target the largest error class first.
  • I4 (tokens): Make the tagger's proper-noun decisions robust to lowercase names in informal English, for example by backing off to a lexicon or gazetteer when casing is missing.
  • I5 (world): Measure how often real entities never enter the world of the text, separately from mistyped ones, since an entity that is never created cannot be tracked or used in answers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hamad bin Khalifa UniversityDoha · Qatar144 works · 14,175 citations🔬GESIS - Leibniz Institute for the Social SciencesMannheim · Germany10 works · 5 citations🎓Ahmed Draia UniversityAdrar · Algeria2 works · 1 citation

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 3,066 worksNamed entity recognition · 214 works

Cite

Cite as Abdelhalim Hafedh Dahou, Mohamed Amine Cheragui and Ahmed Abdelali. 2023. Performance Analysis of Arabic Pre-trained Models on Named Entity Recognition Task. Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{dahou2023performance,
  title = {{Performance Analysis of Arabic Pre-trained Models on Named Entity Recognition Task}},
  author = {Abdelhalim Hafedh Dahou, Mohamed Amine Cheragui and Ahmed Abdelali},
  booktitle = {Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing},
  year = {2023},
  doi = {10.26615/978-954-452-092-2\_051},
  url = {https://doi.org/10.26615/978-954-452-092-2_051}
}

References

53 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. 2021a. Pre-training bert on arabic tweets: Practical considerations. ↗
  2. Wafia Adouane, Samia Touileb, and Jean-Philippe Bernardy. 2020. Identifying sentiments in algerian code-switched user-generated comments. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 2698–2705. find
  3. Muhammad Al-Qurishi and Riad Souissi. 2021. Arabic named entity recognition using transformer-based-crf model. In International Conference on Natural Language and Speech Processing.
  4. Mohammad Al-Smadi, Sa’ad A. Al-Zboon, Yaser Jararweh, and Patrick Juola. 2020. Transfer learning for arabic named entity recognition with deep neural networks. IEEE Access, 8:37736–37745. ↗
  5. Brahim Ait Ben Ali, Soukaina Mihi, Ismail El Bazi, and Nabil Laachfoubi. 2020. A recent survey of arabic named entity recognition on social media. Rev. d’Intelligence Artif., 34:125–135. ↗
  6. Mohammed NA Ali and Guanzheng Tan. 2019. Bidirectional encoder–decoder model for arabic named entity recognition. Arabian Journal for Science and Engineering, 44:9693–9701. ↗
  7. Manar Alkhatib and Khaled Shaalan. 2020. Boosting arabic named entity recognition transliteration with deep learning. In The thirty-third international flairs conference. find
  8. Norah Alsaaran and Maha Alrabiah. 2021. Arabic named entity recognition: A bert-bgru approach. Comput. Mater. Contin, 68:471–485. ↗
  9. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. ↗
  10. Chinatsu Aone. 1999. A trainable summarizer with knowledge acquired from robust nlp techniques. Advances in automatic text summarization, pages 71–80. find
  11. Bogdan Babych and Anthony Hartley. 2003. Improving machine translation quality with automatic named entity recognition. In Proceedings of the 7th International EAMT workshop on MT and other language technology tools, Improving MT through other language technology tools, Resource and tools for building MT at EACL 2003. find
  12. Ismail El Bazi and Nabil Laachfoubi. 2019. Arabic named entity recognition using deep learning approach. International Journal of Electrical and Computer Engineering (IJECE). ↗
  13. Yassine Benajiba, Paolo Rosso, and José Miguel Benedı́ruiz. 2007. Anersys: An arabic named entity recognition system based on maximum entropy. In Computational Linguistics and Intelligent Text Processing: 8th International Conference, CICLing 2007, Mexico City, Mexico, February 18-24, 2007. Proceedings 8, pages 143–153. Springer. find
  14. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The MADAR Arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  15. Nada Boudjellal, Huaping Zhang, Asif Khan, Arshad Ahmad, Rashid Naseem, Jianyun Shang, and Lin Dai. 2021. Abioner: a bert-based model for arabic biomedical named-entity recognition. Complexity, 2021:1–6. entity recognition for electronic health records: A comparison of rule-based and machine learning approaches. ArXiv, abs/1903.03985. ↗
  16. Ralph Grishman and Beth Sundheim. 1996. Message Understanding Conference- 6: A brief history. In COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics.
  17. Jiafeng Guo, Gu Xu, Xueqi Cheng, and Hang Li. 2009. Named entity recognition in query. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’09, page 267–274, New York, NY, USA. Association for Computing Machinery. find
  18. Nizar Y Habash. 2010. Introduction to arabic natural language processing. Synthesis lectures on human language technologies, 3(1):1–187. ↗
  19. Salima Harrat, Karima Meftouh, Mourad Abbas, Khaled-Walid Hidouci, and Kamel Smaili. 2016. An algerian dialect: Study and resources. International Journal of Advanced Computer Science and Applications, 7(3). ↗
  20. Salima Harrat, Karima Meftouh, Mourad Abbas, and Kamel Smaili. 2014. Building resources for algerian arabic dialects. In 15th Annual Conference of the International Communication Association Interspeech. find
  21. Chadi Helwe and Shady Elbassuoni. 2019. Arabic named entity recognition via deep co-learning. Artificial Intelligence Review, 52:197–215. ↗
  22. Abdelhalim Hafedh Dahou and Mohamed Amine Cheragui. 2022. Impact of normalization and data augmentation in ner for algerian arabic dialect. In Modelling and Implementation of Complex Systems: Proceedings of the 7th International Symposium, MISC 2022, Mostaganem, Algeria, October 30-31, 2022, pages 249–262. Springer. find
  23. Huehnergard John and Pat-El Na’ama. 2019. The semitic languages. ↗
  24. Abdelhalim Hafedh Dahou and Mohamed Amine Cheragui. 2023a. Dzner: A large algerian named entity recognition dataset. Natural Language Processing Journal, page 100005. ↗
  25. Aman Kumar and Binil Starly. 2022. “fabner”: information extraction from manufacturing process science domain literature using named entity recognition. Journal of Intelligent Manufacturing. ↗
  26. Abdelhalim Hafedh Dahou and Mohamed Amine Cheragui. 2023b. Named entity recognition for algerian arabic dialect in social media. In 12th International Conference on Information Systems and Advanced Technologies “ICISAT 2022” Intelligent Information, Data Science and Decision Support System, pages 135–145. Springer. find
  27. Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. 2022. A survey on deep learning for named entity recognition. IEEE Transactions on Knowledge and Data Engineering, 34(1):50–70. find
  28. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018a. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  29. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018b. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. ↗
  30. Ibrahim Abu El-khair. 2016. 1.5 billion words arabic corpus. ↗
  31. Mohamed Embarki. 2008. Les dialectes arabes modernes: état et nouvelles perspectives pour la classification géo-sociologique. Arabic, pages 583–604. ↗
  32. Philip John Gorinski, Honghan Wu, Claire Grover, Richard Tobin, Conn Talbot, Heather C. Whalley, Cathie L. M. Sudlow, William Whiteley, and Beatrice Alex. 2019. find
  33. Named Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. ↗
  34. Pierre Lison and Jörg Tiedemann. 2016. OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 923–929, Portorož, Slovenia. European Language Resources Association (ELRA).
  35. Pan Liu, Yanming Guo, Fenglei Wang, and Guohui Li. 2022. Chinese named entity recognition: The state of the art. Neurocomputing, 473:37–53. ↗
  36. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  37. John Maloney and Michael Niv. 1998. Tagarab: a fast, accurate arabic name recognizer using high-precision morphological analysis. In Computational approaches to semitic languages.
  38. Alireza Mansouri, Lilly Suriani Affendey, and Ali Mamat. 2008. Named entity recognition approaches. International Journal of Computer Science and Network Security, pages 339–344. find
  39. Slim Mesfar. 2007. Named entity recognition for arabic using syntactic grammars. In Natural Language Processing and Information Systems, 12th International Conference on Applications of Natural Language to Information Systems, NLDB 2007, Paris, France, June 27-29, 2007, Proceedings, volume 4592 of Lecture Notes in Computer Science, pages 305–316. Springer. find
  40. Deepali Nagrale, Vaibhav Khatavkar, and Parag Kulkarni. 2019. Document theme extraction using named-entity recognition. In Computing, Communication and Signal Processing, pages 499–509, Singapore. Springer Singapore. ↗
  41. Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. Camel tools: An open source python toolkit for arabic natural language processing. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 7022–7032.
  42. Pedro Javier Ortiz Suárez, Laurent Romary, and Benoı̂t Sagot. 2020. A monolingual approach to contextualized word embeddings for mid-resource languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1703–1714, Online. Association for Computational Linguistics.
  43. Mai Oudah and Khaled Shaalan. 2017. Nera 2.0: Improving coverage and performance of rule-based named entity recognition for arabic. Natural Language Engineering, 23(3):441–472. ↗
  44. Houda Saadane, Hosni Seffih, Christian Fluhr, Khalid Choukri, and Nasredine Semmar. 2018. Automatic identification of maghreb dialects using a dictionary-based approach. In International Conference on Language Resources and Evaluation. find
  45. Houda Saâdane. 2015. Le traitement automatique de l’arabe dialectalisé: aspects méthodologiques et algorithmiques. Ph.D. thesis, Université Grenoble Alpes. ↗
  46. Alaa Shaker, Alaa Aldarf, and Igor Bessmertny. 2023. Using lstm and gru with a new dataset for named entity recognition in the arabic language. arXiv preprint arXiv:2304.03399. ↗
  47. Samia Touileb. 2022. Nerdz: A preliminary dataset of named entities for algerian. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing, pages 95– 101. ↗
  48. Samia Touileb and Jeremy Barnes. 2021. The interplay between language similarity and script on a novel multi-layer algerian dialect corpus. arXiv preprint arXiv:2105.07400. ↗
  49. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38–45.
  50. Vikas Yadav and Steven Bethard. 2018. A survey on recent advances in named entity recognition from deep learning models. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2145–2158, Santa Fe, New Mexico, USA. Association for Computational Linguistics. ↗
  51. Omar F. Zaidan and Chris Callison-Burch. 2011. The Arabic online commentary dataset: an annotated dataset of informal Arabic with high dialectal content. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 37–41, Portland, Oregon, USA. Association for Computational Linguistics.
  52. Imad Zeroual, Dirk Goldhahn, Thomas Eckart, and Abdelhak Lakhouaja. 2019. OSIAN: Open source international Arabic news corpus - preparation and integration into the CLARIN-infrastructure. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 175– 182, Florence, Italy. Association for Computational Linguistics. ↗
  53. Wenxuan Zhou and Muhao Chen. 2021. Learning from noisy labels for entity-centric information extraction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dahou-2023-performance-analysis-arabic-pre-trained/