Elamur Research⌕
☰
home / papers / 2022 / Enhancing Deep Learning with Embedded Features for Arabic Named Entity Recognition

Enhancing Deep Learning with Embedded Features for Arabic Named Entity Recognition

Summary

The paper asks whether explicit linguistic knowledge still helps a neural tagger for Arabic named entity recognition on Modern Standard Arabic. Arabic is hard for NER: it has rich affixation, unwritten diacritics, no capital letters to mark proper nouns, and relatively few resources. The authors build a single-layer BiLSTM-CRF in Keras. Each token is represented by a word vector, a CNN-derived character vector, and a set of one-hot features. Three of these features come from the Madamira analyzer: POS, a capitalization signal, and word analysis. A fourth, a hand-crafted quote feature, marks quotation context. Four embedding backbones are compared: AraBert, FastText, GloVe and AraVec. The extra features raise scores for every backbone, though the size of the gain depends on the embedding. AraBert gives the strongest system, with F1 of 86.71% on ANERcorp and 79.48% on AQMAR, which the authors report as better than prior work. The study shows that analyzer outputs and contextual embeddings can complement each other. It also adds an error analysis of what the model gets wrong.

Key points

  • Architecture: one BiLSTM layer with 200 units and a CRF output. Embedding dropout is 0.3 and BiLSTM dropout is 0.5. Training uses Nadam, hard-sigmoid activation and batch size 5. No validation split was kept, because putting that data into training scored higher.
  • Character vectors (dimension 100) come from a 2D CNN with tanh activation, 25 filters and kernel size 3, followed by max-pooling.
  • Word embeddings and their sizes:
    • AraBert large v2.0: 1024, taken as the mean over all layers with mean subtoken pooling, without sentence context and without fine-tuning.
    • FastText: 300.
    • GloVe: 256.
    • AraVec: 300.
  • Features come from the 2019 Madamira release, whose POS accuracy is 95.9%. A self-designed quote feature is added. All features are encoded one-hot and concatenated to the embeddings.
  • Best results come from AraBert plus features: 86.71% F1 on ANERcorp and 79.48% F1 on AQMAR.

Interesting ideas

  • I1. The model gets symbolic analyzer output (POS, morphological analysis) as one-hot vectors appended to dense embeddings, so linguistic knowledge enters without redesigning the network. — For Mova (tree): gparse could take en-crate tags, lemmas and morphological features as extra one-hot inputs beside transformer vectors, keeping parser inputs traceable to explainable upstream decisions.
  • I2. A script without letter case gets a proper-noun cue from an external analyzer, which shows that a missing surface signal can be supplied by another tool. — For Mova (tokens): for all-lowercase or all-caps English input (chat, headlines), the tagger could recover a proper-noun signal from a lexicon or analyzer instead of trusting casing.
  • I3. Feature utility is measured separately for each embedding backbone, because the same features help by different amounts depending on the underlying representation. — For Mova (learn): ablations should cross every added feature with every backbone or witness parser, so a feature is not credited with gains that only appear on a weak base.
  • I4. The authors inspect mispredicted entities because entities repeat and resemble each other within articles, which can make aggregate F1 misleading. — For Mova (learn): evaluation could report separate scores for entities unseen in training, a form of negative control against memorised names.
  • I5. AraBert is used as a frozen feature extractor, averaging all layers without fine-tuning, and it still beats the static embeddings. — For Mova (tree): frozen, layer-averaged transformer vectors could serve as a cheap baseline input for the perceptron parser before investing in fine-tuning gparse.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Named entity recognition · Semitic languages

Institutions

🎓German International UniversityCairo · Egypt2 works · 2 citations

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{hatab2022enhancing,
  title = {{Enhancing Deep Learning with Embedded Features for Arabic Named Entity Recognition}},
  author = {Hatab, Ali L. and Sabty, Caroline and Abdennadher, Slim},
  booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation Conference},
  year = {2022},
  doi = {10.63317/3jocxxfj3pgc},
  url = {https://doi.org/10.63317/3jocxxfj3pgc}
}

References

37 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Al-Jallad, A. (2017). The earliest stages of arabic and its linguistic classification. In The Routledge Handbook of Arabic Linguistics, pages 315–331. Routledge. find
  2. Ali, M. N. A., Tan, G., and Hussain, A. (2019). Boosting arabic named-entity recognition with multi-attention layer. IEEE Access, 7:46575–46582. find
  3. Awad, D., Sabty, C., Elmahdy, M., and Abdennadher, S. (2018a). Arabic name entity recognition using deep learning. In International Conference on Statistical Language and Speech Processing, pages 105–116. Springer. find
  4. Awad, D., Sabty, C., Elmahdy, M., and Abdennadher, S. (2018b). Arabic name entity recognition using deep learning. In International Conference on Statistical Language and Speech Processing, pages 105–116. Springer. find
  5. Bazi, I. E. and Laachfoubi, N. (2018). Arabic named entity recognition using word representations. arXiv preprint arXiv:1804.05630. ↗
  6. Benajiba, Y. and Rosso, P. (2008). Arabic named entity recognition using conditional random fields. In Proc. of Workshop on HLT & NLP within the Arabic World, LREC, volume 8, pages 143–153. Citeseer. find
  7. Benajiba, Y., Diab, M., Rosso, P., et al. (2008). Arabic named entity recognition: An svm-based approach. In Proceedings of 2008 Arab International Conference on Information Technology (ACIT), pages 16– 18. Citeseer. find
  8. Chiu, J. P. and Nichols, E. (2016). Named entity recognition with bidirectional lstm-cnns. Transactions of the Association for Computational Linguistics, 4:357–370. find our card
  9. Chollet, F. et al. (2015). Keras. https://keras.io. ↗
  10. El Bazi, I. and Laachfoubi, N. (2019). Arabic named entity recognition using deep learning approach. International Journal of Electrical & Computer Engineering (2088-8708), 9(3). find
  11. Helwe, C. and Elbassuoni, S. (2019). Arabic named entity recognition via deep co-learning. Artificial Intelligence Review, 52(1):197–215. find
  12. Helwe, C., Dib, G., Shamas, M., and Elbassuoni, S. (2020). A semi-supervised bert approach for arabic named entity recognition. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 49–57. find
  13. Huang, Z., Xu, W., and Yu, K. (2015). Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991. ↗ our card
  14. Khalifa, M. and Shaalan, K. (2019). Character convolutions for arabic named entity recognition with long short-term memory networks. Computer Speech & Language, 58:335–346. find
  15. Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., and Dyer, C. (2016). Neural architectures for named entity recognition. arXiv preprint arXiv:1603.01360. ↗ our card
  16. Li, J., Sun, A., Han, J., and Li, C. (2020). A survey on deep learning for named entity recognition. IEEE Transactions on Knowledge and Data Engineering. find
  17. Liu, L., Shang, J., and Han, J. (2019). Arabic named entity recognition: what works and what’s next. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 60–67. find
  18. Maloney, J. and Niv, M. (1998). Tagarab: a fast, accurate arabic name recognizer using high-precision morphological analysis. In Computational approaches to semitic languages. find
  19. Meselhi, M. A., Bakr, H. M. A., Ziedan, I., and Shaalan, K. (2014). A novel hybrid approach to arabic named entity recognition. In China Workshop on Machine Translation, pages 93–103. Springer. find
  20. Mesfar, S. (2007). Named entity recognition for arabic using syntactic grammars. In International Conference on Application of Natural Language to Information Systems, pages 305–316. Springer. find
  21. Nakayama, H. (2018). seqeval: A python framework for sequence labeling evaluation. Software available from https://github.com/chakki-works/seqeval. ↗
  22. Sabty, C., Elmahdy, M., and Abdennadher, S. (2018). Arabic named entity recognition using clustered word embedding. find
  23. Sabty, C., Elmahdy, M., and Abdennadher, S. (2019a). Named entity recognition on arabic-english code-mixed data. In 2019 IEEE 13th International Conference on Semantic Computing (ICSC), pages 93– 97. IEEE. find
  24. Sabty, C., Sherif, A., Elmahdy, M., and Abdennadher, S. (2019b). Techniques for named entity recognition on arabic-english code-mixed data. International Journal of Transdisciplinary AI, 1(1):44–63. find
  25. Sang, E. F. and De Meulder, F. (2003). Introduction to the conll-2003 shared task: Language-independent named entity recognition. arXiv preprint cs/0306050. find our card
  26. Shaalan, K. and Oudah, M. (2014). A hybrid approach to arabic named entity recognition. Journal of Information Science, 40(1):67–87. find
  27. Wu, M., Liu, F., and Cohn, T. (2018). Evaluating the utility of hand-crafted features in sequence labelling. arXiv preprint arXiv:1808.09075. ↗
  28. Youssef, A., Elattar, M., and El-Beltagy, S. R. (2020). A multi-embeddings approach coupled with deep learning for arabic named entity recognition. In 2020 2nd Novel Intelligent and Leading Emerging Sciences Conference (NILES), pages 456–460. find
  29. Zhang, X., Zhao, J., and LeCun, Y. (2015). Character-level convolutional networks for text classification. Advances in neural information processing systems, 28:649–657. 8. find our card
  30. Language Resource References Antoun, W., Baly, F., and Hajj, H. (2020). Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. ↗ our card
  31. Benajiba, Y., Rosso, P., and Benedı́ruiz, J. M. (2007). Anersys: An arabic named entity recognition system based on maximum entropy. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 143–153. Springer. find
  32. Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T. (2017). Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. find our card
  33. Mohit, B., Schneider, N., Bhowmick, R., Oflazer, K., and Smith, N. A. (2012). Recall-oriented learning of named entities in arabic wikipedia. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 162–173. find
  34. Obeid, O., Zalmout, N., Khalifa, S., Taji, D., Oudah, M., Alhafni, B., Inoue, G., Eryani, F., Erdmann, A., and Habash, N. (2020). Camel tools: An open source python toolkit for arabic natural language processing. In Proceedings of the 12th language resources and evaluation conference, pages 7022– 7032. find our card
  35. Pasha, A., Al-Badrashiny, M., Diab, M. T., El Kholy, A., Eskander, R., Habash, N., Pooleery, M., Rambow, O., and Roth, R. (2014). Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of arabic. In Lrec, volume 14, pages 1094–1101. Citeseer. find our card
  36. Soliman, A. B., Eissa, K., and El-Beltagy, S. R. (2017). Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256–265. find
  37. Tarekeldeeb. (2018). tarekeldeeb/glove-arabic: Glove model for distributed arabic word representation. https://github.com/tarekeldeeb/GloVe-Arabic. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hatab-2022-enhancing-deep-learning-embedded-features/