Elamur Research⌕
☰
home / papers / 2022 / Arabic Named Entity Recognition Using Variant Deep Neural Network Architectures and Combinatorial Feature Embedding Based on CNN, LSTM and BERT

Arabic Named Entity Recognition Using Variant Deep Neural Network Architectures and Combinatorial Feature Embedding Based on CNN, LSTM and BERT

Paper · 2022 · Manel Affi and Chiraz Latiri

Summary

The paper addresses Arabic Named Entity Recognition. Arabic's rich morphology has pushed earlier systems toward heavy hand-crafted feature engineering, and the authors avoid that entirely: no external knowledge and no manual features.

Each token is represented by concatenating three vectors:

  • a character-level vector built by a CNN,
  • a second character-level vector built by an LSTM,
  • a contextual word embedding taken from AraBERT.

This combined representation feeds a bidirectional recurrent context encoder, either Bi-LSTM or Bi-GRU. A CRF layer then decodes the tag sequence.

On the ANERCorp dataset, the authors run an incremental ablation across five embedding setups. The full combination (CLB) is best, reaching 93.34% F1 with Bi-LSTM-CRF (BLC) and 93.68% F1 with Bi-GRU-CRF (BGC). The best prior system they compare against, a BERT plus Bi-GRU model, scored 92.28%, so the gain is about 1.4 points.

The work matters because it shows two things for a morphologically complex, lower-resource language. Pretrained contextual embeddings alone leave measurable headroom, and two complementary character encoders can fill part of that gap.

Key points

  • AraBERT (v0.1) uses the BERT-Base configuration: 12 transformer layers, 12 attention heads, 768 dimensions. It was trained on roughly 70M sentences, about 3B words. A token's embedding is obtained by averaging its representations over the last four layers.
  • Five experiments per encoder:
    1. baseline RWE only
      • CNN character vectors
      • LSTM character vectors
    4. AraBERT embeddings
    5. CNN + LSTM + AraBERT (CLB)
  • Character features help a lot over the baseline. With BGC, F1 rose from 85.38 to about 90.7–90.9. The CNN character encoder beat the LSTM one by 0.12 F1 (BLC) and 0.17 F1 (BGC).
  • AraBERT was the strongest single feature source. Adding both character encoders on top of it gave the best results, and Bi-GRU beat Bi-LSTM by about 0.34 F1.
  • Systems compared on ANERCorp: BERT-BGRU (92.28), Multi-Attention (90.52 in the table), CW-Bi-LSTM-CRF (90.60), and the authors' CLB-BLC (93.34) and CLB-BGC (93.68).

Interesting ideas

  • I1. Pairing a convolutional character encoder with a recurrent one gives two different views of word shape: local affix-like n-grams from the CNN and order-sensitive spelling patterns from the LSTM. Concatenating both with contextual vectors helps unseen words without hand-written morphology rules.
  • I2. Building a token vector by averaging the top four transformer layers is a cheap and stable alternative to using only the final layer. It blends more general and more task-specific signals.
  • I3. The evaluation adds feature sources one at a time under two different encoders. This makes each source's contribution visible and shows whether a gain holds across architectures.
  • I4. A CRF decoder over the recurrent outputs enforces consistent label transitions, for example keeping multi-token entity spans coherent, instead of tagging each token independently.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): The en tagger and lemmatizer could add a dual character channel for out-of-vocabulary words such as rare names, coinages and typos, while keeping each channel's contribution inspectable.
  • I2 (tree): gparse could test last-four-layer averaging as its token input and check whether LAS improves, especially on long or noisy sentences.
  • I3 (learn): Every new feature or witness parser could get the same ladder-style ablation run on two parser backbones, so improvements are attributed honestly rather than assumed.
  • I4 (world): Entity mentions that feed the world of the text could be decoded with transition constraints, so that "New York Times" enters the world as one organization rather than broken pieces.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksNamed entity recognition · 200 works

Cite

Cite as Manel Affi and Chiraz Latiri. 2022. Arabic Named Entity Recognition Using Variant Deep Neural Network Architectures and Combinatorial Feature Embedding Based on CNN, LSTM and BERT. Proceedings of the 36th Pacific Asia Conference on Language, Information and Computation.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{affi2022arabic,
  title = {{Arabic Named Entity Recognition Using Variant Deep Neural Network Architectures and Combinatorial Feature Embedding Based on CNN, LSTM and BERT}},
  author = {Manel Affi and Chiraz Latiri},
  booktitle = {Proceedings of the 36th Pacific Asia Conference on Language, Information and Computation},
  year = {2022},
  url = {https://aclanthology.org/2022.paclic-1.33/}
}

References

41 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sherief Abdallah, Khaled Shaalan, and Muhammad Shoaib. 2012. Integrating rule-based system with classification for arabic named entity recognition. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 311–322. Springer. find
  2. Manel Affi and Chiraz Latiri. 2021. Be-blc: Bert-elmo-based deep neural network architecture for english named entity recognition task. Procedia Computer Science, 192:168–181. find
  3. Mohammad Al-Smadi, Saad Al-Zboon, Yaser Jararweh, and Patrick Juola. 2020. Transfer learning for arabic named entity recognition with deep neural networks. Ieee Access, 8:37736–37745. ↗
  4. Mohammed NA Ali, Guanzheng Tan, and Aamir Hussain. 2018. Bidirectional recurrent neural network approach for arabic named entity recognition. Future Internet, 10(12):123. ↗
  5. Mohammed Nadher Abdo Ali, Guanzheng Tan, and Aamir Hussain. 2019. Boosting arabic named-entity recognition with multi-attention layer. IEEE Access, 7:46575–46582. ↗
  6. Norah Alsaaran and Maha Alrabiah. 2021. Arabic named entity recognition: A bert-bgru approach. CMC-COMPUTERS MATERIALS & CON-TINUA, 68(1):471–485. ↗
  7. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. ↗
  8. David Awad, Caroline Sabty, Mohamed Elmahdy, and Slim Abdennadher. 2018. Arabic name entity recognition using deep learning. In International Conference on Statistical Language and Speech Processing, pages 105–116. Springer. find
  9. Yassine Benajiba and Paolo Rosso. 2008. Arabic named entity recognition using conditional random fields. In Proc. of Workshop on HLT & NLP within the Arabic World, LREC, volume 8, pages 143–153. Citeseer. find
  10. Yassine Benajiba, Paolo Rosso, and José Miguel Benedı́ruiz. 2007. Anersys: An arabic named entity recognition system based on maximum entropy. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 143–153. Springer. find
  11. Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259. ↗
  12. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of machine learning research, 12(ARTICLE):2493–2537.
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  14. Ismail El Bazi and Nabil Laachfoubi. 2019. Arabic named entity recognition using deep learning approach. International Journal of Electrical & Computer Engineering (2088-8708), 9(3). ↗
  15. Ali Farghaly and Khaled Shaalan. 2009. Arabic natural language processing: Challenges and solutions. ACM Transactions on Asian Language Information Processing (TALIP), 8(4):1–22. ↗
  16. G David Forney. 1973. The viterbi algorithm. Proceedings of the IEEE, 61(3):268–278. ↗
  17. Palash Goyal, Sumit Pandey, and Karan Jain. 2018. Deep learning for natural language processing. New York: Apress. ↗
  18. Jie Hu, Li Shen, and Gang Sun. 2018. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141. ↗
  19. Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991. ↗
  20. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2016. Character-aware neural language models. In Thirtieth AAAI conference on artificial intelligence.
  21. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. ↗
  22. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25.
  23. Matthieu Labeau, Kevin Löser, and Alexandre Allauzen. 2015. Non-lexical neural architecture for fine-grained pos tagging. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 232–237. find
  24. John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. find
  25. Siwei Lai, Kang Liu, Shizhu He, and Jun Zhao. 2016. How to generate a good word embedding. IEEE Intelligent Systems, 31(6):5–14. ↗
  26. Liyuan Liu, Jingbo Shang, Xiang Ren, Frank Xu, Huan Gui, Jian Peng, and Jiawei Han. 2018. Empower sequence labeling with task-aware neural language model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
  27. Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional lstm-cnns-crf. arXiv preprint arXiv:1603.01354. ↗
  28. Slim Mesfar. 2007. Named entity recognition for arabic using syntactic grammars. In International Conference on Application of Natural Language to Information Systems, pages 305–316. Springer. find
  29. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26. ↗
  30. Chandrahas Mishra and DL Gupta. 2017. Deep machine learning and neural networks: An overview. IAES International Journal of Artificial Intelligence, 6(2):66. ↗
  31. Naji F Mohammed and Nazlia Omar. 2012. Arabic named entity recognition using artificial neural network. Journal of Computer Science, 8(8):1285. find
  32. David Nadeau and Satoshi Sekine. 2007. A survey of named entity recognition and classification. Lingvisticae Investigationes, 30(1):3–26. ↗
  33. Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. 2016. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499. ↗
  34. Mai Oudah and Khaled Shaalan. 2017. Nera 2.0: Improving coverage and performance of rule-based named entity recognition for arabic. Natural Language Engineering, 23(3):441–472. ↗
  35. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. ↗
  36. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365. ↗
  37. D A Alzboun Sa’a, Saia Khaled Tawalbeh, Mohammad Al-Smadi, and Yaser Jararweh. 2018. Using bidirectional long short-term memory and conditional random fields for labeling arabic named entities: A comparative study. In 2018 Fifth International Conference on Social Networks Analysis, Management and Security (SNAMS), pages 135–140. IEEE. find
  38. Cicero Nogueira dos Santos and Victor Guimaraes. 2015. Boosting named entity recognition with neural character embeddings. arXiv preprint arXiv:1505.05008. ↗
  39. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  40. Wajdi Zaghouani. 2012. Renar: A rule-based arabic named entity recognition system. ACM Transactions on Asian Language Information Processing (TALIP), 11(1):1–13. find
  41. Ayah Zirikly and Mona Diab. 2015. Named entity recognition for arabic social media. In Proceedings of the 1st workshop on vector space modeling for natural language processing, pages 176–185.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/affi-2022-arabic-named-entity-recognition-variant/