Elamur Research⌕
☰
home / papers / 2023 / AlphaBrains at WojoodNER shared task: Arabic Named Entity Recognition by Using Character-based Context-Sensitive Word Representations

AlphaBrains at WojoodNER shared task: Arabic Named Entity Recognition by Using Character-based Context-Sensitive Word Representations

Paper · 2023 · Toqeer Ehsan, Amjad Ali and Ala Al-Fuqaha

Summary

The paper describes the AlphaBrains submission to the WojoodNER 2023 shared task on Arabic named entity recognition. The task has two tracks: flat NER and nested NER, where entities can sit inside other entities. The authors use a version of the Wojood corpus split 70/10/20 into train, dev and test. They build BiLSTM sequence labelers whose input combines learned word embeddings with pre-trained character-based ELMo vectors for Arabic. They compare two designs. In the single-task (STL) design, one softmax head predicts one tag per token, and nested tags are merged into compound labels. In the multi-task (MTL) design, each of 21 entity types gets its own softmax head. STL performed best in both tracks, with micro F1 of 0.8751 on flat NER (10th place) and 0.8884 on nested NER (7th place). It beat the flat baseline and an earlier BERT-based nested result, but stayed below the transformer baseline for nested NER. The paper's main lesson is that contextual character-level embeddings helped far more than POS features or Word2Vec.

Key points

  • Data sizes: 16,817 / 3,133 / 5,989 sentences and 394,499 / 55,826 / 111,951 tokens for train / dev / test. Labels use an IOB scheme over 21 entity types such as GPE, ORG, DATE, PERS and OCC.
  • Architecture and training:
    • three stacked BiLSTM layers, each with 256 units and dropout 0.2;
    • 256-dimensional internal embeddings plus 1024-dimensional third-layer ELMo vectors (ELMoForManyLangs);
    • RMSprop with learning rate 0.001, 15 epochs, batch size 128, sequence length 256;
    • implemented in Keras/TensorFlow.
  • Flat NER (micro F1, scored with seqeval): baseline 0.8681, STL 0.8751, MTL 0.8726.
  • Nested NER (micro F1): transformer baseline 0.9047, Jarrar et al. (2022) 0.8840, STL 0.8884, MTL 0.8846.
    • For STL, nested tags were joined with the delimiter "∼", which produced 298 distinct composite labels.
  • Weaker feature variants:
    • Stanford POS tags concatenated to the input left F1 around 0.78.
    • Arabic Word2Vec stayed below 0.82.
    • ELMo was the component that made the system competitive.

Interesting ideas

  • I1. Nested annotation can be collapsed into ordinary single-tag labeling by gluing all layers' tags into one composite label, at the cost of a larger label set (298 here). This turns an overlapping-structure problem into a plain sequence-labeling problem.
  • I2. An alternative to composite labels is one output head per entity type, each choosing IOB-or-O. This lets a token carry several overlapping memberships without enumerating every combination.
  • I3. When the per-type heads disagree, the paper falls back on a deliberately naive, explicit tie-break: it keeps the left-most predicted label. Such a rule is crude but transparent, and its cost can be measured against the single-head model.
  • I4. Character-based contextual input gives usable vectors for out-of-vocabulary words, where context-free word vectors fail.
  • I5. The authors ran a simple feature ablation: POS tags, then Word2Vec, then ELMo, under the same architecture. It showed that POS features gave no gain while contextual character embeddings did most of the work.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): a tagger could predict joint labels (e.g., POS plus a morphological feature bundle) as single composite tags, keeping one decision per token that is easy to explain.
  • I2 (world): entity typing could run independent per-type detectors, so a span can be both an organization and part of a place name, with each head's evidence inspectable separately.
  • I3 (learn): conflicts between witness parsers could first be resolved by a stated simple rule, used as a control baseline, before any learned arbitration is trusted.
  • I4 (tokens): feeding character-level features into the tagger and lemmatizer would help with rare names, misspellings and novel compounds in tales and school texts.
  • I5 (learn): when evaluating parser or tagger inputs, the same kind of one-feature-at-a-time comparison would reveal which signals actually earn their place in the pipeline.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hamad bin Khalifa UniversityDoha · Qatar141 works · 14,175 citations

Also on the byline: University of Gujrat.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksNamed entity recognition · 200 works

Cite

Cite as Toqeer Ehsan, Amjad Ali and Ala Al-Fuqaha. 2023. AlphaBrains at WojoodNER shared task: Arabic Named Entity Recognition by Using Character-based Context-Sensitive Word Representations. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.88

@inproceedings{ehsan2023alphabrains,
  title = {{AlphaBrains at WojoodNER shared task: Arabic Named Entity Recognition by Using Character-based Context-Sensitive Word Representations}},
  author = {Toqeer Ehsan, Amjad Ali and Ala Al-Fuqaha},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.88},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.88}
}

References

30 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ankit Agrawal, Sarsij Tripathi, Manu Vardhan, Vikas Sihag, Gaurav Choudhary, and Nicola Dragoni. 2022. BERT-based transfer-learning approach for nested named-entity recognition using joint labeling. Applied Sciences, 12(3):976. ↗
  2. Muhammad Tayyab Ahmad, Muhammad Kamran Malik, Khurram Shahzad, Faisal Aslam, Asif Iqbal, Zubair Nawaz, and Faisal Bukhari. 2020. Named entity recognition and classification for Punjabi Shahmukhi. ACM Transactions on Asian and Low-Resource Language Information Processing (TAL-LIP), 19(4):1–13. ↗
  3. Moustafa Al-Hajj and Mustafa Jarrar. 2022. ArabGloss-Bert: Fine-tuning BERT on Context-Gloss Pairs for WSD. arXiv preprint arXiv:2205.09685. ↗
  4. Veera Sekhar Reddy Bhumireddypalli, Srinivas Rao Koppula, and Neeraja Koppula. 2023. Enhanced conditional random field-long short-term memory for name entity recognition in English texts. Concurrency and Computation: Practice and Experience, 35(9):e7640. find
  5. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the association for computational linguistics, 5:135–146. ↗
  6. Wanxiang Che, Yijia Liu, Yuxuan Wang, Bo Zheng, and Ting Liu. 2018. Towards better UD parsing: Deep contextualized word embeddings, ensemble, and treebank concatenation. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 55–64, Brussels, Belgium. Association for Computational Linguistics. ↗
  7. Leyang Cui, Yu Wu, Jian Liu, Sen Yang, and Yue Zhang. 2021. Template-based Named Entity Recognition using BART. arXiv preprint arXiv:2106.01760. ↗
  8. Sławomir Dadas and Jarosław Protasiewicz. 2020. A bidirectional iterative algorithm for nested named entity recognition. IEEE Access, 8:135091–135102. ↗
  9. Xiaojing Du, Yuxiang Jia, and Hongying Zan. 2022. MRC-based Medical NER with Multi-task Learning and Multi-strategies. In China National Conference on Chinese Computational Linguistics, pages 149– 162. Springer. find
  10. Qin Fang, Yane Li, Hailin Feng, and Yaoping Ruan. 2023. Chinese Named Entity Recognition Model Based on Multi-Task Learning. Applied Sciences, 13(8):4770. ↗
  11. Murhaf Fares, Andrey Kutuzov, Stephan Oepen, and Erik Velldal. 2017. Word vectors, reuse, and replicability: Towards a community repository of large-text resources. In Proceedings of the 21st Nordic Conference on Computational Linguistics, pages 271–276, Gothenburg, Sweden. Association for Computational Linguistics. find
  12. Xiaocheng Feng, Bing Qin, and Ting Liu. 2018. A language-independent neural network for event detection. Science China Information Sciences, 61:1–12. ↗
  13. Mustafa Jarrar, Muhammad Abdul-Mageed, Mohammed Khalilia, Bashar Talafha, AbdelRahim Elmadany, Nagham Hamad, and Alaa Omer. 2023. WojoodNER 2023: The First Arabic Named Entity Recognition Shared Task. In Proceedings of the First Arabic Natural Language Processing Conference (ArabicNLP 2023).
  14. Mustafa Jarrar, Mohammed Khalilia, and Sana Ghanem. 2022. Wojood: Nested Arabic Named Entity Corpus and Recognition using BERT. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3626–3636.
  15. Mohammad Ebrahim Khademi and Mohammad Fakhredanesh. 2020. Persian Automatic Text Summarization based on Named Entity Recognition. Iranian Journal of Science and Technology, Transactions of Electrical Engineering, pages 1–12. ↗
  16. Lobna Ahmed Mady, Yasmine A Afify, and Nagwa Badr. 2022. Nested Biomedical Named Entity Recognition. International Journal of Intelligent Computing and Information Sciences, 22(1):98–107. ↗
  17. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26. ↗
  18. Nita Patil, Ajay Patil, and BV Pawar. 2020. Named entity recognition using conditional random fields. Procedia Computer Science, 167:1181–1188. find
  19. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: Global Vectors for Word Representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. ↗
  20. Van-Hai Vu, Quang-Phuoc Nguyen, Kiem-Hieu Nguyen, Joon-Choul Shin, and Cheol-Young Ock. 2020. Korean-vietnamese neural machine translation with named entity recognition and part-of-speech tags. IEICE Transactions on Information and Systems, 103(4):866–873. ↗
  21. Jue Wang, Lidan Shou, Ke Chen, and Gang Chen. 2020. Pyramid: A layered model for nested named entity recognition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5918–5928. find
  22. Yibo Yan, Peng Zhu, Dawei Cheng, Fangzhou Yang, and Yifeng Luo. 2023. Adversarial Multi-Task Learning for Efficient Chinese Named Entity Recognition. ACM Transactions on Asian and Low-Resource Language Information Processing. ↗
  23. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations. arXiv preprint arXiv:1802.05365. ↗
  24. Gorjan Popovski, Barbara Koroušić Seljak, and Tome Eftimov. 2020. A survey of named-entity recognition methods for food information extraction. IEEE Access, 8:31586–31594. ↗
  25. Mohammad Golam Sohrab and Makoto Miwa. 2018. Deep exhaustive model for nested named entity recognition. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 2843–2849.
  26. Abu Bakr Soliman, Kareem Eissa, and Samhaa R El-Beltagy. 2017. AraVec: A set of Arabic Word Embedding Models for use in Arabic NLP. Procedia Computer Science, 117:256–265. find
  27. Jana Straková, Milan Straka, and Jan Hajič. 2019. Neural architectures for nested NER through linearization. arXiv preprint arXiv:1908.06926. ↗
  28. Amina Tehseen, Toqeer Ehsan, Hannan Bin Liaqat, Amjad Ali, and Ala Al-Fuqaha. 2022. Neural POS Tagging of Shahmukhi by Using Contextualized Word Representations. Journal of King Saud University-Computer and Information Sciences. ↗
  29. Amina Tehseen, Toqeer Ehsan, Hannan Bin Liaqat, Xiangjie Kong, Amjad Ali, and Ala Al-Fuqaha. 2023. Shahmukhi Named Entity Recognition by using Contextualized Word Embeddings. Expert Systems with Applications, 229:120489. ↗
  30. Kristina Toutanova, Dan Klein, Christopher D Manning, and Yoram Singer. 2003. Feature-rich Part-of-Speech tagging with a cyclic dependency network. In Proceedings of the 2003 human language technology conference of the north american chapter of the association for computational linguistics, pages 252–259.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ehsan-2023-alphabrains-wojoodner-shared-task-arabic/