Elamur Research⌕
☰
home / papers / 2023 / Alex-U 2023 NLP at WojoodNER shared task: AraBINDER (Bi-encoder for Arabic Named Entity Recognition)

Alex-U 2023 NLP at WojoodNER shared task: AraBINDER (Bi-encoder for Arabic Named Entity Recognition)

Paper · 2023 · Mariam Hussein, Sarah Khaled, Marwan Torki et al.

Summary

The paper tackles Arabic named entity recognition for the WojoodNER 2023 shared task, which has a flat track and a nested track. Arabic is hard here because of its morphology and because annotated data is scarce. The authors adapt BINDER, a bi-encoder originally shown for English and Chinese, and call the result AraBINDER. One transformer encodes textual descriptions of entity types and a second, separate transformer encodes the sentence. Candidate spans are then scored against each type in a shared vector space using contrastive training. Because spans are matched to types independently, overlapping (nested) entities need no special decoding. Non-entities are not given an "O" label. Instead, a threshold computed per input from the [CLS] representation decides whether a span counts as an entity. The encoders are built on AraBERTv2-Twitter, which beat CAMeLBERT and the other AraBERT variants. On the test set the system reaches micro F1 0.918 for flat NER and 0.90 for nested NER. This shows that retrieval-style span typing carries over to a morphologically rich, lower-resource language.

Key points

  • Data: the Wojood corpus has about 27K sentences and 550K tokens, covering MSA and dialectal Arabic across several domains. It contains about 75K entities, 22.5% of them nested, labelled with 21 types (including the new OCC, WEBSITE, UNIT and CURR tags). The split is 70/10/20.
  • Training loss: a weighted sum of a start-position loss, an end-position loss and a span loss. A similarity score between [CLS] and each entity type acts as an input-specific threshold for rejecting non-entities.
  • Setup: AraBERTv2-Twitter base, sequences of up to 128 tokens with a stride of 16 and sentence boundaries ignored, 20 epochs, learning rate 3e-5, batch size 8, one V100 GPU.
  • Results: on the development set, flat F1 is 0.916 and nested F1 is 0.929. On the test set, flat reaches F1 0.918 (P 0.914, R 0.924) and nested reaches F1 0.90, slightly under the 0.91 of the Jarrar et al. baseline.
  • Error source: the Arabic conjunction "و" attaches directly to the following name. As a result, two separate people can be predicted as a single person entity.

Interesting ideas

  • I1. Entity types are written as natural-language descriptions and embedded in the same space as text spans, so classifying a span becomes nearest-neighbour matching rather than choosing from fixed softmax labels. This makes adding a new type cheap.
  • I2. Instead of a catch-all "outside" class, the model learns a cutoff for each input from a summary of the whole context, and a span must score above that cutoff to count.
  • I3. A separate objective for start and end boundaries means near-miss spans that share one edge with the gold span are not penalised as harshly as completely wrong spans.
  • I4. The model scored higher on nested than flat NER on the dev set but the order flipped on the test set. The authors read this as a sign of poor generalization rather than a real advantage.
  • I5. The main error type comes from a clitic conjunction joined to the next name, which merges two people into one entity. This shows that tokenization decisions directly cap how well entities can be segmented.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): Entity typing could compare a mention's representation against embedded WordNet glosses or Wikidata class descriptions, and the chosen gloss would itself explain the assigned type.
  • I2 (frames): When deciding whether a span fills any semantic role at all, a context-dependent "no role" threshold could replace a single global confidence cutoff.
  • I3 (learn): Silver-data training and scoring for span-level outputs (arguments, mentions) could give graded credit for partial boundary overlap, which would reveal boundary errors separately from labelling errors.
  • I4 (learn): Evaluation reports should check whether rankings between conditions stay the same across held-out splits before drawing conclusions, much like a negative control.
  • I5 (tokens): Cases where English coordination or possessive splitting interacts with proper names (e.g. "Smith and Wesson" vs. two people) should be audited so that the tokenizer and tagger do not fuse or split entities before parsing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Alexandria UniversityAlexandria · Egypt39 works · 287 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksNamed entity recognition · 200 works

Cite

Cite as Mariam Hussein, Sarah Khaled, Marwan Torki and Nagwa El-Makky. 2023. Alex-U 2023 NLP at WojoodNER shared task: AraBINDER (Bi-encoder for Arabic Named Entity Recognition). Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.90

@inproceedings{hussein2023alex,
  title = {{Alex-U 2023 NLP at WojoodNER shared task: AraBINDER (Bi-encoder for Arabic Named Entity Recognition)}},
  author = {Mariam Hussein, Sarah Khaled, Marwan Torki and Nagwa El-Makky},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.90},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.90}
}

References

25 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed and Laks VS Lakshmanan. 2022. A benchmark study of contrastive learning for arabic social meaning. WANLP 2022, page 63.
  2. Manel Affi and Chiraz Latiri. 2022. Arabic named entity recognition using variant deep neural network architectures and combinatorial feature embedding based on cnn, lstm and bert. In Proceedings of the 36th Pacific Asia Conference on Language, Information and Computation, pages 302–312.
  3. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15. ↗
  4. Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. 1993. Signature verification using a" siamese" time delay neural network. Advances in neural information processing systems, 6. ↗
  5. Sumit Chopra, Raia Hadsell, and Yann LeCun. 2005. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), volume 1, pages 539–546. IEEE. find
  6. Abbas Ghaddar, Yimeng Wu, Sunyam Bagga, Ahmad Rashid, Khalil Bibi, Mehdi Rezagholizadeh, Chao Xing, Yasheng Wang, Xinyu Duan, Zhefeng Wang, et al. 2022. Revisiting pre-trained language models and their evaluation for arabic natural language processing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3135–3151.
  7. Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6112–6121.
  8. Daniel Gillick, Alessandro Presta, and Gaurav Singh Tomar. 2018. End-to-end retrieval in continuous space. arXiv preprint arXiv:1811.08008. ↗
  9. Ridewaan Hanslo. 2022. Deep learning transformer architecture for named-entity recognition on low-resourced languages: State of the art results. In 2022 17th Conference on Computer Science and Intelligence Systems (FedCSIS), pages 53–60. IEEE. find
  10. Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2019. Poly-encoders: Architectures and pre-training strategies for fast and accurate multi-sentence scoring. In International Conference on Learning Representations. find
  11. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine (Online). Association for Computational Linguistics.
  12. Mustafa Jarrar, Muhammad Abdul-Mageed, Mohammed Khalilia, Bashar Talafha, AbdelRahim Elmadany, Nagham Hamad, and Alaa Omer. 2023. WojoodNER: The Arabic Named Entity Recognition Shared Task. In Proceedings of The First Arabic Natural Language Processing Conference (ArabicNLP 2023). find
  13. Mustafa Jarrar, Mohammed Khalilia, and Sana Ghanem. 2022. Wojood: Nested arabic named entity corpus and recognition using bert. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3626–3636.
  14. Kareem Darwish and Hamdy Mubarak. 2016. Farasa: A new fast and accurate arabic word segmenter. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 1070–1074.
  15. Shereen Elkordi, Noha Adly, and Marwan Torki. 2023. Alexu-aic at wojoodner shared task: Sequence labeling vs mrc and swa for arabic named entity recog-Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781.
  16. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume 1, page 2.
  17. Xiaoya Li, Jingrong Feng, Yuxian Meng, Qinghong Han, Fei Wu, and Jiwei Li. 2020. A unified mrc framework for named entity recognition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5849–5859.
  18. Xiaoye Qu, Yingjie Gu, Qingrong Xia, Zechang Li, Zhefeng Wang, and Baoxing Huai. 2023. A survey on arabic named entity recognition: Past, recent advances, and future trends. arXiv preprint arXiv:2302.03512. ↗
  19. Ahmad Shapiro, Ayman Khalafallah, and Marwan Torki. 2022. Alexu-aic at arabic hate speech 2022: Contrast to classify. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur’an QA and Fine-Grained Hate Speech Detection, pages 200–208. ↗
  20. Yu Wang, Yun Li, Ziye Zhu, Bin Xia, and Zheng Liu. 2019. Sc-ner: A sequence-to-sequence model with sentence classification for named entity recognition. In Advances in Knowledge Discovery and Data Mining: 23rd Pacific-Asia Conference, PAKDD 2019, Macau, China, April 14-17, 2019, Proceedings, Part I 23, pages 198–209. Springer. find
  21. Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2020. Scalable zero-shot entity linking with dense entity retrieval. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6397–6407.
  22. Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019. End-to-end open-domain question answering with bertserini.
  23. Juntao Yu, Bernd Bohnet, and Massimo Poesio. 2020. Named entity recognition as dependency parsing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6470– 6476.
  24. Haoyu Zhang, Jingjing Cai, Jianjun Xu, and Ji Wang. 2019. Pretraining-based natural language generation for text summarization. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL). Association for Computational Linguistics. find
  25. Sheng Zhang, Hao Cheng, Jianfeng Gao, and Hoifung Poon. 2022. Optimizing bi-encoder for named entity recognition via contrastive learning. In The Eleventh International Conference on Learning Representations. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hussein-2023-alex-u-2023-nlp-wojoodner/