Elamur Research⌕
☰
home / papers / 2023 / ELYADATA at WojoodNER Shared Task: Data and Model-centric Approaches for Arabic Flat and Nested NER

ELYADATA at WojoodNER Shared Task: Data and Model-centric Approaches for Arabic Flat and Nested NER

Paper · 2023 · Imen Laouirine, Haroun Elleuch and Fethi Bougares

Summary

The paper describes ELYADATA's entries in the WojoodNER shared task at ArabicNLP 2023, which covered both flat and nested named entity recognition for Arabic text that mixes MSA and dialects. The team tested two kinds of improvement. The data-centric one rebalanced the skewed Wojood training data by re-sampling sentences and also tried text cleaning. The model-centric one fine-tuned two span-oriented architectures, PIQN and DiffusionNER, on top of several BERT-family encoders.

First, the team reproduced the organizers' AraBERT multitask baseline. After that, most of the gains came from the choice of model, not from changes to the data:

  • Re-sampling gave at most small improvements.
  • Removing stop-words and punctuation sharply lowered scores.
  • DiffusionNER, which treats entity detection as denoising noisy span boundaries, combined with AraBERT, was the strongest setup.

It ranked first of eight in the nested sub-task and third of eleven in the flat one. The work matters because nested NER for Arabic has received little study. It also shows that domain-matched encoders and boundary-denoising decoders can beat data manipulation tricks.

Key points

  • Wojood data splits:
    • train: 16,817 sentences, average length about 23.45 words;
    • dev: 3,133 sentences;
    • test: 5,990 sentences.
    • The most frequent class is GPE in nested and ORG in flat. The rarest are PRODUCT (nested) and UNIT (flat).
  • Re-sampling on the flat dev set (methods from Wang and Wang, 2022: sC, sCR, sCRD, nsCRD):
    • sC raised the baseline from 87.55 to 87.92;
    • sCR gave only a marginal gain;
    • the density-based variants sCRD and nsCRD lowered scores.
  • Cleaning hurt badly. Dropping 974 stop-words plus ! and ? cut the sC model to 80.71 F1, so cleaning was abandoned.
  • Encoder results:
    • BioBERT performed below the baseline;
    • AraBERT outperformed MARBERT; with DiffusionNER, nested dev F1 fell from 93.19 to 90.39 when MARBERT was used;
    • PIQN+AraBERT reached 90.59 (flat) and 92.54 (nested) on dev;
    • adding re-sampling to PIQN made it worse.
  • Official test results:
    • DiffusionNER_AraBERT: 93.73 micro F1 on nested (1st) and 91.92 on flat (3rd);
    • PIQN_AraBERT auxiliary runs: 91.86 nested and 90.87 flat.
    • Stated limitations: model complexity and GPU cost, a 512-token cap handled by splitting long sentences, and reliance on labeled data.

Interesting ideas

  • I1. Before testing any improvement, the authors first re-trained the organizers' baseline and checked that it matched the published numbers, so every later gain is measured against a verified reference point.
  • I2. Sentences can be re-sampled according to how many labeled items they contain, how rare those items are, how dense they are, and how varied their labels are, yet here the simplest count-based scheme helped most and the fancier ones hurt.
  • I3. Entity extraction is framed as recovering span boundaries from random noise through learned iterative denoising, which handles overlapping and nested spans without a token-level tagging scheme.
  • I4. Aggressively stripping stop-words and punctuation removed context the model needed and caused a large score drop, so "cleaner" input was actually less informative.
  • I5. A plausible hypothesis that a dialect-pretrained encoder would suit mixed-dialect data was tested and turned out false; the authors suggest the encoder's pretraining domain (tweets) or the data's actual composition as possible causes.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Each new training recipe for the perceptron parser or gparse could begin by reproducing the previous scores on the same split, so improvements are not illusions caused by a drifted setup.
  • I2 (learn): Silver treebank sentences containing rare UD relations could be oversampled with a plain count-based weight, with each more elaborate weighting kept only if it beats that baseline in an ablation.
  • I3 (world): Entity-mention detection for the world model could predict start/end spans directly instead of BIO tags, so nested mentions like an organization inside a location are kept rather than flattened.
  • I4 (tokens): The tokenizer and tagger should keep function words and punctuation intact, since they carry the structural cues that later UD parsing and frame building rely on.
  • I5 (tree): When choosing or swapping the transformer encoder for gparse, the "better matched" candidate should be benchmarked against the incumbent, and the real genre mix of the target text checked, before adopting it.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksNamed entity recognition · 200 works

Cite

Cite as Imen Laouirine, Haroun Elleuch and Fethi Bougares. 2023. ELYADATA at WojoodNER Shared Task: Data and Model-centric Approaches for Arabic Flat and Nested NER. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.84

@inproceedings{laouirine2023elyadata,
  title = {{ELYADATA at WojoodNER Shared Task: Data and Model-centric Approaches for Arabic Flat and Nested NER}},
  author = {Imen Laouirine, Haroun Elleuch and Fethi Bougares},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.84},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.84}
}

References

21 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yassine Benajiba, Paolo Rosso, and José Miguel BenedíRuiz. 2007b. Anersys: An arabic named entity recognition system based on maximum entropy. In Computational Linguistics and Intelligent Text Processing, pages 143–153, Berlin, Heidelberg. Springer Berlin Heidelberg. find
  2. Darina Benikova, Chris Biemann, Max Kisselew, and Sebastian Padó. 2014. Germeval 2014 named entity recognition shared task. find
  3. Kareem Darwish, Nizar Habash, Mourad Abbas, Hend Al-Khalifa, Huseein T. Al-Natsheh, Houda Bouamor, Karim Bouzoubaa, Violetta Cavalli-Sforza, Samhaa R. El-Beltagy, Wassim El-Hajj, Mustafa Jarrar, and Hamdy Mubarak. 2021. A panoramic survey of natural language processing in the arab world. Commun. ACM, 64(4):72–81. ↗
  4. Sarkar Snigdha Sarathi Das, Arzoo Katiyar, Rebecca Passonneau, and Rui Zhang. 2022. CONTaiNER: Few-shot named entity recognition via contrastive learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6338–6353, Dublin, Ireland. Association for Computational Linguistics.
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  6. Mourad Gridach. 2016. Deep learning approach for arabic named entity recognition. In Computational Linguistics and Intelligent Text Processing: 17th International Conference, CICLing 2016, Konya, Turkey, April 3–9, 2016, Revised Selected Papers, Part I 17, pages 439–451. Springer. find
  7. Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pages 6840–6851. Curran Associates, Inc. ↗
  8. Mustafa Jarrar, Muhammad Abdul-Mageed, Mohammed Khalilia, Bashar Talafha, AbdelRahim Elmadany, Nagham Hamad, and Alaa Omer. 2023. WojoodNER: The Arabic Named Entity Recognition Shared Task. In Proceedings of The First Arabic Natural Language Processing Conference (ArabicNLP 2023). find
  9. Mustafa Jarrar, Mohammed Khalilia, and Sana Ghanem. 2022. Wojood: Nested Arabic named entity corpus and recognition using BERT. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3626–3636, Marseille, France. European Language Resources Association.
  10. Basra Jehangir, Saravanan Radhakrishnan, and Rahul Agarwal. 2023. A survey on named entity recognition – datasets, tools, and methodologies. Natural Language Processing Journal, 3:100017. ↗
  11. Diederik P. Kingma and Jimmy Ba. 2017. Adam: A method for stochastic optimization. find
  12. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240. ↗
  13. Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Diffusion-NER: Boundary diffusion for named entity recognition. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3875–3890, Toronto, Canada. Association for Computational Linguistics. find
  14. Yongliang Shen, Xiaobin Wang, Zeqi Tan, Guangwei Xu, Pengjun Xie, Fei Huang, Weiming Lu, and Yueting Zhuang. 2022. Parallel instance query network for named entity recognition. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 947–961, Dublin, Ireland. Association for Computational Linguistics. ↗
  15. Erik F. Tjong Kim Sang. 2002. Introduction to the CoNLL-2002 shared task: Language-independent named entity recognition. In COLING-02: The 6th Conference on Natural Language Learning 2002 (CoNLL-2002). ↗
  16. Tatiana Tsygankova, Stephen Mayhew, and Dan Roth. 2019. BSNLP2019 shared task submission: Multi-source neural NER transfer. In Proceedings of the 7th Workshop on Balto-Slavic Natural Language Processing, pages 75–82, Florence, Italy. Association for Computational Linguistics. find
  17. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
  18. Xiaochen Wang and Yue Wang. 2022. Sentence-level resampling for named entity recognition. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2151–2165, Seattle, United States. Association for Computational Linguistics. ↗
  19. Huyen Thi Minh Nguyen, Quyen The Ngo, Luong X Vu, Vu Mai Tran, and Hien T. Nguyen. 2019. Vlsp shared task: Named entity recognition. Journal of Computer Science and Cybernetics. ↗
  20. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. ↗
  21. Xiaoye Qu, Yingjie Gu, Qingrong Xia, Zechang Li, Zhefeng Wang, and Baoxing Huai. 2023. A survey on arabic named entity recognition: Past, recent advances, and future trends. arXiv preprint arXiv:2302.03512. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/laouirine-2023-elyadata-wojoodner-shared-task-data/