Elamur Research⌕
☰
home / papers / 2023 / UM6P & UL at WojoodNER shared task: Improving Multi-Task Learning for Flat and Nested Arabic Named Entity Recognition

UM6P & UL at WojoodNER shared task: Improving Multi-Task Learning for Flat and Nested Arabic Named Entity Recognition

Paper · 2023 · Abdelkader El Mahdaouy, Salima Lamsiyah, Hamza Alami et al.

Summary

The paper describes a system entered in the WojoodNER 2023 shared task, which covers flat and nested named entity recognition for Arabic, spanning Modern Standard Arabic and dialects. The data is the Wojood corpus: about 550k tokens, 21 entity types, and annotation that follows the OntoNotes 5 guidelines. A single BERT-style encoder is shared by many classification heads, one per entity type, and each head assigns IOB2 tags to sub-word tokens. Because overlapping entities of different types are handled by different heads, the same model serves both the flat and the nested sub-task. The authors compare three Arabic encoders (QARiB, CAMeLBERT-Mix, ARBERTv2) and four training objectives. The strongest objective is a weighted blend of cross-entropy, Dice, Tversky and Focal losses, plus a penalty on the variance of the per-head losses. Their best setup, ARBERTv2 with the blended loss and the variance penalty, reached a test micro-F1 of 0.9113 on flat NER (6th place) and 0.9303 on nested NER (2nd place). The main lesson is that imbalance-aware loss mixing gives clear gains without changing the architecture.

Key points

  • Data: Wojood is split 70/10/20 into train, dev and test. Inter-annotator agreement is high (macro Cohen's Kappa of 0.98 with the 'O' tag, 0.979 without it). The authors apply no text preprocessing or normalization.
  • Loss blend: The unified loss weights are 0.4 for cross-entropy and 0.2 each for Dice, Tversky and Focal. The variance penalty weight is p = 5, Tversky α = 0.5 and Focal γ = 2. These values were fixed from preliminary runs, not tuned.
  • Training: 15 epochs, batch size 16, learning rate 2e-5 and weight decay 1e-3. Biases and LayerNorm are excluded from weight decay. The code builds on the ArabicNER baseline.
  • Flat test F1 (ARBERTv2): 0.8837 with plain cross-entropy, 0.9095 with the unified loss, and 0.9113 with the variance penalty added.
  • Nested test F1 (ARBERTv2): 0.9185, then 0.9287, then 0.9303 across the same three objectives.
  • Gap to the winners: 0.0083 F1 in Sub-Task 1 and 0.007 in Sub-Task 2. ARBERTv2 was the best encoder in both sub-tasks.

Interesting ideas

  • I1. Nested structure is handled by giving each entity type its own tagging head over a shared encoder, so overlapping spans never compete for one label slot. This is a cheap way to allow multi-layer annotation.
  • I2. A penalty on the variance of the per-task losses pushes all heads to improve together, so the shared encoder does not favour easy tasks at the expense of harder ones.
  • I3. Cross-entropy is blended with overlap- and difficulty-oriented losses (Dice, Tversky, Focal), which targets rare classes; this alone gave the largest jump in F1.
  • I4. The Tversky loss has a single knob, α, that shifts the cost between false positives and false negatives. This lets a component be biased toward precision or toward recall on purpose.
  • I5. Every combination of encoder and loss is reported on both dev and test, which separates the effect of the representation from the effect of the objective.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): a token often plays roles in several frames at once. One role-tagging layer per frame type could capture this without inventing a complex combined label scheme.
  • I2 (learn): when training gparse jointly on arcs, deprels and tags, this term could keep weaker heads from lagging behind. Its effect should be checked against a run without the penalty.
  • I3 (tree): rare dependency relations (e.g., vocative, reparandum) could get more weight in parser training without resampling the treebanks.
  • I4 (frames): detectors of irony or implicature, where false alarms are costly, could be tuned toward precision explicitly.
  • I5 (learn): ablations of witness parsers and training objectives could use the same full grid, so gains are attributed to the right component.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Université Mohammed VI PolytechniqueMarrakesh · Morocco18 works · 171 citations🎓University of LuxembourgLuxembourg · Luxembourg27 works · 337 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksNamed entity recognition · 200 works

Cite

Cite as Abdelkader El Mahdaouy, Salima Lamsiyah, Hamza Alami, Christoph Schommer and Ismail Berrada. 2023. UM6P & UL at WojoodNER shared task: Improving Multi-Task Learning for Flat and Nested Arabic Named Entity Recognition. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.87

@inproceedings{mahdaouy2023um6p,
  title = {{UM6P \& UL at WojoodNER shared task: Improving Multi-Task Learning for Flat and Nested Arabic Named Entity Recognition}},
  author = {Abdelkader {El Mahdaouy}, Salima Lamsiyah, Hamza Alami, Christoph Schommer and Ismail Berrada},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.87},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.87}
}

References

22 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. 2021. Pre-training bert on arabic tweets: Practical considerations. ArXiv, abs/2102.10684. ↗
  2. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics. ↗
  3. Yassine Benajiba, Paolo Rosso, and José Miguel BenedíRuiz. 2007. Anersys: An arabic named entity recognition system based on maximum entropy. In Computational Linguistics and Intelligent Text Processing, pages 143–153, Berlin, Heidelberg. Springer Berlin Heidelberg. find
  4. Kareem Darwish. 2013. Named entity recognition using cross-lingual resources: Arabic as an example. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1558–1567, Sofia, Bulgaria. Association for Computational Linguistics.
  5. AbdelRahim Elmadany, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2022. Orca: A challenging benchmark for arabic language understanding. arXiv preprint arXiv:2212.10758. ↗
  6. Jenny Rose Finkel and Christopher D. Manning. 2009. Nested named entity recognition. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 141–150, Singapore. Association for Computational Linguistics.
  7. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 92– 104, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
  8. Mustafa Jarrar, Muhammad Abdul-Mageed, Mohammed Khalilia, Bashar Talafha, AbdelRahim Elmadany, Nagham Hamad, and Alaa Omer. 2023. WojoodNER: The Arabic Named Entity Recognition Shared Task. In Proceedings of The First Arabic Natural Language Processing Conference (ArabicNLP 2023). find
  9. Mustafa Jarrar, Mohammed Khalilia, and Sana Ghanem. 2022. Wojood: Nested Arabic named entity corpus and recognition using BERT. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3626–3636, Marseille, France. European Language Resources Association.
  10. Arzoo Katiyar and Claire Cardie. 2018. Nested named entity recognition revisited. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 861–871, New Orleans, Louisiana. Association for Computational Linguistics.
  11. Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. 2022. A survey on deep learning for named entity recognition. IEEE Trans. Knowl. Data Eng., 34(1):50–70. find
  12. Xiaoya Li, Xiaofei Sun, Yuxian Meng, Junjun Liang, Fei Wu, and Jiwei Li. 2020. Dice loss for data-imbalanced NLP tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 465–476, Online. Association for Computational Linguistics.
  13. Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2020. Focal loss for dense object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(2):318–327. find
  14. Liyuan Liu, Jingbo Shang, and Jiawei Han. 2019. Arabic named entity recognition: What works and what’s next. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 60–67, Florence, Italy. Association for Computational Linguistics.
  15. Behrang Mohit, Nathan Schneider, Rishav Bhowmick, Kemal Oflazer, and Noah A. Smith. 2012. Recall-oriented learning of named entities in Arabic Wikipedia. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 162–173, Avignon, France. Association for Computational Linguistics.
  16. Xiaoye Qu, Yingjie Gu, Qingrong Xia, Zechang Li, Zhefeng Wang, and Baoxing Huai. 2023. A survey on arabic named entity recognition: Past, recent advances, and future trends. ↗
  17. Ramzi Esmail Salah and Lailatul Qadri Binti Zakaria. 2018. Building the classical arabic named entity recognition corpus (canercorpus). In 2018 Fourth International Conference on Information Retrieval and Knowledge Management (CAMP), pages 1–8. find
  18. Seyed Sadegh Mohseni Salehi, Deniz Erdogmus, and Ali Gholipour. 2017. Tversky loss function for image segmentation using 3d fully convolutional deep networks. In Machine Learning in Medical Imaging, pages 379–387, Cham. Springer International Publishing. find
  19. Khaled Shaalan. 2014. A survey of arabic named entity recognition and classification. Comput. Linguist., 40(2):469–510. ↗
  20. Yu Wang, Hanghang Tong, Ziye Zhu, and Yun Li. 2022. Nested named entity recognition: A survey. ACM Trans. Knowl. Discov. Data, 16(6). ↗
  21. Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013. Ontonotes release 5.0 LDC2013t19. Linguistic Data Consortium, Philadelphia, PA, 23:170. find
  22. Vikas Yadav and Steven Bethard. 2018. A survey on recent advances in named entity recognition from deep learning models. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2145–2158, Santa Fe, New Mexico, USA. Association for Computational Linguistics. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mahdaouy-2023-um6p-ul-wojoodner-shared-task/