Elamur Research⌕
☰
home / papers / 2023 / Lotus at WojoodNER Shared Task: Multilingual Transformers: Unveiling Flat and Nested Entity Recognition

Lotus at WojoodNER Shared Task: Multilingual Transformers: Unveiling Flat and Nested Entity Recognition

Paper · 2023 · Jiyong Li, Dilshod Azizov, Hilal AlQuabeh et al.

Summary

The paper describes the Lotus team's entry in the WojoodNER shared task at ArabicNLP 2023. The task covers Arabic named entity recognition in two settings. The flat setting gives each token one label. The nested setting lets a token carry several overlapping entity types. The data is the Wojood corpus, about 550k tokens of MSA and dialectal Arabic annotated with 21 entity types. Most tokens in it are not entities.

The authors fine-tuned the multilingual XLM-R encoder for both subtasks and changed only the output layer:

  • Flat NER: one softmax classifier over all BIO labels.
  • Nested NER: 21 separate heads, one per entity type, each choosing B, I or O for every token.

Before settling on XLM-R, they tried it against the Arabic-specific AraBERT and MARBERT, and XLM-R came out best on their development split. On the official leaderboard the system reached micro-F1 0.83 (flat) and 0.76 (nested). The work shows that a multilingual encoder can match or beat Arabic-specific encoders here. It also shows that splitting nested NER into one tagger per type is a cheap way to handle overlapping spans.

Key points

  • Data: the Wojood corpus has about 550k tokens and 21 entity types. Roughly 60% of tokens are non-entities. The most frequent entity types are Date, PERS and GPE.

  • Training setup:

    SettingSubtask 1Subtask 2
    Sequence length256same as Subtask 1
    Batch size816
    Learning rate2e-51e-5
    Epochs5same as Subtask 1

    Both used AdamW, weight decay 0.01, gradient clipping at 1.0 and no warm-up, on 2 Tesla T4 GPUs.

  • Development results (10% held out from train):

    • Subtask 1: XLM-R micro-F1 0.829, AraBERT 0.713. MARBERT was lowest, reported as 0.563 in the text and 0.663 in Table 1.
    • Subtask 2: XLM-R 0.879, MARBERT 0.870, AraBERT 0.848. MARBERT had the best recall (0.884).

  • Leaderboard results: 0.83 micro-F1 on flat NER and 0.76 on nested NER. The nested score is far below the 0.879 seen on development.

  • Planned future work: data augmentation (sentence mixing, back-translation) and a meta-ensemble of AraBERT, MARBERT, XLM-R and AR-BERT.

Interesting ideas

  • I1. Nested NER is split into one independent B/I/O tagger per entity type, so overlapping spans of different types never compete for a single label slot.
  • I2. Only the first subword piece of each word carries a training label, and continuation pieces and padding are masked out of the loss. This keeps predictions aligned with whole words despite subword tokenization.
  • I3. The authors argue that treating B-XX and I-XX as ordinary separate classes is a weak fit for nested structure. How the output layer is shaped mattered more than which encoder was used.
  • I4. The nested score fell from 0.879 on a self-cut development split to 0.76 on the official test, which shows that a random split of the training data can badly overestimate quality.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): entity mentions in a story can nest (a place inside an organization name), and per-type span heads would let the world layer keep both entities instead of forcing one choice.
  • I2 (tree): gparse can predict heads and UD relations only from the first piece of each UD word, so tree output stays word-aligned with the treebank tokenization.
  • I3 (tokens): when the tagger must produce layered or multi-valued features (e.g. several morphological attributes), separate small heads per attribute may work better than one flat combined label set.
  • I4 (learn): evaluation should include held-out data from a different source than training, reported next to in-distribution scores, as a guard against inflated numbers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Mohamed bin Zayed University of Artificial IntelligenceAbu Dhabi · United Arab Emirates124 works · 5,676 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 3Research subjects and languages the work is about.Semitic languages · 2,908 worksMultilingual & cross-lingual NLP · 828 worksNamed entity recognition · 200 works

Cite

Cite as Jiyong Li, Dilshod Azizov, Hilal AlQuabeh and Shangsong Liang. 2023. Lotus at WojoodNER Shared Task: Multilingual Transformers: Unveiling Flat and Nested Entity Recognition. Proceedings of ArabicNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.arabicnlp-1.85

@inproceedings{li2023lotus,
  title = {{Lotus at WojoodNER Shared Task: Multilingual Transformers: Unveiling Flat and Nested Entity Recognition}},
  author = {Jiyong Li, Dilshod Azizov, Hilal AlQuabeh and Shangsong Liang},
  booktitle = {Proceedings of ArabicNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.arabicnlp-1.85},
  url = {https://doi.org/10.18653/v1/2023.arabicnlp-1.85}
}

References

29 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2020. Arbert & marbert: deep bidirectional transformers for arabic. arXiv preprint arXiv:2101.01785. ↗
  2. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104. ↗
  3. Yassine Benajiba and Paolo Rosso. 2008. Arabic named entity recognition using conditional random fields. In Proc. of Workshop on HLT & NLP within the Arabic World, LREC, volume 8, pages 143–153. find
  4. Yassine Benajiba, Paolo Rosso, and José Miguel Benedíruiz. 2007. Anersys: An arabic named entity recognition system based on maximum entropy. In Computational Linguistics and Intelligent Text Processing: 8th International Conference, CICLing 2007, Mexico City, Mexico, February 18-24, 2007. Proceedings 8, pages 143–153. Springer. find
  5. Qiao Cheng, Juntao Liu, Xiaoye Qu, Jin Zhao, Jiaqing Liang, Zhefeng Wang, Baoxing Huai, Nicholas Jing Yuan, and Yanghua Xiao. 2021. Hacred: A large-scale relation extraction dataset toward hard cases in practical applications. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2819–2831. find
  6. Kevin Clark and Christopher D Manning. 2016. Improving coreference resolution by learning entity-level distributed representations. arXiv preprint arXiv:1606.01323. ↗
  7. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. ↗
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  9. Alaa Elgibali. 2005. Investigating Arabic: Current parameters in analysis and learning, volume 42. Brill. find
  10. Yingjie Gu, Xiaoye Qu, Zhefeng Wang, Baoxing Huai, Nicholas Jing Yuan, and Xiaolin Gui. 2021. Read, retrospect, select: An mrc framework to short text entity linking. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 12920– 12928. find
  11. Imane Guellil, Houda Saâdane, Faical Azouaou, Billel Gueni, and Damien Nouvel. 2021. Arabic natural language processing: An overview. Journal of King Saud University-Computer and Information Sciences, 33(5):497–507.
  12. Arzoo Katiyar and Claire Cardie. 2018. Nested named entity recognition revisited. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, volume 1.
  13. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. arXiv preprint arXiv:1603.01360. ↗
  14. Xiaoya Li, Jingrong Feng, Yuxian Meng, Qinghong Han, Fei Wu, and Jiwei Li. 2019. A unified mrc framework for named entity recognition. arXiv preprint arXiv:1910.11476. ↗
  15. Behrang Mohit, Nathan Schneider, Rishav Bhowmick, Kemal Oflazer, and Noah A Smith. 2012. Recall-oriented learning of named entities in arabic wikipedia. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 162–173.
  16. Xiaoye Qu, Yingjie Gu, Qingrong Xia, Zechang Li, Zhefeng Wang, and Baoxing Huai. 2023. A survey on arabic named entity recognition: Past, recent advances, and future trends. arXiv preprint arXiv:2302.03512. ↗
  17. Ramzi Esmail Salah and Lailatul Qadri Binti Zakaria. 2018. Building the classical arabic named entity recognition corpus (canercorpus). In 2018 Fourth International Conference on Information Retrieval and Knowledge Management (CAMP), pages 1–8. IEEE. find
  18. Khaled Shaalan and Hafsa Raza. 2009. Nera: Named entity recognition for arabic. Journal of the American Society for Information Science and Technology, 60(8):1652–1663. ↗
  19. Mustafa Jarrar, Muhammad Abdul-Mageed, Mohammed Khalilia, Bashar Talafha, AbdelRahim Elmadany, Nagham Hamad, and Alaa Omer. 2023. WojoodNER: The Arabic Named Entity Recognition Shared Task. In Proceedings of The First Arabic Natural Language Processing Conference (ArabicNLP 2023). find
  20. Mustafa Jarrar, Mohammed Khalilia, and Sana Ghanem. 2022a. Named entity recognition, multi-task learning, nested entities, bert, arabic ner corpus. arXiv preprint arXiv:2205.09651. ↗
  21. Mustafa Jarrar, Mohammed Khalilia, and Sana Ghanem. 2022b. Wojood: Nested arabic named entity corpus and recognition using bert. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3626–3636.
  22. Meizhi Ju, Makoto Miwa, and Sophia Ananiadou. 2018. A neural layered model for nested named entity recognition. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1446–1459.
  23. Jana Straková, Milan Straka, and Jan Hajič. 2019. Neural architectures for nested ner through linearization. arXiv preprint arXiv:1908.06926. ↗
  24. Arata Ugawa, Akihiro Tamura, Takashi Ninomiya, Hiroya Takamura, and Manabu Okumura. 2018. Neural machine translation incorporating named entity. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3240–3250. find
  25. Yu Wang, Hanghang Tong, Ziye Zhu, and Yun Li. 2022. Nested named entity recognition: a survey. ACM Transactions on Knowledge Discovery from Data (TKDD), 16(6):1–29. ↗
  26. Yuesong Wang, Tao Guan, Zhuo Chen, Yawei Luo, Keyang Luo, and Lili Ju. 2020. Mesh-guided multi-view stereo with pyramid architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2039–2048. find
  27. Ralph Weischedel, Sameer Pradhan, Lance Ramshaw, Martha Palmer, Nianwen Xue, Mitchell Marcus, Ann Taylor, Craig Greenberg, Eduard Hovy, Robert Belvin, et al. 2011. Ontonotes release 4.0. LDC2011T03, Philadelphia, Penn.: Linguistic Data Consortium. ↗
  28. Tong Zhu, Xiaoye Qu, Wenliang Chen, Zhefeng Wang, Baoxing Huai, Nicholas Jing Yuan, and Min Zhang. 2021. Efficient document-level event extraction via pseudo-trigger-aware pruned complete graph. arXiv preprint arXiv:2112.06013. ↗
  29. Ayah Zirikly and Mona Diab. 2015. Named entity recognition for arabic social media. In Proceedings of the 1st workshop on vector space modeling for natural language processing, pages 176–185.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2023-lotus-wojoodner-shared-task-multilingual/