Elamur Research⌕
☰
home / papers / 2022 / MultiCoNER: A Large-scale Multilingual Dataset for Complex Named Entity Recognition

MultiCoNER: A Large-scale Multilingual Dataset for Complex Named Entity Recognition

Summary

Standard NER benchmarks such as CoNLL03 reward models for things real inputs often lack: clean casing and punctuation, long context, mostly simple names, and heavy overlap of entities between train and test. MultiCoNER is a benchmark built to test these harder conditions. It covers 11 languages, plus a multilingual and a code-mixed subset, across three domains: low-context Wikipedia sentences (LOWNER), questions (MSQ-NER) and search queries (ORCAS-NER). The roughly 26M-token corpus was assembled from public sources such as Wikipedia and Wikidata. The authors used heuristic sentence selection, template extraction with entity slotting, and machine translation. The text is lowercased and short, and it contains syntactically complex entities like creative-work titles. Two systems were evaluated. A fine-tuned XLM-RoBERTa baseline reaches only about 54% average macro-F1. GEMNET, which adds Wikidata gazetteers, raises this by about 30 points. The dataset served as the basis of SemEval 2022 Task 11. It shows that large pretrained models still struggle without external entity knowledge, which makes it a useful testbed for knowledge-infused NER.

Key points

  • Scope: 11 languages (BN, ZH, NL, EN, HI, KO, RU, ES, DE, TR, FA) organized into 13 subsets: 11 monolingual, MULTI and MIX. The dataset was used in SemEval 2022 Task 11, which drew more than 236 participants.
  • Tag set: six classes taken from the WNUT 2017 taxonomy: PER, LOC, CORP, GRP, PROD and CW.
  • Sources:
    • MSQ-NER is built from MS-MARCO QnA (V2.1).
    • ORCAS-NER is built from 10 million Bing queries in the ORCAS dataset.
  • Training split:
    • Each language gets 15,300 sentences, a size comparable to CoNLL03.
    • Almost all of it comes from LOWNER, with only 100 instances from the question and query domains.
  • Results (average F1, baseline → GEMNET):
    • Macro-F1 rises from 0.542 to 0.846, and micro-F1 from 0.552 to 0.846.
    • CW is the hardest class (0.430 → 0.779).
    • The largest micro-F1 gains are for TR (+41.5 points) and KO (+33 points).
    • The weakest baseline language is BN (micro-F1 0.397).

Interesting ideas

  • I1. Keep out-of-domain training data deliberately small. Only a token amount of question and query data goes into training, so test scores on those domains reflect true adaptation rather than in-domain fitting. The same split design could serve any benchmark that targets domain shift.
  • I2. Synthesize code-mixing by swapping entities. Code-mixed examples are made by replacing an entity's surface form with its equivalent in another language, using cross-lingual entity links. This is a cheap way to create mixed-language test data, for example Ukrainian–Russian or Ukrainian–English, without new annotation.
  • I3. Lowercase and shorten the input on purpose. Removing casing and context forces models to rely on knowledge rather than surface cues. This makes evaluation closer to ASR transcripts or search queries.
  • I4. Gate between context and gazetteer signals. GEMNET uses a mixture-of-experts gate to weigh a contextual token representation against a gazetteer match. The model can then trust or ignore noisy lexicon hits, such as a name that matches both CORP and LOC. The pattern could transfer to any task that pairs a neural model with an imperfect dictionary.
  • I5. Report mention detection separately from typed F1. Scoring boundary detection without the entity class shows whether errors come from finding spans or from labeling them. This is a useful diagnostic for complex entities like titles.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Named entity recognition

Cited by

98 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 94 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{malmasi2022multiconer,
  title = {{MultiCoNER: A Large-scale Multilingual Dataset for Complex Named Entity Recognition}},
  author = {Malmasi, Shervin and Fang, Anjie and Fetahu, Besnik and Kar, Sudipta and Rokhlenko, Oleg},
  booktitle = {Proceedings of the 29th International Conference on Computational Linguistics},
  year = {2022},
  url = {https://aclanthology.org/2022.coling-1.334/}
}

References

17 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gustavo Aguilar, Suraj Maharjan, Adrian Pastor López-Monroy, and Thamar Solorio. 2017. A multi-task approach for named entity recognition in social media data. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 148–153. find
  2. Isabelle Augenstein, Leon Derczynski, and Kalina Bontcheva. 2017. Generalisation in named entity recognition: A quantitative analysis. Computer Speech & Language, 44:61–83. find
  3. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268. find our card
  4. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  5. Nick Craswell, Daniel Campos, Bhaskar Mitra, Emine Yilmaz, and Bodo Billerbeck. 2020. Orcas: 18 million clicked query-document pairs for analyzing search. arXiv preprint arXiv:2006.05324. — [arXiv:2006.05324](https://arxiv.org/abs/2006.05324) find
  6. Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. Results of the wnut2017 shared task on novel and emerging entity recognition. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 140–147. find our card
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (1). Association for Computational Linguistics. find
  8. Besnik Fetahu, Anjie Fang, Oleg Rokhlenko, and Shervin Malmasi. 2021. Gazetteer enhanced named entity recognition for code-mixed web queries. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1677–1681. find
  9. Besnik Fetahu, Anjie Fang, Oleg Rokhlenko, and Shervin Malmasi. 2022. Dynamic gazetteer integration in multilingual models for cross-lingual and cross-domain named entity recognition. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics… find
  10. Daniel Khashabi, Mark Sammons, Ben Zhou, Tom Redman, Christos Christodoulopoulos, Vivek Srikumar, Nicholas Rizzolo, Lev-Arie Ratinov, Guanheng Luo, Quang Do, Chen-Tse Tsai, Subhro Roy, Stephen Mayhew, Zhili Feng, John Wieting, Xiaodong Yu, Yangqiu Song, Shashank Gupta, Shyam Upadhyay, Naveen Arivazh… find
  11. Shervin Malmasi, Anjie Fang, Besnik Fetahu, Sudipta Kar, and Oleg Rokhlenko. 2022. Semeval-2022 task 11: Multilingual complex named entity recognition (multiconer). In Proceedings of the 16th International Workshop on Semantic Evaluation, SemEval@NAACL 2022, Seattle, Washington, United States, July… find our card
  12. Stephen Mayhew, Tatiana Tsygankova, and Dan Roth. 2019. ner and pos when nothing is capitalized. In EMNLP/IJCNLP (1), pages 6255–6260. Association for Computational Linguistics. find
  13. Tao Meng, Anjie Fang, Oleg Rokhlenko, and Shervin Malmasi. 2021. GEMNET: effective gated gazetteer representations for recognizing complex entities in low-context input. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua… find
  14. Erik Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142–147. find
  15. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2… find
  16. Marlies van der Wees, Arianna Bisazza, Wouter Weerkamp, and Christof Monz. 2015. What’s in a domain? analyzing genre and topic differences in statistical machine translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Co… find
  17. Tongshuang Wu, Kanit Wongsuphasawat, Donghao Ren, Kayur Patel, and Chris DuBois. 2020. Tempura: Query analysis with structural templates. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1–12. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/malmasi-2022-multiconer-large-scale-multilingual-dataset/