Elamur Research⌕
☰
home / papers / 2023 / Do “English” Named Entity Recognizers Work Well on Global Englishes?

Do “English” Named Entity Recognizers Work Well on Global Englishes?

Paper · 2023 · Alexander Shan, John Bauer, Riley Carlson et al.

Summary

Standard English NER benchmarks such as CoNLL 2003 and OntoNotes are drawn almost entirely from British and American newswire. This leaves open whether models trained on them can handle English written elsewhere. The authors built the Worldwide English NER Dataset to test this. It contains news articles from English-language outlets in Africa, Asia, Latin America, the Middle East, and Indigenous communities in Oceania and Canada, annotated with nine entity classes. They evaluated Stanza with three embedding setups (word2vec plus character model, RoBERTa, ELECTRA), CoreNLP, Flair, and spaCy.

Models trained only on CoNLL or OntoNotes lost substantial accuracy on the global data, sometimes more than 10 F1 points. Indigenous and African texts were hardest, and Asian and Middle Eastern texts were comparatively easy. Large pretrained encoders narrowed the gap but did not close it. Training on the old and new data together kept accuracy on the original benchmark within about 1–2 F1 while greatly improving it on the global set. The work shows that "solved" English NER does not transfer evenly across varieties of English.

Key points

  • Dataset: 1075 articles and about 674,000 tokens, split 70/10/20 into train/dev/test with stratified sampling inside each region. It is released on GitHub (stanfordnlp/en-worldwide-newswire), and a fine-tuned model ships with Stanza.

  • Labels: the 18 OntoNotes 5.0 categories were reduced to nine entity types: Person, Organization, Location, Facility, Date, Money, Product, NORP, and Miscellaneous.

    • Nested spans are allowed.
    • Labels are collapsed to four CoNLL classes when comparing with CoNLL-based models.
    • Data is converted to BIOES format.

  • Annotation: performed by Aya Data, a labeling service in Ghana, through MLTwist and Datasaur, with consensus review. The reported Cohen's Kappa is 77.47.

  • Stanza with RoBERTa, by training data:

    Trained onCoNLL test F1Worldwide test F1
    CoNLL92.5483.29
    Worldwide75.6990.23
    Combined91.9990.10

  • Errors:

    • The Indigenous bucket scored lowest for CoNLL-trained models (roughly 70–77 F1).
    • Cultural and currency names were difficult in every region.
    • Unfamiliar tokens caused mistakes even when the sentence context was clear.

Interesting ideas

  • I1. Splitting the test set into regional buckets and reporting per-bucket scores exposes failures that a single averaged number hides, for example the Indigenous texts trailing the rest.
  • I2. Training on the old and new corpora together, rather than switching to the new one, keeps performance on the original domain while adding coverage. Training only on the new data hurt the old domain badly.
  • I3. Nested entity annotation lets one phrase be an event while its parts keep their own types: "2022 FIFA World Cup" is MISC, with "2022" as Date and "FIFA" as Organization.
  • I4. An explicit, documented mapping that collapses a fine label scheme into a coarser one made models with different schemes comparable and allowed combined training.
  • I5. The paper cites name-regularity bias, where a familiar name like "Obama" overrides context showing it is a town, and the authors found unknown regional tokens confuse models despite clear context.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Mova's evaluation could report results per text source or variety (fables, school tests, non-US/UK English) as separate slices, so a regression in a thin slice is not hidden by the average.
  • I2 (learn): when adding silver data from new genres, Mova could mix it with the existing training set and always re-test on the old benchmark as a guard against forgetting.
  • I3 (world): the entity layer could store such nested mentions explicitly, so an event entity links to the organization and date inside it instead of flattening them.
  • I4 (tree): the same approach fits UD dialect conversion: a written, testable collapse table between treebank conventions lets differently annotated corpora be merged and scored fairly.
  • I5 (world): entity typing could give more weight to explicit contextual cues (such as "is located in … Prefecture") than to name priors, and record which cue decided the type so the choice stays explainable.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States434 works · 314,419 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksNamed entity recognition · 200 works

Cite

Cite as Alexander Shan, John Bauer, Riley Carlson and Christopher Manning. 2023. Do “English” Named Entity Recognizers Work Well on Global Englishes?. Findings of the Association for Computational Linguistics: EMNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-emnlp.788

@inproceedings{shan2023do,
  title = {{Do “English” Named Entity Recognizers Work Well on Global Englishes?}},
  author = {Alexander Shan, John Bauer, Riley Carlson and Christopher Manning},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.788},
  url = {https://doi.org/10.18653/v1/2023.findings-emnlp.788}
}

References

25 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Adelani, Graham Neubig, Sebastian Ruder, Shruti Rijhwani, Michael Beukman, Chester Palen-Michel, Constantine Lignos, Jesujoba Alabi, Shamsuddeen Muhammad, Peter Nabende, Cheikh M. Bamba Dione, Andiswa Bukula, Rooweither Mabuya, Bonaventure F. P. Dossou, Blessing Sibanda, Happy Buzaaba, Jonathan Mukiibi, Godson Kalipe, Derguene Mbaye, Amelia Taylor, Fatoumata Kabore, Chris Chinenye Emezue, Anuoluwapo Aremu, Perez Ogayo, Catherine Gitau, Edwin Munkoh-Buabeng, Victoire Memdjokam Koagne, Allahsera Auguste Tapo, Tebogo Macucwa, Vukosi Marivate, Mboning Tchiaze Elvis, Tajuddeen Gwadabe, Tosin Adewumi, Orevaoghene Ahia, Joyce Nakatumba-Nabende, Neo Lerato Mokono, Ignatius Ezeani, Chiamaka Chukwuneke, Mofetoluwa Oluwaseun Adeyemi, Gilles Quentin Hacheme, Idris Abdulmumin, Odunayo Ogundepo, Oreen Yousuf, Tatiana Moteu, and Dietrich Klakow. 2022. MasakhaNER 2.0: Africa-centric transfer learning for named entity recognition. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4488–4508, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. ↗
  2. Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. FLAIR: An easy-to-use framework for state-of-the-art NLP. In NAACL 2019, 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 54–59.
  3. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations. find
  4. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. CoRR, abs/1911.02116. ↗
  5. Ryan Cotterell and Kevin Duh. 2017. Low-resource named entity recognition with cross-lingual, character-level neural conditional random fields. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 91–96, Taipei, Taiwan. Asian Federation of Natural Language Processing. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. ↗
  7. Maud Ehrmann, Matteo Romanello, Sven Najem-Meyer, Antoine Doucet, and Simon Clematide. 2022. Extended overview of HIPE-2022: Named Entity Recognition and Linking in Multilingual Historical Documents. In Proceedings of the Working Notes of CLEF 2022 - Conference and Labs of the Evaluation Forum, volume 3180. CEUR-WS. find
  8. Elena Epure and Romain Hennequin. 2023. A human subject study of named entity recognition (NER) in conversational music recommendation queries. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics (EACL). ↗
  9. Jenny Rose Finkel, Trond Grenager, and Christopher Manning. 2005. Incorporating non-local information into information extraction systems by Gibbs sampling. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 363–370, Ann Arbor, Michigan. Association for Computational Linguistics.
  10. Abbas Ghaddar, Philippe Langlais, Ahmad Rashid, and Mehdi Rezagholizadeh. 2021. Context-aware adversarial training for name regularity bias in named entity recognition. Transactions of the Association for Computational Linguistics, 9:586–604. ↗
  11. Filip Ginter, Jan Hajič, Juhani Luotolahti, Milan Straka, and Daniel Zeman. 2017. CoNLL 2017 shared task -automatically annotated raw texts and word embeddings. LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University. find
  12. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing.
  13. Sam Henry, Kevin Buchan, Michele Filannino, Amber Stubbs, and Ozlem Uzuner. 2019. 2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records. Journal of the American Medical Informatics Association, 27(1):3–12. ↗
  14. Matthew Honnibal, Ines Montani, Sofie Van Lan-deghem, and Adriane Boyd. 2020. spaCy: Industrial-strength Natural Language Processing in Python. find
  15. David Lim. 2023. Dslim/bert-base-ner · hugging face. find
  16. Shuheng Liu and Alan Ritter. 2023. Do CoNLL-2003 named entity taggers still work well in 2023? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8254–8271, Toronto, Canada. Association for Computational Linguistics. ↗
  17. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. ↗
  18. Anita Louis, Alta De Waal, and Cobus Venter. 2006. Named entity recognition in a South African context. In Proceedings of the 2006 Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists on IT Research in Developing Countries, SAICSIT ’06, page 170–179, ZAF. South African Institute for Computer Scientists and Information Technologists. find
  19. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In Annual Meeting of the Association for Computational Linguistics.
  20. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. ↗
  21. Stefan Schweter and Alan Akbik. 2020. FLERT: Document-level features for named entity recognition. ↗
  22. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142– 147. ↗
  23. Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, Mohammed El-Bachouti, Robert Belvin, and Ann Houston. 2013. OntoNotes Release 5.0. ↗
  24. Yuhao Zhang, Yuhui Zhang, Peng Qi, Christopher D Manning, and Curtis P Langlotz. 2021. Biomedical and clinical English model packages for the Stanza Python NLP library. Journal of the American Medical Informatics Association, 28(9):1892–1899. ↗
  25. Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, and Diyi Yang. 2023. Multi-value: A framework for cross-dialectal English NLP.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shan-2023-do-english-named-entity-recognizers/