Elamur Research⌕
☰
home / papers / 2021 / The Utility and Interplay of Gazetteers and Entity Segmentation for Named Entity Recognition in English

The Utility and Interplay of Gazetteers and Entity Segmentation for Named Entity Recognition in English

Paper · 2021 · Oshin Agarwal and Ani Nenkova

Summary

Gazetteers (lists of typed entity names) and entity segmentation (finding entity spans without types) have each been proposed to help neural NER. Earlier papers used different resources and mixed in unrelated features, so their results could not be compared. The authors survey both families. They re-implement representative methods inside one biLSTM-CRF setup, using GloVe+char or ELMo+GloVe+char inputs. They evaluate on four English datasets covering newswire, conversation and Twitter: CoNLL, OntoNotes, BTC and TTC. The gazetteer is the Wikipedia/Web-derived resource of Ratinov and Roth (2009).

Gazetteer-based models improve results consistently across datasets. Segmentation methods sometimes give large gains on one corpus but hurt others. In each family, the simplest variant wins: binary word-level gazetteer match features, and multi-task learning with an auxiliary span/no-span classifier. Contrary to expectation, gazetteers mostly help by marking where entities are, not by supplying their types. They also beat dedicated segmentation models on long entities. This gives practitioners a cheap, stable way to add entity knowledge.

Key points

  • The gazetteer has about 3M entries in about 30 fine-grained categories. About 24% of entries have four or more words. In the NER datasets, such long mentions are only about 2% of entities (about 10% in OntoNotes).
  • Mapping gazetteer categories to dataset labels is unnecessary, because the original gazetteer types work equally well. With GloVe+char, dropping type information entirely gives better results. With ELMo, the typeless version still beats the baseline but not typed WORD GAZ.
  • Segmentation F1 (spans without types) rises with gazetteers by margins similar to full NER F1. This shows gazetteers induce segmentation.
  • By entity length (ELMo+GloVe+char), WORD GAZ beats HSCRF on mentions of 4+ words in CoNLL, OntoNotes and BTC (BTC: 44.6 vs 36.9). HSCRF is better on short mentions and on TTC long ones.
  • Gazetteer features help less on CoNLL and OntoNotes. In these datasets, most matched words fall into two or more gazetteer categories.

Interesting ideas

  • I1. Breaking down evaluation by mention length and by span-only versus typed F1 shows which sub-problem a resource fixes, not just that the score went up.
  • I2. A plain "this token is inside some known name" flag, with no type, can carry most of a dictionary's benefit, because it marks boundaries.
  • I3. Large name lists are rich in long multi-word entries. They can substitute for dedicated span-structure models when finding long mentions.
  • I4. When a dictionary entry belongs to several categories, its type signal becomes ambiguous and its benefit shrinks. Measuring per-word category ambiguity predicts how useful the list will be.
  • I5. Updating a name list is suggested as a cheaper alternative to re-annotating data when vocabulary drifts quickly, as on Twitter.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Report span-detection and typing accuracy separately, and bucket results by phrase length. This would show whether a new lexicon or rule improves boundaries or labels.
  • I2 (tokens): Add an untyped multi-word-name match feature to the tagger. This could keep proper-noun spans like long organisation names together before parsing.
  • I3 (tree): Use Wikidata/WordNet multi-word names to guide flat/compound attachment of long names in the parser, as an explainable boundary hint.
  • I4 (world): Store the ambiguity count with each knowledge-graph name. Entity typing in the world model can then down-weight highly ambiguous matches.
  • I5 (world): Keep entity lexicons refreshable from Wikidata independently of trained models. New names in tales or tests can then be recognised without retraining.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of PennsylvaniaPhiladelphia · United States221 works · 63,234 citations🏢Adobe SystemsSan Jose · United States35 works · 3,711 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksNamed entity recognition · 200 works

Cite

Cite as Oshin Agarwal and Ani Nenkova. 2021. The Utility and Interplay of Gazetteers and Entity Segmentation for Named Entity Recognition in English. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.findings-acl.349

@inproceedings{agarwal2021utility,
  title = {{The Utility and Interplay of Gazetteers and Entity Segmentation for Named Entity Recognition in English}},
  author = {Oshin Agarwal and Ani Nenkova},
  booktitle = {Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021},
  year = {2021},
  doi = {10.18653/v1/2021.findings-acl.349},
  url = {https://doi.org/10.18653/v1/2021.findings-acl.349}
}

References

60 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Oshin Agarwal, Yinfei Yang, Byron C. Wallace, and A. Nenkova. 2020. Entity-switched datasets: An approach to auditing the in-domain robustness of named entity recognition models. ArXiv, abs/2004.04123. ↗
  2. Oshin Agarwal, Yinfei Yang, Byron C. Wallace, and Ani Nenkova. 2021. Interpretability Analysis for Named Entity Recognition to Understand System Predictions and How They Can Improve. Computational Linguistics, 47(1):117–140. ↗
  3. Gustavo Aguilar, Adrian Pastor López-Monroy, Fabio González, and Thamar Solorio. 2018. Modeling noisiness to recognize named entities using multi-task neural networks on social media. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1401–1412, New Orleans, Louisiana. Association for Computational Linguistics. find
  4. Gustavo Aguilar, Suraj Maharjan, Adrian Pastor López-Monroy, and Thamar Solorio. 2017. A multi-task approach for named entity recognition in social media data. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 148–153, Copenhagen, Denmark. Association for Computational Linguistics. find
  5. Isabelle Augenstein, Mrinal Das, Sebastian Riedel, Lakshmi Vikraman, and Andrew McCallum. 2017a. SemEval 2017 task 10: ScienceIE - extracting keyphrases and relations from scientific publications. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 546–555, Vancouver, Canada. Association for Computational Linguistics.
  6. Isabelle Augenstein, Leon Derczynski, and Kalina Bontcheva. 2017b. Generalisation in named entity recognition: A quantitative analysis. ArXiv, abs/1701.02877. ↗
  7. Isabelle Augenstein and Anders Søgaard. 2017. Multi-task learning of keyphrase boundary classification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 341–346, Vancouver, Canada. Association for Computational Linguistics. ↗
  8. Colin Cherry and Hongyu Guo. 2015. The unreasonable effectiveness of word representations for Twitter named entity recognition. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 735–745, Denver, Colorado. Association for Computational Linguistics. find
  9. Jason P.C. Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional LSTM-CNNs. Transactions of the Association for Computational Linguistics, 4:357–370. ↗
  10. Michael Collins and Yoram Singer. 1999. Unsupervised models for named entity classification. In 1999 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora.
  11. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of machine learning research, 12(ARTICLE):2493–2537.
  12. Xiang Dai, Sarvnaz Karimi, Ben Hachey, and Cecile Paris. 2019. Using similarity measures to select pre-training data for NER. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1460–1470, Minneapolis, Minnesota. Association for Computational Linguistics. find
  13. Leon Derczynski, Kalina Bontcheva, and Ian Roberts. 2016. Broad Twitter corpus: A diverse named entity recognition resource. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1169– 1179, Osaka, Japan. find
  14. The COLING 2016 Organizing Committee. find
  15. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  16. Oliver Bender, Franz Josef Och, and Hermann Ney. 2003. Maximum entropy models for named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 148–151. find
  17. Rezarta Islamaj Doğan, Robert Leaman, and Zhiyong Lu. 2014. Ncbi disease corpus: a resource for disease name recognition and concept normalization. Journal of biomedical informatics, 47:1–10. ↗
  18. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2014. One billion word benchmark for measuring progress in statistical language modeling. In Fifteenth Annual Conference of the International Speech Communication Association.
  19. Doug Downey, Matthew Broadhead, and Oren Etzioni. 2007. Locating complex named entities in web text. In Proceedings of the 20th International Joint Conference on Artifical Intelligence, IJCAI’07, page 2733–2739, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. find
  20. Jacob Eisenstein. 2013. What to do about bad language on the internet. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 359–369, Atlanta, Georgia. Association for Computational Linguistics.
  21. Martin Krallinger, Obdulia Rabal, Florian Leitner, Miguel Vazquez, David Salgado, Zhiyong Lu, Robert Leaman, Yanan Lu, Donghong Ji, Daniel M Lowe, et al. 2015. The chemdner corpus of chemicals and drugs and its annotation principles. Journal of cheminformatics, 7(1):1–17. ↗
  22. Jenny Rose Finkel, Trond Grenager, and Christopher Manning. 2005. Incorporating non-local information into information extraction systems by Gibbs sampling. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 363–370, Ann Arbor, Michigan. Association for Computational Linguistics.
  23. John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. find
  24. JinLan Fu, Peng fei Liu, Qi Zhang, and Xuanjing Huang. 2020a. Rethinking generalization of neural models: A named entity recognition case study. ArXiv, abs/2001.03844. ↗
  25. Jinlan Fu, Pengfei Liu, and Graham Neubig. 2020b. Interpretable multi-dataset evaluation for named entity recognition. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6058–6069, Online. Association for Computational Linguistics. find
  26. Octavian-Eugen Ganea and Thomas Hofmann. 2017. Deep joint entity disambiguation with local neural attention. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2619–2629, Copenhagen, Denmark. Association for Computational Linguistics. ↗
  27. Abbas Ghaddar and Phillippe Langlais. 2018. Robust lexical features for improved neural network named-entity recognition. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1896–1907, Santa Fe, New Mexico, USA. Association for Computational Linguistics. ↗
  28. Ralph Grishman and Beth Sundheim. 1996. Message Understanding Conference- 6: A brief history. In COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics, pages 466–471.
  29. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. ↗
  30. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. OntoNotes: The 90% solution. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers, pages 57–60, New York City, USA. Association for Computational Linguistics. ↗
  31. Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991. ↗
  32. J-D Kim, Tomoko Ohta, Yuka Tateisi, and Jun’ichi Tsujii. 2003. Genia corpus—a semantically annotated corpus for bio-textmining. Bioinformatics, 19(suppl 1):i180–i182. ↗
  33. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 260–270, San Diego, California. Association for Computational Linguistics. ↗
  34. Tianyu Liu, Jin-Ge Yao, and Chin-Yew Lin. 2019a. Towards improving neural named entity recognition with gazetteers. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5301–5307, Florence, Italy. Association for Computational Linguistics. ↗
  35. Yijin Liu, Fandong Meng, Jinchao Zhang, Jinan Xu, Yufeng Chen, and Jie Zhou. 2019b. GCDT: A global context enhanced deep transition architecture for sequence labeling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2431–2441, Florence, Italy. Association for Computational Linguistics. find
  36. Teng Long, Ryan Lowe, Jackie Chi Kit Cheung, and Doina Precup. 2016. Leveraging lexical resources for learning entity embeddings in multi-relational data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 112–117, Berlin, Germany. Association for Computational Linguistics. find
  37. Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1064–1074, Berlin, Germany. Association for Computational Linguistics. ↗
  38. Simone Magnolini, Valerio Piccioni, Vevake Balaraman, Marco Guerini, and Bernardo Magnini. 2019. How to use gazetteers for entity recognition with neural models. In Proceedings of the 5th Workshop on Semantic Deep Learning (SemDeep-5), pages 40– 49, Macau, China. Association for Computational Linguistics. find
  39. Xue Mengge, Bowen Yu, Zhenyu Zhang, Tingwen Liu, Yue Zhang, and Bin Wang. 2020. Coarse-to-Fine Pre-training for Named Entity Recognition. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6345–6354, Online. Association for Computational Linguistics. ↗
  40. Andrei Mikheev, Marc Moens, and Claire Grover. 1999. Named entity recognition without gazetteers. In Ninth Conference of the European Chapter of the Association for Computational Linguistics, pages 1– 8, Bergen, Norway. Association for Computational Linguistics.
  41. Alan Ritter, Sam Clark, Mausam, and Oren Etzioni. 2011. Named entity recognition in tweets: An experimental study. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1524–1534, Edinburgh, Scotland, UK. Association for Computational Linguistics.
  42. Scott Miller, Jethran Guinness, and Alex Zamanian. 2004. Name tagging with word clusters and discriminative training. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 337– 342, Boston, Massachusetts, USA. Association for Computational Linguistics.
  43. Sunita Sarawagi and William W Cohen. 2004. Semi-markov conditional random fields for information extraction. Advances in neural information processing systems, 17:1185–1192. find
  44. Einat Minkov, Richard C. Wang, and William W. Cohen. 2005. Extracting personal names from email: Applying named entity recognition to informal text. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 443–450, Vancouver, British Columbia, Canada. Association for Computational Linguistics. find
  45. Naoaki Okazaki. 2007. Crfsuite: a fast implementation of conditional random fields (crfs). find
  46. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. ↗
  47. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  48. Duangmanee Putthividhya and Junling Hu. 2011. Bootstrapped named entity recognition for product attribute extraction. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1557–1567, Edinburgh, Scotland, UK. Association for Computational Linguistics. find
  49. Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009), pages 147–155, Boulder, Colorado. Association for Computational Linguistics.
  50. Shruti Rijhwani and Daniel Preotiuc-Pietro. 2020. Temporally-informed analysis of named entity recognition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7605–7617, Online. Association for Computational Linguistics. find
  51. Motoki Sato, Hiroyuki Shindo, Ikuya Yamada, and Yuji Matsumoto. 2017. Segment-level neural conditional random fields for named entity recognition. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 97–102, Taipei, Taiwan. Asian Federation of Natural Language Processing. find
  52. Dominic Seyler, Tatiana Dembelova, Luciano Del Corro, Johannes Hoffart, and Gerhard Weikum. 2018. A study of the importance of external knowledge in the named entity recognition task. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 241–246, Melbourne, Australia. Association for Computational Linguistics. find
  53. Chan Hee Song, Dawn Lawrie, Tim Finin, and James Mayfield. 2020. Improving neural named entity recognition with gazetteers. arXiv preprint arXiv:2003.03072. ↗
  54. Karl Stratos. 2017. Entity identification as multitasking. In Proceedings of the 2nd Workshop on Structured Prediction for Natural Language Processing, pages 7–11, Copenhagen, Denmark. Association for Computational Linguistics. find
  55. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142–147. ↗
  56. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30:5998–6008.
  57. Shiyuan Xiao, Yuanxin Ouyang, Wenge Rong, Jianxin Yang, and Zhang Xiong. 2019. Similarity based auxiliary classifier for named entity recognition. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1140– 1149, Hong Kong, China. Association for Computational Linguistics. • MONEY: Currency Eunsuk Yang, Young-Bum Kim, Ruhi Sarikaya, and Yu-Seop Kim. 2016. Drop-out conditional random fields for Twitter with huge mined gazetteer. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 282–288, San Diego, California. Association for Computational Linguistics. find
  58. Zhixiu Ye and Zhen-Hua Ling. 2018. Hybrid semi-Markov CRF for neural sequence labeling. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 235–240, Melbourne, Australia. Association for Computational Linguistics. find
  59. Xiaodong Yu, Stephen Mayhew, Mark Sammons, and Dan Roth. 2018. On the strength of character language models for multilingual named entity recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3073–3077, Brussels, Belgium. Association for Computational Linguistics. find
  60. Jingwei Zhuo, Yong Cao, Jun Zhu, Bo Zhang, and Zaiqing Nie. 2016. Segment-level sequence modeling using gated recursive semi-Markov conditional random fields. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1413–1423, Berlin, Germany. Association for Computational Linguistics. • NORP: Nationalities, EthnicGroups • QUANTITY: Measurements • WORK OF ART: TV.Programs, ArtWork, Films BTC and TTC • PER: People • LOC: Locations, Locations.Generic, Parks • ORG: Organization, PoliticalParties, Corporations, Government find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/agarwal-2021-utility-interplay-gazetteers-entity-segmentation/