Elamur Research
☰
home / papers / 2009 / Design Challenges and Misconceptions in Named Entity Recognition

Design Challenges and Misconceptions in Named Entity Recognition

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/W09-1119.pdf

Summary

The paper looks at how to build an accurate named entity recognizer that also holds up on new domains. It compares four core design choices in NER systems: how to encode text chunks, which decoding algorithm to use, how to capture dependencies beyond a local window, and how to bring in outside knowledge. The base learner is a regularized averaged perceptron specified in LBJ. All models are trained on CoNLL-2003 and tested without retraining on CoNLL03, MUC7, and a newly annotated Webpages set.

The authors report several findings that go against common practice. BILOU chunk encoding beats the popular BIO scheme. Simple greedy left-to-right decoding is about as accurate as beamsearch or Viterbi and is far cheaper to run. No single published method for non-local features wins on every dataset, but the methods add up when combined. Word class models induced from unlabeled text and Wikipedia-derived gazetteers each give independent gains.

The resulting public system, LBJ-NER, reaches 90.8 F1 on CoNLL03, the best published result for that dataset at the time. It also outperforms the Stanford NER tagger on all three test sets.

Key points

  • Encoding (end system): BILOU beats BIO on CoNLL03 test (90.57 vs 89.15) and on both MUC7 splits. The only exception is CoNLL03 dev, where BIO scores slightly higher (93.61 vs 93.28).
  • Decoding: Greedy, beamsearch and Viterbi score almost the same.
    • With baseline features: 83.29 (greedy) vs 83.71 (Viterbi).
    • Greedy is over 100× faster than Viterbi, since BILOU over four entity types yields 21 tag states.
    • Viterbi cannot be used in the final system because it is incompatible with non-local features.
  • Knowledge sources:
    • Wikipedia gazetteers raise the system's CoNLL03 test score to 87.22.
    • Adding word class models and non-local features brings it to 90.57.
    • Training on train+dev data gives the final 90.8.
  • Webpages dataset: 20 manually annotated pages (personal, academic, conference homepages) with 783 entities. Because entity types were often ambiguous, it is scored by token-level entity identification only.
  • Gazetteers: Built from Wikipedia categories covering people, organizations, locations, named objects, art works, films, songs and events. The people category alone yields about 495K titles. The system and the Webpages data are released for research use.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Named entity recognition

Cited by

386 papers of our library cite it (of the 140673 whose bibliographies we read); 15 of them have a page here, 371 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{ratinov2009design,
  title = {{Design Challenges and Misconceptions in Named Entity Recognition}},
  author = {Ratinov, Lev and Roth, Dan},
  booktitle = {Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009)},
  year = {2009},
  url = {https://aclanthology.org/W09-1119/}
}

References

32 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. K. Ando and T. Zhang. 2005. A high-performance semi-supervised learning method for text chunking. In ACL. find
  2. P. F. Brown, P. V. deSouza, R. L. Mercer, V. J. D. Pietra, and J. C. Lai. 1992. Class-based n-gram models of natural language. Computational Linguistics, 18(4):467–479. find
  3. X. Carreras, L. Màrquez, and L. Padró. 2003. Learning a perceptron-based named entity chunker via online recognition feedback. In CoNLL. find
  4. H. Chieu and H. T. Ng. 2003. Named entity recognition with a maximum entropy approach. In Proceedings of CoNLL. find
  5. W. W. Cohen. 2004. Exploiting dictionaries in named entity extraction: Combining semi-markov extraction processes and data integration methods. In KDD. find
  6. M. Collins. 2002. Discriminative training methods for hidden markov models: Theory and experiments with perceptron algorithms. In EMNLP. find our card
  7. L. Edward. 2007. Finding good sequential model structures using output transformations. In EMNLP). find
  8. O. Etzioni, M. J. Cafarella, D. Downey, A. Popescu, T. Shaked, S. Soderland, D. S. Weld, and A. Yates. 2005. Unsupervised named-entity extraction from the web: An experimental study. Artificial Intelligence, 165(1):91–134. find
  9. J. R. Finkel, T. Grenager, and C. D. Manning. 2005. Incorporating non-local information into information extraction systems by gibbs sampling. In ACL. find
  10. R. Florian, A. Ittycheriah, H. Jing, and T. Zhang. 2003. Named entity recognition through classifier combination. In CoNLL. find
  11. Y. Freund and R. Schapire. 1999. Large margin classification using the perceptron algorithm. Machine Learning, 37(3):277–296. find
  12. J. Kazama and K. Torisawa. 2007a. Exploiting wikipedia as external knowledge for named entity recognition. In EMNLP. find
  13. J. Kazama and K. Torisawa. 2007b. A new perceptron algorithm for sequence labeling with non-local features. In EMNLP-CoNLL. find
  14. T. Koo, X. Carreras, and M. Collins. 2008. Simple semi-supervised dependency parsing. In ACL. find our card
  15. V. Krishnan and C. D. Manning. 2006. An effective two-stage model for exploiting non-local dependencies in named entity recognition. In ACL. find
  16. J. Lafferty, A. McCallum, and F. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In ICML. Morgan Kaufmann. find
  17. P. Liang. 2005. Semi-supervised learning for natural language. Masters thesis, Massachusetts Institute of Technology. find
  18. S. Miller, J. Guinness, and A. Zamanian. 2004. Name tagging with word clusters and discriminative training. In HLT-NAACL. find
  19. A. Molina and F. Pla. 2002. Shallow parsing using specialized hmms. The Journal of Machine Learning Research, 2:595–613. find
  20. A. Niculescu-Mizil and R. Caruana. 2005. Predicting good probabilities with supervised learning. In ICML. find
  21. V. Punyakanok and D. Roth. 2001. The use of classifiers in sequential inference. In NIPS. find
  22. L. R. Rabiner. 1989. A tutorial on hidden markov models and selected applications in speech recognition. In IEEE. find
  23. E. Riloff and R. Jones. 1999. Learning dictionaries for information extraction by multi-level bootstrapping. In AAAI. find
  24. N. Rizzolo and D. Roth. 2007. Modeling discriminative global inference. In ICSC. find
  25. D. Roth and D. Zelenko. 1998. Part of speech tagging using a network of linear separators. In COLING-ACL. find
  26. H. Shen and A. Sarkar. 2005. Voting between multiple data representations for text chunking. Advances in Artificial Intelligence, pages 389–400. find
  27. J. Suzuki and H. Isozaki. 2008. Semi-supervised sequential labeling and segmentation using giga-word scale unlabeled data. In ACL. find
  28. E. Tjong, K. and F. De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In CoNLL. find
  29. A. Toral and R. Munoz. 2006. A proposal to automatically build and maintain gazetteers for named entity recognition by using wikipedia. In EACL. find
  30. K. Toutanova, D. Klein, C. Manning, and Y. Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In NAACL. find
  31. J. Veenstra. 1999. Representing text chunks. In EACL. find
  32. T. Zhang and D. Johnson. 2003. A robust risk minimization based named entity recognition system. In CoNLL. find