Elamur Research⌕
☰
home / papers / 2011 / Named Entity Recognition in Tweets: An Experimental Study

Named Entity Recognition in Tweets: An Experimental Study

Summary

Standard NLP tools trained on newswire perform poorly on Twitter, where messages are short, noisy and full of nonstandard spelling and capitalization. Ritter and colleagues rebuild the shallow-processing pipeline for tweets in stages. They retrain a POS tagger (T-POS), a chunker (T-CHUNK) and a classifier that judges whether a tweet's capitalization can be trusted (T-CAP). They then use the outputs of these tools as features for named entity segmentation. Each component combines a small set of hand-annotated tweets with out-of-domain labeled data and with word clusters learned from millions of unlabeled tweets. Entity types are assigned with a distant-supervision approach. LabeledLDA is constrained by Freebase entity dictionaries and pools contextual evidence across all mentions of an entity, so no manually typed training examples are needed. The full system, T-NER, roughly doubles the F1 of Stanford NER on the three MUC types, and LabeledLDA improves F1 by 25% over co-training. The work showed early that pipelines adapted to a new domain, combined with knowledge-base supervision, can make informal social media text usable for information extraction. The tools are publicly released.

Key points

  • With news-trained tools, POS accuracy falls from about 0.97 on news to roughly 0.80 on tweets. The annotated corpus has 800 tweets (about 16K tokens), labeled with Penn Treebank tags plus extra tags for retweets, @usernames, #hashtags and URLs.
  • T-POS is a CRF built with MALLET. It is trained on Twitter data together with 40K tokens of IRC chat and 50K tokens of PTB, reaching 0.883 accuracy. This is a 41% error reduction compared with the Stanford POS tagger.
  • Brown clusters were computed with Jcluster on 52 million tweets, using bit-prefixes of length 4, 8 and 12. The clusters group dozens of spelling variants together, for example many forms of "tomorrow."
  • T-CHUNK reaches 0.875 token accuracy, a 22% error reduction over OpenNLP, using 16K in-domain tokens plus 210K CoNLL tokens. Entity segmentation F1 improves by 52% over off-the-shelf tools.
  • For entity classification over ten common types, LabeledLDA with Freebase constraints beats DL-Cotrain by 25% in F1. End to end, T-NER doubles the F1 of Stanford NER on the MUC types. The code is available at github.com/aritter/twitter_nlp.

Interesting ideas

  • I1. Mix domains for training data. Combining a small in-domain sample with a stylistically similar corpus (IRC chat) and a large formal corpus (PTB) worked better than any single source. This is a cheap strategy for low-resource genres, such as Ukrainian social media, where a related informal corpus can supplement a few hundred annotated examples.
  • I2. Classify whether a signal is reliable. Rather than always using capitalization as an NER cue, the authors train a separate model that predicts whether a given message's casing is informative, and feed that prediction in as a feature. This pattern of gating a cue by its estimated reliability can apply to other noisy cues, such as punctuation or diacritics in user-generated text.
  • I3. Use clusters to handle spelling variation. Hierarchical Brown clusters with several prefix lengths absorb misspellings and abbreviations without an explicit normalization step. This makes the approach useful wherever spelling is inconsistent.
  • I4. Pool evidence across all mentions of an entity. LabeledLDA treats all contexts of an entity string as one document and learns a distribution over its possible types. That distribution then serves as a prior when typing an individual mention. This compensates for the very short context of each tweet and helps in any setting with terse texts.
  • I5. Use a knowledge base to constrain unsupervised models. Freebase dictionaries restrict which types an entity may take inside a topic model, so many types can be covered without hand-labeled examples. The same idea could be applied with other knowledge bases or gazetteers, such as geographic or company registries.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Named entity recognition · Part-of-speech tagging

Cited by

287 papers of our library cite it (of the 140884 whose bibliographies we read); 28 of them have a page here, 259 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{ritter2011named,
  title = {{Named Entity Recognition in Tweets: An Experimental Study}},
  author = {Ritter, Alan and Clark, Sam and Mausam and Etzioni, Oren},
  booktitle = {Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing},
  year = {2011},
  url = {https://aclanthology.org/D11-1141/}
}

References

43 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Edward Benson, Aria Haghighi, and Regina Barzilay. 2011. Event discovery in social media feeds. In The 49th Annual Meeting of the Association for Computational Linguistics, Portland, Oregon, USA. To appear. find
  2. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. J. Mach. Learn. Res. find our card
  3. Avrim Blum and Tom M. Mitchell. 1998. Combining labeled and unlabeled sata with co-training. In COLT, pages 92–100. find
  4. Peter F. Brown, Peter V. deSouza, Robert L. Mercer, Vincent J. Della Pietra, and Jenifer C. Lai. 1992. Class-based n-gram models of natural language. Comput. Linguist. find
  5. Andrew Carlson, Justin Betteridge, Richard C. Wang, Estevam R. Hruschka, Jr., and Tom M. Mitchell. 2010. Coupled semi-supervised learning for information extraction. In Proceedings of the third ACM international conference on Web search and data mining, WSDM ’10. find
  6. Eugene Charniak, Curtis Hendrickson, Neil Jacobson, and Mike Perkowitz. 1993. Equations for part-of-speech tagging. In AAAI, pages 784–789. find
  7. Michael Collins and Yoram Singer. 1999. Unsupervised models for named entity classification. In Empirical Methods in Natural Language Processing. find our card
  8. Doug Downey, Matthew Broadhead, and Oren Etzioni. 2007. Locating complex named entities in web text. In Proceedings of the 20th international joint conference on Artifical intelligence. find
  9. Doug Downey, Oren Etzioni, and Stephen Soderland. 2010. Analysis of a probabilistic model of redundancy in unsupervised information extraction. Artif. Intell., 174(11):726–748. find
  10. Micha Elsner, Eugene Charniak, and Mark Johnson. 2009. Structured generative models for unsupervised named-entity clustering. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, NAACL ’09. find
  11. Oren Etzioni, Michael Cafarella, Doug Downey, Ana-Maria Popescu, Tal Shaked, Stephen Soderland, Daniel S. Weld, and Alexander Yates. 2005. Unsupervised named-entity extraction from the web: an experimental study. Artif. Intell. find
  12. Tim Finin, Will Murnane, Anand Karandikar, Nicholas Keller, Justin Martineau, and Mark Dredze. 2010. Annotating named entities in Twitter data with crowdsourcing. In Proceedings of the NAACL Workshop on Creating Speech and Text Language Data With Amazon’s Mechanical Turk. Association for Computation… find
  13. Jenny Rose Finkel, Trond Grenager, and Christopher Manning. 2005. Incorporating non-local information into information extraction systems by gibbs sampling. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, ACL ’05. find our card
  14. Radu Florian. 2002. Named entity recognition as a house of cards: classifier stacking. In Proceedings of the 6th conference on Natural language learning - Volume 20, COLING-02. find
  15. Eric N. Forsythand and Craig H. Martell. 2007. Lexical and discourse analysis of online chat dialog. In Proceedings of the International Conference on Semantic Computing. find
  16. Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A. Smith. 2011. Part-of-speech tagging for twitter: Annotation, features, and experiments. In ACL. find our card
  17. Joshua T. Goodman. 2001. A bit of progress in language modeling. Technical report, Microsoft Research. find
  18. Stephan Gouws, Donald Metzler, Congxing Cai, and Eduard Hovy. 2011. Contextual bearing on linguistic variation in social media. In ACL Workshop on Language in Social Media, Portland, Oregon, USA. To appear. find
  19. T. L. Griffiths and M. Steyvers. 2004. Finding scientific topics. Proceedings of the National Academy of Sciences, April. find
  20. Mark Hachman. 2011. Humanity’s tweets: Just 20 terabytes. In PCMAG.COM. find
  21. Bo Han and Timothy Baldwin. 2011. Lexical normalisation of short text messages: Makn sens a \#twitter. In The 49th Annual Meeting of the Association for Computational Linguistics, Portland, Oregon, USA. To appear. find our card
  22. Catherine Kobus, François Yvon, and Géraldine Damnati. 2008. Normalizing sms: are two metaphors better than one ? In COLING, pages 441–448. find
  23. Zornitsa Kozareva and Eduard H. Hovy. 2010. Not all seeds are equal: Measuring the quality of text mining seeds. In HLT-NAACL. find
  24. John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML ’01, pages 282–289, San Francisco, CA, USA. Morgan Kaufm… find
  25. Xiaohua Liu, Shaodian Zhang, Furu Wei, and Ming Zhou. 2011. Recognizing named entities in tweets. In ACL. find
  26. Brian Locke and James Martin. 2009. Named entity recognition: Adapting to microblogging. In Senior Thesis, University of Colorado. find
  27. Mitchell P. Marcus, Beatrice Santorini, and Mary A. Marcinkiewicz. 1994. Building a large annotated corpus of english: The penn treebank. Computational Linguistics. find our card
  28. Andrew Kachites McCallum. 2002. Mallet: A machine learning for language toolkit. In http://mallet.cs.umass.edu. ↗
  29. Tara McIntosh. 2010. Unsupervised discovery of negative categories in lexicon bootstrapping. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, EMNLP ’10. find
  30. Einat Minkov, Richard C. Wang, and William W. Cohen. 2005. Extracting personal names from email: applying named entity recognition to informal text. In Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, HLT ’05, pages 443–450, Morristown,… find
  31. Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of ACL-IJCNLP 2009. find
  32. Christoph Müller and Michael Strube. 2006. Multi-level annotation of linguistic data with MMAX2. In Sabine Braun, Kurt Kohn, and Joybrato Mukherjee, editors, Corpus Technology and Language Pedagogy: New Resources, New Tools, New Methods, pages 197–214. Peter Lang, Frankfurt a.M., Germany. find
  33. Daniel Ramage, David Hall, Ramesh Nallapati, and Christopher D. Manning. 2009. Labeled lda: a supervised topic model for credit attribution in multi-labeled corpora. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1 - Volume 1, EMNLP ’09, pages 248–2… find
  34. Christina Sauper, Aria Haghighi, and Regina Barzilay. 2010. Incorporating content structure into text analysis applications. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, EMNLP ’10, pages 377–387, Morristown, NJ, USA. Association for Computational Linguis… find
  35. Fei Sha and Fernando Pereira. 2003. Shallow parsing with conditional random fields. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1, NAACL ’03. find
  36. Sameer Singh, Dustin Hillard, and Chris Leggetter. 2010. Minimally-supervised extraction of entities from text advertisements. In Human Language Technologies: Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL HLT). find
  37. Charles Sutton. 2004. Collective segmentation and labeling of distant entities in information extraction. find
  38. Partha Pratim Talukdar and Fernando Pereira. 2010. Experiments in graph-based semi-supervised learning methods for class-instance acquisition. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1473–1481. Association for Computational Linguistics. find
  39. Erik F. Tjong Kim Sang and Sabine Buchholz. 2000. Introduction to the conll-2000 shared task: chunking. In Proceedings of the 2nd workshop on Learning language in logic and the 4th conference on Computational natural language learning - Volume 7, ConLL ’00. find our card
  40. Kristina Toutanova, Dan Klein, Christopher D. Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Vol… find our card
  41. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, ACL ’10. find our card
  42. Limin Yao, David Mimno, and Andrew McCallum. 2009. Efficient methods for topic model inference on streaming document collections. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining. find
  43. David Yarowsky. 1995. Unsupervised word sense disambiguation rivaling supervised methods. In Proceedings of the 33rd annual meeting on Association for Computational Linguistics, ACL ’95. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ritter-2011-named-entity-recognition-tweets-experimental/