Elamur Research⌕
☰
home / papers / 2011 / Part-of-Speech Tagging for Twitter: Annotation, Features, and Experiments

Part-of-Speech Tagging for Twitter: Annotation, Features, and Experiments

Summary

Taggers trained on newswire, such as those built from the Penn Treebank WSJ corpus, perform badly on tweets. Tweets are conversational, often misspelled, short, and full of platform-specific tokens. The authors built an English POS tagger designed for Twitter. They defined a coarse 25-tag inventory that adds categories for hashtags, at-mentions, URLs, emoticons, discourse markers like RT, and fused forms such as noun+verb contractions. Seventeen annotators corrected automatic pre-tags on 1,827 tweets, followed by review and consistency passes. The tagger is a CRF that combines base lexical features with Twitter-oriented ones: orthographic rules, capitalization gazetteers, a soft PTB tag dictionary, distributional clusters from unlabeled tweets, and Metaphone phonetic keys. It reaches about 89.4% test accuracy, cutting errors by roughly a quarter compared with a retrained Stanford tagger. The whole effort took about 200 person-hours over two months. The corpus and tagger were released as TweetNLP. The work shows that a careful tagset and well-chosen features can quickly produce a usable tool for a messy new domain.

Key points

  • Corpus: 1,827 English tweets (26,436 tokens), sampled from one day in October 2010 and split into 1,000 train, 327 dev and 500 test tweets.
  • Agreement: On 72 independently re-annotated tweets, inter-annotator agreement was 92.2%, with Cohen's κ = 0.914.
  • Accuracy:
    • Full model: 89.37% test.
    • Base features only: 83.38%.
    • Stanford tagger retrained on the same data: 85.85%.
  • Ablations:
    • Removing DISTSIM or TAGDICT costs about 1.06 points each.
    • Removing TWORTH costs about 1.00.
    • Removing METAPH costs about 0.42.
    • NAMES gave essentially no gain.
  • Weak spots and data efficiency: Proper-noun recall is only 71%, and the miscellaneous G class is the hardest. Training on just 500 tweets still yields 87.66%.

Interesting ideas

  • I1. Pre-tag, then correct: Annotators fixed the output of an existing out-of-domain tagger plus heuristics rather than labeling from scratch. This can sharply cut annotation time when building a corpus for a new domain or a low-resource variety.
  • I2. Leave fused forms whole: Instead of forcing contractions and possessives to split, the tagset adds combined tags such as nominal+verbal. This avoids brittle tokenization of nonstandard spellings and could suit colloquial text in other languages with clitics or slang fusions.
  • I3. Soft external dictionaries: Tag-dictionary entries from an external treebank enter the model as features rather than hard constraints. This is a cheap form of type-level domain adaptation that tolerates poor lexical coverage and dialect drift.
  • I4. Phonetic keys for spelling variants: Metaphone keys group variants (e.g., several spellings of "thanks") into one feature, and the model also checks the most common PTB tag for words sharing that key. A language-appropriate phonetic or transliteration normalizer could do the same for noisy user text elsewhere.
  • I5. Context-based features for rare words: Predecessor and successor statistics from about 134,000 unlabeled in-domain tweets help with words seen rarely or never in training. This is useful whenever labeled data is tiny but raw text is plentiful.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Part-of-speech tagging

Cited by

276 papers of our library cite it (of the 140884 whose bibliographies we read); 43 of them have a page here, 233 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{gimpel2011part,
  title = {{Part-of-Speech Tagging for Twitter: Annotation, Features, and Experiments}},
  author = {Gimpel, Kevin and Schneider, Nathan and O'Connor, Brendan and Das, Dipanjan and Mills, Daniel and Eisenstein, Jacob and Heilman, Michael and Yogatama, Dani and Flanigan, Jeffrey and Smith, Noah A.},
  booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies},
  year = {2011},
  doi = {10.21236/ada547371},
  url = {https://doi.org/10.21236/ada547371}
}

References

18 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sitaram Asur and Bernardo A. Huberman. 2010. Predicting the future with social media. In Proc. of WI-IAT. find
  2. Luciano Barbosa and Junlan Feng. 2010. Robust sentiment detection on Twitter from biased and noisy data. In Proc. of COLING. find our card
  3. Lauren Collister. 2010. Meaning variation of the iconic deictics ˆ and \<— in an online community. In New Ways of Analyzing Variation. find
  4. Christiane Fellbaum. 1998. WordNet: An Electronic Lexical Database. Bradford Books. find
  5. Tim Finin, Will Murnane, Anand Karandikar, Nicholas Keller, Justin Martineau, and Mark Dredze. 2010. Annotating named entities in Twitter data with crowdsourcing. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk. find
  6. John Lafferty, Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proc. of ICML. find
  7. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19:313–330. find our card
  8. Brendan O’Connor, Ramnath Balasubramanyan, Bryan R. Routledge, and Noah A. Smith. 2010a. From tweets to polls: Linking text sentiment to public opinion time series. In Proc. of ICWSM. find
  9. Brendan O’Connor, Michel Krieger, and David Ahn. 2010b. TweetMotif: Exploratory search and topic summarization for Twitter. In Proc. of ICWSM (demo track). find
  10. Slav Petrov, Dipanjan Das, and Ryan McDonald. 2011. A universal part-of-speech tagset. ArXiv:1104.2086. find
  11. Lawrence Philips. 1990. Hanging on the Metaphone. Computer Language, 7(12). find
  12. Alan Ritter, Colin Cherry, and Bill Dolan. 2010. Unsupervised modeling of Twitter conversations. In Proc. of NAACL. find our card
  13. Hinrich Schütze and Jan Pedersen. 1993. A vector model for syntagmatic and paradigmatic relatedness. In Proceedings of the 9th Annual Conference of the UW Centre for the New OED and Text Research. find
  14. Beaux Sharifi, Mark-Anthony Hutton, and Jugal Kalita. 2010. Summarizing microblogs automatically. In Proc. of NAACL. find
  15. Mike Thelwall, Kevan Buckley, and Georgios Paltoglou. 2011. Sentiment in Twitter events. Journal of the American Society for Information Science and Technology, 62(2):406–418. find
  16. Kristina Toutanova, Dan Klein, Christopher D. Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In Proc. of HLT-NAACL. find our card
  17. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Proc. of ACL. find our card
  18. Grady Ward. 1996. Moby lexicon. http://icon.shef.ac.uk/Moby. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gimpel-2011-part-speech-tagging-twitter-annotation/