Elamur Research
☰
home / papers / 2013 / Improved Part-of-Speech Tagging for Online Conversational Text with Word Clusters

Improved Part-of-Speech Tagging for Online Conversational Text with Word Clusters

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/N13-1039.pdf

Summary

Off-the-shelf part-of-speech taggers struggle with chat-like online writing such as tweets and IRC messages. This genre is full of misspellings, slang, abbreviations, emoticons, and invented spellings. The authors extend the earlier Twitter tagger of Gimpel et al. (2011) into a fast first-order MEMM tagger with elastic-net regularization. Its main new ingredient is distributional information from Brown clustering, run on tens of millions of unlabeled English tweets collected over about four years. They also add new lexical features, an improved emoticon detector, and the first annotation guidelines for this kind of text, along with a newly labeled tweet set. The tagger raises Twitter accuracy from roughly 90% to 93%. It also beats earlier results on the RITTERTW and NPSCHAT IRC corpora. The clusters help most on words missing from standard dictionaries. This matters because social-media pipelines need robust, high-throughput tagging. The code, clusters, guidelines, and data are all publicly distributed as version 0.3 of CMU's Twitter POS tagging tool.

Key points

  • Word clusters: Brown clustering grouped words into 1,000 hierarchical clusters. The input was 56 million deduplicated tweets (847 million tokens), and the result covers 216,856 word types.
  • Twitter results: Accuracy rises from about 90% to 93%. On words absent from a 260,985-type dictionary, accuracy without clusters is only 79.2%, and adding clusters improves it by 5.7 points.
  • Other benchmarks:
    • RITTERTW: 90.0% (±0.5), compared with 88.3% for Ritter et al.'s best tagger.
    • NPSCHAT IRC: 93.4% (±0.3), compared with Forsyth's 90.8%.
  • Speed: Greedy decoding processes about 800 tweets/sec (10,000 tokens/sec) on a single CPU core. That is roughly 40 times faster than the earlier system, with under 0.1% accuracy loss compared to Viterbi.
  • Released resources: A 25-tag coarse tagset, the new DAILY547 dataset of 547 annotated tweets, corrected annotations for the earlier dataset, and the guidelines, all available at the TweetNLP site.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Part-of-speech tagging

Cited by

243 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 232 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{owoputi2013improved,
  title = {{Improved Part-of-Speech Tagging for Online Conversational Text with Word Clusters}},
  author = {Owoputi, Olutobi and O'Connor, Brendan and Dyer, Chris and Gimpel, Kevin and Schneider, Nathan and Smith, Noah A.},
  booktitle = {Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2013},
  url = {https://aclanthology.org/N13-1039/}
}

References

38 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Allan and H. Raghavan. 2002. Using part-of-speech patterns to reduce query ambiguity. In Proc. of SIGIR. find
  2. G. Andrew and J. Gao. 2007. Scalable training of L 1 - regularized log-linear models. In Proc. of ICML. find
  3. P. Blunsom and T. Cohn. 2011. A hierarchical Pitman-Yor process HMM for unsupervised part of speech induction. In Proc. of ACL. find
  4. S. Brody and N. Diakopoulos. Cooooooooooooooollllllllllllll!!!!!!!!!!!!!!: using word lengthening to detect sentiment in microblogs. In Proc. of EMNLP. find
  5. P. F. Brown, P. V. de Souza, R. L. Mercer, V. J. Della Pietra, and J. C. Lai. 1992. Class-based n-gram models of natural language. Computational Linguistics, 18(4). find
  6. A. Carlson, J. Betteridge, B. Kisiel, B. Settles, E. R. Hruschka Jr, and T. M. Mitchell. 2010. Toward an architecture for never-ending language learning. In Proc. of AAAI. find
  7. A. Clark. 2003. Combining distributional and morphological information for part of speech induction. In Proc. of EACL. find
  8. J. Eisenstein, N. A. Smith, and E. P. Xing. 2011. Discovering sociolinguistic associations with structured sparsity. In Proc. of ACL. find
  9. J. Eisenstein. 2013. What to do about bad language on the internet. In Proc. of NAACL. find
  10. A. Fader, S. Soderland, and O. Etzioni. 2011. Identifying relations for open information extraction. In Proc. of EMNLP. find
  11. E. N. Forsyth and C. H. Martell. 2007. Lexical and discourse analysis of online chat dialog. In Proc. of ICSC. find
  12. E. N. Forsyth. 2007. Improving automated lexical and discourse analysis of online chat dialog. Master’s thesis, Naval Postgraduate School. find
  13. J. Foster, O. Cetinoglu, J. Wagner, J. L. Roux, S. Hogan, J. Nivre, D. Hogan, and J. van Genabith. 2011. \#hardtoparse: POS tagging and parsing the Twitterverse. In Proc. of AAAI-11 Workshop on Analysing Microtext. find
  14. K. Gimpel, N. Schneider, B. O’Connor, D. Das, D. Mills, J. Eisenstein, M. Heilman, D. Yogatama, J. Flanigan, and N. A. Smith. 2011. Part-of-speech tagging for Twitter: Annotation, features, and experiments. In Proc. of ACL. find
  15. Google. 2012. Freebase data dumps. http://download.freebase.com/datadumps/. ↗
  16. B. Han and T. Baldwin. 2011. Lexical normalisation of short text messages: Makn sens a \#twitter. In Proc. of ACL. find
  17. T. Koo, X. Carreras, and M. Collins. 2008. Simple semi-supervised dependency parsing. In Proc. of ACL. find our card
  18. J. Lafferty, A. McCallum, and F. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proc. of ICML. find
  19. P. Liang. 2005. Semi-supervised learning for natural language. Master’s thesis, Massachusetts Institute of Technology. find
  20. D. C. Liu and J. Nocedal. 1989. On the limited memory BFGS method for large scale optimization. Mathematical programming, 45(1). 2011. find
  21. X. Liu, S. Zhang, F. Wei, and M. Zhou. 2011. Recognizing named entities in tweets. In Proc. of ACL. find
  22. M. Lui and T. Baldwin. 2012. langid.py: An off-the-shelf language identification tool. In Proc. of ACL. find our card
  23. M. P. Marcus, B. Santorini, and M. A. Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2). find our card
  24. A. McCallum, D. Freitag, and F. Pereira. 2000. Maximum entropy Markov models for information extraction and segmentation. In Proc. of ICML. find
  25. D. McClosky, E. Charniak, and M. Johnson. 2010. Automatic domain adaptation for parsing. In Proc. of NAACL. find
  26. B. O’Connor, M. Krieger, and D. Ahn. 2010. TweetMotif: exploratory search and topic summarization for Twitter. In Proc. of AAAI Conference on Weblogs and Social Media. find
  27. O. Owoputi, B. O’Connor, C. Dyer, K. Gimpel, and N. Schneider. 2012. Part-of-speech tagging for Twitter: Word clusters and other advances. Technical Report CMU-ML-12-107, Carnegie Mellon University. find
  28. B. Pang and L. Lee. 2008. Opinion mining and sentiment analysis. Now Publishers. find
  29. S. Petrov and R. McDonald. 2012. Overview of the 2012 shared task on parsing the web. Notes of the First Workshop on Syntactic Analysis of Non-Canonical Language (SANCL). find
  30. S. Petrov, D. Das, and R. McDonald. 2011. A universal part-of-speech tagset. arXiv preprint arXiv:1104.2086. find
  31. A. Ratnaparkhi. 1996. A maximum entropy model for part-of-speech tagging. In Proc. of EMNLP. find
  32. A. Ritter, S. Clark, Mausam, and O. Etzioni. 2011. Named entity recognition in tweets: An experimental study. In Proc. of EMNLP. find
  33. T. Schnoebelen. 2012. Do you smile with your nose? Stylistic variation in Twitter emoticons. University of Pennsylvania Working Papers in Linguistics, 18(2):14. find
  34. N. A. Smith and J. Eisner. 2005. Contrastive estimation: Training log-linear models on unlabeled data. In Proc. of ACL. find
  35. O. Täckström, R. McDonald, and J. Uszkoreit. 2012. Cross-lingual word clusters for direct transfer of linguistic structure. In Proc. of NAACL. find our card
  36. J. Turian, L. Ratinov, and Y. Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In Proc. of ACL. find our card
  37. P. D. Turney. 2002. Thumbs up or thumbs down?: semantic orientation applied to unsupervised classification of reviews. In Proc. of ACL. find
  38. H. Zou and T. Hastie. 2005. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 67(2):301–320. find