Elamur Research⌕
☰
home / papers / 2013 / Twitter Part-of-Speech Tagging for All: Overcoming Sparse and Noisy Data

Twitter Part-of-Speech Tagging for All: Overcoming Sparse and Noisy Data

Summary

Standard PoS taggers trained on newswire perform badly on English tweets. Tweets are short, noisy and full of slang, and only a few small annotated tweet corpora exist. The authors compare four off-the-shelf taggers (TnT, SVMTool, TBL, Stanford) on Twitter data and categorise the errors they make on unknown words. They then extend the Stanford tagger in three ways: handling of rare and slang words, prior tag probabilities for unambiguous tokens, and a new vote-constrained bootstrapping scheme. In that scheme, two existing taggers that use different tagsets label unlabeled tweets. A tweet is kept as training data only if the two outputs agree under predefined tag-equivalence classes. This produced about 1.5M tokens of high-confidence silver data. The final tagger reaches 88.7% token accuracy on held-out data and 90.5% on development data, using the PTB-compatible T-Pos tagset. Compared with T-Pos, this cuts token error by 26.8% and whole-tweet error by 12.2%. The tagger, bootstrapped corpus and creation tool are released, including through GATE TwitIE.

Key points

  • Accuracy drops sharply on tweets:
    • A WSJ-trained Stanford tagger gets only 73.37% token accuracy on T-dev.
    • Adding IRC (NPS corpus) and tweet training data raises this to 83.14%.
    • Accuracy on unknown words stays at just 38.56%.
  • Data and evaluation setup:
    • The T-Pos corpus (~15K tokens) is split 70:15:15 into T-train, T-dev and T-eval.
    • The DCU data is converted to the T-Pos tokenisation and tagset to give D-dev and D-eval.
    • Sentence accuracy is measured strictly, as fully correct whole tweets.
  • Error analysis of 89 mis-tagged unknown words: in-vocabulary-type errors and slang each account for 27%, proper nouns 11.2%, and tokenisation 9%.
  • Bootstrapping results:
    • From 832,135 English tweets taken from the garden hose, the taggers agreed on 19.2%, giving 159,492 tweets (~1.54M tokens).
    • On T-dev, agreed labelings were correct for 97.4% of tokens.
  • Final results versus T-Pos:
    • On T-eval: 88.69% vs 84.55% token accuracy, and 20.34% vs 9.32% sentence accuracy.
    • On D-eval: 89.37% vs 84.83% token accuracy.
    • Significance was tested with McNemar's test (p<0.01).

Interesting ideas

  • I1. Agreement filtering across tagsets. Taggers with incompatible tagsets can still be combined: map their labels into equivalence classes and keep only whole items on which they agree. This turns heterogeneous tools into a source of high-precision silver data. It is useful for any low-resource language or genre where existing tools follow different annotation schemes.
  • I2. Whole-item agreement as a quality gate. The filter requires agreement across the entire tweet, not per token. Only about a fifth of the data survives, but token precision of the kept data is very high (~97%). This trades coverage for reliability, which suits massive unlabeled streams.
  • I3. Report whole-sequence accuracy. Counting fully correct tweets alongside token accuracy exposes how often a downstream component receives a flawless input. That is a more honest picture for pipelines than token scores alone.
  • I4. Normalise test sets to one schema. Converting a second corpus (DCU) to the same tokenisation and tagset allows cross-corpus testing. This showed that gains on one tweet corpus transfer only partly to another, a cheap way to detect overfitting to a single dataset.
  • I5. Track the learning curve of silver data. Plotting accuracy against the amount of bootstrapped data revealed an early dip followed by steady gains, and suggested that more data could help further. Such curves are a sensible check before scaling up any self-training setup.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

TopicsPart-of-speech tagging

Institutions

🎓University of WashingtonSeattle · United States206 works · 158,423 citations🎓University of SheffieldSheffield · United Kingdom67 works · 17,924 citations

Cite

Cite as Leon Derczynski, Alan Ritter, Sam Clark and Kalina Bontcheva. 2013. Twitter Part-of-Speech Tagging for All: Overcoming Sparse and Noisy Data. Proceedings of the International Conference Recent Advances in Natural Language Processing RANLP 2013.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{derczynski2013twitter,
  title = {{Twitter Part-of-Speech Tagging for All: Overcoming Sparse and Noisy Data}},
  author = {Leon Derczynski, Alan Ritter, Sam Clark and Kalina Bontcheva},
  booktitle = {Proceedings of the International Conference Recent Advances in Natural Language Processing RANLP 2013},
  year = {2013},
  url = {https://aclanthology.org/R13-1026/}
}

References

34 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. T. Almeida, J. Hidalgo, and A. Yamakami. 2011. Contributions to the study of SMS spam filtering: new collection and results. Proceedings of the 11th ACM symposium on Document engineering, pages 259– 62. find
  2. O. Ashtari. 2013. The super tweets of \#sb47. http://blog.twitter.com/2013/02/the-super-tweets-of-sb47.html. ↗
  3. S. Bird, E. Loper, and E. Klein. 2009. Natural Language Processing with Python. O’Reilly Media Inc. find
  4. K. Bontcheva, L. Derczynski, A. Funk, M. A. Greenwood, D. Maynard, and N. Aswani. 2013. TwitIE: A Fully-featured Information Extraction Pipeline for Microblog Text. In Proceedings of the International Conference on Recent Advances in Natural Language Processing. Association for Computational Linguis… find
  5. T. Brants. 2000. TnT: a statistical part-of-speech tagger. In Proceedings of the sixth conference on Applied Natural Language Processing, pages 224–231. ACL. our card
  6. E. Brill. 1995. Transformation-based error-driven learning and natural language processing: A case study in part-of-speech tagging. Computational Linguistics, 21(4):543–565. our card
  7. S. Clark, J. Curran, and M. Osborne. 2003. Bootstrapping PoS taggers using unlabelled data. In Proceedings of the seventh Conference on Natural Language Learning, pages 49–55. ACL. find
  8. A. Clark. 2003. Combining distributional and morphological information for part of speech induction. In Proceedings of the tenth conference of the European chapter of the Association for Computational Linguistics, pages 59–66. ACL. our card
  9. S. Cucerzan and D. Yarowsky. 2002. Bootstrapping a multilingual part-of-speech tagger in one person-day. In Proceedings of the 6th Conference on Natural Language Learning, pages 1–7. ACL. find
  10. A. Culotta. 2010. Towards detecting influenza epidemics by analyzing twitter messages. In Proceedings of the First Workshop on Social Media Analytics, pages 115–122. ACM. find
  11. H. Cunningham, D. Maynard, K. Bontcheva, and V. Tablan. 2002. GATE: an Architecture for Development of Robust HLT Applications. In Proceedings of the 40th Anniversary Meeting of the Association for Computational Linguistics. ACL. find
  12. H. Cunningham, V. Tablan, A. Roberts, and K. Bontcheva. 2013. Getting more out of biomedical documents with gate’s full lifecycle open source text analytics. PLoS Computational Biology, 9(2):e1002854, 02. find
  13. W. Darling, M. Paul, and F. Song. 2012. Unsupervised part-of-speech tagging in noisy and esoteric domains with a syntactic-semantic Bayesian HMM. In Proceedings of the conference of the European chapter of the Association for Computational Linguistics, pages 1–9. ACL. find
  14. L. Derczynski, D. Maynard, N. Aswani, and K. Bontcheva. 2013. Microblog-Genre Noise and Impact on Semantic Annotation Accuracy. In Proceedings of the 24th ACM Conference on Hypertext and Social Media. ACM. find
  15. E. Forsyth and C. Martell. 2007. Lexical and discourse analysis of online chat dialog. In International Conference on Semantic Computing, pages 19–26. IEEE. find
  16. J. Foster, O. Cetinoglu, J. Wagner, J. Le Roux, S. Hogan, J. Nivre, D. Hogan, and J. van Genabith. 2011. \#hardtoparse: POS Tagging and Parsing the Twitterverse. In Proceedings of the AAAI Workshop on Analyzing Microtext. find
  17. P. Gadde, L. Subramaniam, and T. Faruquie. 2011. Adapting a wsj trained part-of-speech tagger to noisy text: preliminary results. In Proceedings of the Joint Workshop on Multilingual OCR and Analytics for Noisy Unstructured Text Data, pages 5:1–5:8. ACM. find
  18. J. Giménez and L. Marquez. 2004. SVMTool: A general POS tagger generator based on Support Vector Machines. In Proceedings of the 4th International Conference on Language Resources and Evaluation. our card
  19. K. Gimpel, N. Schneider, B. O’Connor, D. Das, D. Mills, J. Eisenstein, M. Heilman, D. Yogatama, J. Flanigan, and N. Smith. 2011. Part-of-speech tagging for twitter: annotation, features, and experiments. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Huma… our card
  20. S. Goldman and Y. Zhou. 2000. Enhancing supervised learning with unlabeled data. In Proceedings of the International Conference on Machine Learning, pages 327–334. find
  21. B. Han and T. Baldwin. 2011. Lexical normalisation of short text messages: Makn sens a\# twitter. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, volume 1, pages 368–378. our card
  22. M. Kaufmann and J. Kalita. 2010. Syntactic normalization of twitter messages. In International Conference on Natural Language Processing. find
  23. F. Liu, F. Weng, and X. Jiang. 2012. A broadcoverage normalization system for social media language. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics. find
  24. C. Manning. 2011. Part-of-speech tagging from 97% to 100%: is it time for some linguistics? Computational Linguistics and Intelligent Text Processing, pages 171–189. our card
  25. M. Marcus, M. Marcinkiewicz, and B. Santorini. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330. our card
  26. Q. McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157. find
  27. O’Carroll. 2012. Twitter active users pass 200 million. http://www.guardian.co.uk/technology/2012/dec/18/twitter-users-pass-200-million. ↗
  28. O. Owoputi, B. O’Connor, C. Dyer, K. Gimpel, and N. Schneider. 2012. Part-of-Speech Tagging for Twitter: Word Clusters and Other Advances. Technical Report CMU-ML-12-107, Machine Learning Department, Carnegie Mellon University. find
  29. J. Park, V. Barash, C. Fink, and M. Cha. 2013. Emoticon Style: Interpreting Differences in Emoticons Across Cultures. In Proceedings of the Seventh International AAAI Conference on Weblogs and Social Media, pages 466–475. AAAI Press. find
  30. D. Preotiuc-Pietro, S. Samangooei, T. Cohn, N. Gibbins, and M. Niranjan. 2012. Trendminer: An architecture for real time analysis of social media text. In Proceedings of the workshop on Real-Time Analysis and Mining of Social Streams. find
  31. A. Ritter, S. Clark, O. Etzioni, et al. 2011. Named entity recognition in tweets: an experimental study. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1524– 1534. ACL. our card
  32. T. Sakaki, M. Okazaki, and Y. Matsuo. 2010. Earthquake shakes Twitter users: real-time event detection by social sensors. In Proceedings of the 19th international conference on World Wide Web (WWW), pages 851–860. ACM. find
  33. D. Terdiman. 2012. Report: Twitter hits half a billion tweets a day. http://news.cnet.com/8301-1023 3-57541566-93/report-twitter-hits-half-a-billion-tweets-a-day/. ↗
  34. K. Toutanova, D. Klein, C. Manning, and Y. Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, pages 173–180. ACL. our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/derczynski-2013-twitter-part-speech-tagging-all/