Elamur Research⌕
☰
home / papers / 2010 / Twitter as a Corpus for Sentiment Analysis and Opinion Mining

Twitter as a Corpus for Sentiment Analysis and Opinion Mining

Summary

The paper asks whether Twitter can serve as a cheap, ever-growing source of training data for sentiment classification, including the harder three-way split into positive, negative and neutral. The authors collected 300,000 English tweets through the Twitter API without manual annotation. Tweets with happy or sad emoticons became the positive and negative classes. Posts from the accounts of 44 newspapers and magazines served as the objective class. They checked that the corpus behaves like natural text and tagged it with TreeTagger to compare part-of-speech distributions across classes. On this data they trained a multinomial Naïve Bayes classifier using n-gram and POS-tag features. They also tested ways to discard uninformative n-grams. The authors report that their approach beats earlier emoticon-based systems, which struggled with the neutral class. The work matters because it shows distant supervision can produce a three-class sentiment corpus of any size, and the same recipe should carry over to other languages.

Key points

  • Corpus: 300,000 tweets split evenly across three classes.
    • Positive and negative tweets were selected by emoticon queries, such as ":)" and ":-(".
    • Neutral tweets came from 44 news outlets, such as New York Times and Washington Post.
  • Corpus sanity checks:
    • Word frequencies follow Zipf's law.
    • Comparing halves of the same class showed similar POS distributions, which suggests the corpus is homogeneous.
  • POS differences between classes (TreeTagger tagset):
    • Subjective tweets use more interjections (UH), personal pronouns (PP, PP$), first- and second-person verbs (VBP), superlatives (JJS) and adverbs (RB).
    • Objective tweets favour nouns (NN/NP variants), third-person verbs (VBZ) and comparatives (JJR).
    • Negative tweets contain more past-tense verbs, for example "missed", "lost" and "stuck".
  • Classifier: multinomial Naïve Bayes using binary n-gram presence plus POS tags.
    • Salience filtered out common n-grams more effectively than entropy.
  • Prior baselines:
    • Read (2005) reached about 70% accuracy with emoticon-trained classifiers.
    • Go et al. (2009) reached up to 81% for two classes but performed poorly with three.

Interesting ideas

  • I1. Use non-emotional sources as the neutral class. Instead of hand-labelling neutral data, take posts from news accounts. This gives a third class automatically, and any language community with active media accounts could build a three-way dataset this way.
  • I2. Compare tag usage with a simple normalised difference. For each POS tag, take the gap between its counts in two subsets and divide by their sum. This ratio is a quick, interpretable way to find which grammatical features separate any two text collections, such as genres, regions or registers.
  • I3. Check homogeneity by splitting a class in half. If the two halves have closely matching tag distributions, the class is internally consistent. This is a cheap sanity test for any automatically gathered corpus.
  • I4. Treat accuracy and coverage as a trade-off. The classifier can abstain on low-confidence items, and evaluation weighs accuracy against the share of items it labels ("decision"), using an F-measure with β = 0.5. This suits search or monitoring tools, where precise answers on fewer items beat noisy labels on everything.
  • I5. Inspect the corpus when results look odd. An unexpected link between "whose" and positive tweets turned out to be slang for "who's". Looking at actual examples prevents misreading such artefacts in noisy user-generated text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Part-of-speech tagging

Cited by

200 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 197 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{pak2010twitter,
  title = {{Twitter as a Corpus for Sentiment Analysis and Opinion Mining}},
  author = {Pak, Alexander and Paroubek, Patrick},
  booktitle = {Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)},
  year = {2010},
  doi = {10.63317/47akcj8foo4d},
  url = {https://doi.org/10.63317/47akcj8foo4d}
}

References

16 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. G. Adda, J. Mariani, J. Lecomte, P. Paroubek, and M. Rajman. 1998. The GRACE French part-of-speech tagging evaluation task. In A. Rubio, N. Gallardo, R. Castro, and A. Tejada, editors, LREC, volume I, pages 433–441, Granada, May. find
  2. Ethem Alpaydin. 2004. Introduction to Machine Learning (Adaptive Computation and Machine Learning). The MIT Press. find
  3. Hayter Anthony J. 2007. Probability and Statistics for Engineers and Scientists. Duxbury, Belmont, CA, USA. find
  4. Kushal Dave, Steve Lawrence, and David M. Pennock. 2003. Mining the peanut gallery: opinion extraction and semantic classification of product reviews. In WWW ’03: Proceedings of the 12th international conference on World Wide Web, pages 519–528, New York, NY, USA. ACM. find
  5. Alec Go, Lei Huang, and Richa Bhayani. 2009. Twitter sentiment analysis. Final Projects from CS224N for Spring 2008/2009 at The Stanford Natural Language Processing Group. find
  6. Bernard J. Jansen, Mimi Zhang, Kate Sobel, and Abdur Chowdury. 2009. Micro-blogging as online word of mouth branding. In CHI EA ’09: Proceedings of the 27th international conference extended abstracts on Human factors in computing systems, pages 3859–3864, New York, NY, USA. ACM. find
  7. John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In ICML ’01: Proceedings of the Eighteenth International Conference on Machine Learning, pages 282–289, San Francisco, CA, USA. Morgan Kaufm… find
  8. Christopher D. Manning and Hinrich Schütze. 1999. Foundations of statistical natural language processing. MIT Press, Cambridge, MA, USA. find
  9. Bo Pang and Lillian Lee. 2008. Opinion mining and sentiment analysis. Found. Trends Inf. Retr., 2(1-2):1–135. find
  10. Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up? sentiment classification using machine learning techniques. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 79–86. find
  11. Ted Pedersen. 2000. A simple approach to building ensembles of naive bayesian classifiers for word sense disambiguation. In Proceedings of the 1st North American chapter of the Association for Computational Linguistics conference, pages 63–69, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. find
  12. Jonathon Read. 2005. Using emoticons to reduce dependency in machine learning techniques for sentiment classification. In ACL. The Association for Computer Linguistics. find
  13. Helmut Schmid. 1994. Probabilistic part-of-speech tagging using decision trees. In Proceedings of the International Conference on New Methods in Language Processing, pages 44–49. find
  14. Claude E. Shannon and Warren Weaver. 1963. A Mathematical Theory of Communication. University of Illinois Press, Champaign, IL, USA. find
  15. Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In HLT ’05: Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, pages 347– 354, Morristown, NJ, USA. Association for… find
  16. Changhua Yang, Kevin Hsin-Yih Lin, and Hsin-Hsi Chen. 2007. Emotion classification using web blog corpora. In WI ’07: Proceedings of the IEEE/WIC/ACM International Conference on Web Intelligence, pages 275–278, Washington, DC, USA. IEEE Computer Society. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/pak-2010-twitter-as-corpus-sentiment-analysis/