Thumbs up? Sentiment Classification using Machine Learning Techniques

Summary

The paper asks whether deciding if a whole document expresses a positive or negative opinion can be treated as ordinary topic categorization, or whether it needs special methods. The authors built a corpus of movie reviews from the IMDb archive of the rec.arts.movies.reviews newsgroup. Labels came automatically from star or numeric ratings, and each author's contribution was capped so that prolific reviewers would not dominate. They first tested baselines built from small word lists chosen by people. They then trained Naive Bayes, maximum entropy and support vector machine classifiers on bag-of-features representations, trying unigrams, bigrams, adjectives and positional tags. All three learners clearly beat the human word lists. SVMs tended to perform best and Naive Bayes worst, though the gaps were modest, and simple unigram presence was the most effective feature. Even so, accuracy remained below typical topic-classification levels. An error analysis blames rhetorical patterns in which much of a review's vocabulary contradicts its final verdict. The work provides an early knowledge-free benchmark and a publicly released dataset for sentiment classification.

Key points

Corpus: 2053 reviews (752 negative, 1301 positive) from 144 reviewers, with no single author contributing 20 or more reviews to either sentiment class. Neutral reviews were extracted but not used.
Human word-list baselines: on a balanced 700/700 set, lists from two people scored 58% and 64% accuracy, with high tie rates of 75% and 39%. A 14-item list informed by corpus counts, including "?" and "still", reached 69% with only 16% ties.
Feature results: binary presence of unigrams beat frequency counts, which is the opposite of earlier topic-classification findings. The 2633 adjectives did worse than the 2633 most frequent unigrams, which performed comparably to all 16165 unigrams.
Position tagging: marking words by first quarter, middle half or last quarter barely changed results.
Prior work comparison: Turney (2002) had reported 65.83% on 120 movie reviews and found this domain the hardest he tested.

Interesting ideas

I1. Free labels from ratings: using ratings already attached to texts (stars, scores) as automatic labels avoids manual annotation. This transfers to any domain with review scores, such as product or service feedback.
I2. Capping items per author: limiting how many documents one author contributes stops a few voices from biasing the model. This is a cheap fix for any user-generated corpus.
I3. Introspection as a baseline: turning people's guessed keyword lists into a counting classifier gives a concrete reference point. It showed that intuition misses useful cues like punctuation, which is a good argument for data-driven feature selection.
I4. Presence versus frequency: for opinion-like tasks, binary word presence can outperform counts. It is worth testing both rather than assuming topic-classification defaults carry over.
I5. Error analysis by rhetorical pattern: grouping failures into types such as "thwarted expectations" (praise followed by a negative verdict) or a strong actor in a weak film shows where bag-of-features models break. It also points to next steps, such as detecting whether a sentence is actually about the main subject.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

737 papers of our library cite it (of the 140884 whose bibliographies we read); 56 of them have a page here, 681 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/1118693.1118704

@inproceedings{pang2002thumbs,
  title = {{Thumbs up? Sentiment Classification using Machine Learning Techniques}},
  author = {Pang, Bo and Lee, Lillian and Vaithyanathan, Shivakumar},
  booktitle = {Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002)},
  year = {2002},
  doi = {10.3115/1118693.1118704},
  url = {https://doi.org/10.3115/1118693.1118704}
}

References

31 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Shlomo Argamon-Engelson, Moshe Koppel, and Galit Avneri. 1998. Style-based text categorization: What newspaper am I reading? In Proc. of the AAAI Workshop on Text Categorization, pages 1–4. find
Adam L. Berger, Stephen A. Della Pietra, and Vincent J. Della Pietra. 1996. A maximum entropy approach to natural language processing. Computational Linguistics, 22(1):39–71. find our card
Douglas Biber. 1988. Variation across Speech and Writing. Cambridge University Press. find
Stanley Chen and Ronald Rosenfeld. 2000. A survey of smoothing techniques for ME models. IEEE Trans. Speech and Audio Processing, 8(1):37–50. find
Sanjiv Das and Mike Chen. 2001. Yahoo! for Amazon: Extracting market sentiment from stock message boards. In Proc. of the 8th Asia Pacific Finance Association Annual Conference (APFA 2001). find
Stephen Della Pietra, Vincent Della Pietra, and John Lafferty. 1997. Inducing features of random fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(4):380–393. find
Pedro Domingos and Michael J. Pazzani. 1997. On the optimality of the simple Bayesian classifier under zero-one loss. Machine Learning, 29(2-3):103– 130. find
Aidan Finn, Nicholas Kushmerick, and Barry Smyth. 2002. Genre classification and domain transfer for information filtering. In Proc. of the European Colloquium on Information Retrieval Research, pages 353–362, Glasgow. find
Vasileios Hatzivassiloglou and Kathleen McKeown. 1997. Predicting the semantic orientation of adjectives. In Proc. of the 35th ACL/8th EACL, pages 174–181. find
Vasileios Hatzivassiloglou and Janyce Wiebe. 2000. Effects of adjective orientation and gradability on sentence subjectivity. In Proc. of COLING. find
Marti Hearst. 1992. Direction-based text interpretation as an information access refinement. In Paul Jacobs, editor, Text-Based Intelligent Systems. Lawrence Erlbaum Associates. find
Alison Huettner and Pero Subasic. 2000. Fuzzy typing for document management. In ACL 2000 Companion Volume: Tutorial Abstracts and Demonstration Notes, pages 26–27. find
Thorsten Joachims. 1998. Text categorization with support vector machines: Learning with many relevant features. In Proc. of the European Conference on Machine Learning (ECML), pages 137– 142. find our card
Thorsten Joachims. 1999. Making large-scale SVM learning practical. In Bernhard Schölkopf and Alexander Smola, editors, Advances in Kernel Methods - Support Vector Learning, pages 44–56. MIT Press. find
Jussi Karlgren and Douglass Cutting. 1994. Recognizing text genres with simple metrics using discriminant analysis. In Proc. of COLING. find
Brett Kessler, Geoffrey Nunberg, and Hinrich Schütze. 1997. Automatic detection of text genre. In Proc. of the 35th ACL/8th EACL, pages 32–38. find
David D. Lewis. 1998. Naive (Bayes) at forty: The independence assumption in information retrieval. In Proc. of the European Conference on Machine Learning (ECML), pages 4–15. Invited talk. find
Andrew McCallum and Kamal Nigam. 1998. A comparison of event models for Naive Bayes text classification. In Proc. of the AAAI-98 Workshop on Learning for Text Categorization, pages 41–48. find
Frederick Mosteller and David L. Wallace. 1984. Applied Bayesian and Classical Inference: The Case of the Federalist Papers. Springer-Verlag. find
Kamal Nigam, John Lafferty, and Andrew McCallum. 1999. Using maximum entropy for text classification. In Proc. of the IJCAI-99 Workshop on Machine Learning for Information Filtering, pages 61–67. find
Ted Pedersen. 2001. A decision tree of bigrams is an accurate predictor of word sense. In Proc. of the Second NAACL, pages 79–86. find
Warren Sack. 1994. On the computation of point of view. In Proc. of the Twelfth AAAI, page 1488. Student abstract. find
Ellen Spertus. 1997. Smokey: Automatic recognition of hostile messages. In Proc. of Innovative Applications of Artificial Intelligence (IAAI), pages 1058–1065. find
Junichi Tatemura. 2000. Virtual reviewers for collaborative exploration of movie reviews. In Proc. of the 5th International Conference on Intelligent User Interfaces, pages 272–275. find
Loren Terveen, Will Hill, Brian Amento, David Mc-Donald, and Josh Creter. 1997. PHOAKS: A system for sharing recommendations. Communications of the ACM, 40(3):59–62. find
Laura Mayfield Tomokiyo and Rosie Jones. 2001. You’re not from round here, are you? Naive Bayes detection of non-native utterance text. In Proc. of the Second NAACL, pages 239–246. find
Richard M. Tong. 2001. An operational system for detecting and tracking opinions in on-line discussion. Workshop note, SIGIR 2001 Workshop on Operational Text Classification. find
Peter D. Turney and Michael L. Littman. 2002. Unsupervised learning of semantic orientation from a hundred-billion-word corpus. Technical Report EGB-1094, National Research Council Canada. find
Peter Turney. 2002. Thumbs up or thumbs down? Semantic orientation applied to unsupervised classification of reviews. In Proc. of the ACL. find our card
Janyce M. Wiebe, Theresa Wilson, and Matthew Bell. 2001. Identifying collocations for recognizing opinions. In Proc. of the ACL/EACL Workshop on Collocation. find
Yorick Wilks and Mark Stevenson. 1998. The grammar of sense: Using part-of-speech tags as a first step in semantic disambiguation. Journal of Natural Language Engineering, 4(2):135–144. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pang-2002-thumbs-up-sentiment-classification-machine/

📄 Thumbs up? Sentiment Classification using Machine Learning Techniques