Elamur Research⌕
☰
home / papers / 2002 / Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews

Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews

Summary

The paper addresses whether a review recommends or rejects its subject without any labeled training data. Turney proposes a three-stage unsupervised method. First, a Brill part-of-speech tagger finds two-word phrases built around adjectives or adverbs. Second, each phrase gets a semantic orientation score from PMI-IR: the phrase's association with "excellent" minus its association with "poor," estimated from AltaVista hit counts using the NEAR operator. Third, the review is labeled thumbs up if the average phrase score is positive and thumbs down otherwise. The evaluation uses 410 Epinions reviews, each by a different author, from four domains: automobiles, banks, movies and travel destinations. The method reaches 74% average accuracy, against a 59% majority-class baseline. Accuracy is highest on cars (84%) and lowest on movies (66%), where the overall verdict is often not simply the sum of the opinions about individual elements. The work matters because it shows that web-scale co-occurrence statistics can stand in for annotated corpora in sentiment classification. It also scores whole phrases, not just isolated adjectives.

Key points

  • Data. The test set has 410 Epinions reviews: 240 recommended and 170 not recommended, with labels given by the review authors themselves.
  • Phrase extraction. Five tag patterns (JJ, NN/NNS, RB/RBR/RBS, VB-family) select two-word phrases. Some patterns also restrict the following word. Proper nouns (NNP/NNPS) are excluded so that product names cannot affect the result.
  • Search setup. AltaVista indexed about 350 million English pages, and its NEAR operator matches words within ten words of each other in either order. Earlier work found NEAR to be a better association signal than AND. PMI-IR had scored 74% on 80 TOEFL synonym questions, compared with 64% for LSA.
  • Heuristics. Hit counts are smoothed by adding 0.01. A phrase is skipped if both of its co-occurrence counts are below 4. Every query excludes the Epinions host.
  • Cost. The reviews produced 10,658 phrases. With caching and a five-second delay between queries, processing the full set took roughly 30 hours.

Interesting ideas

  • I1. Anchor polarity to the rating scale. The seed words "excellent" and "poor" were chosen because they conventionally label five stars and one star. Choosing seeds that match how the target labels are defined could carry over to other languages or label scales.
  • I2. Score short phrases, not single words. Pairing an adjective or adverb with a neighboring word lets context decide polarity. For example, "unpredictable" is bad for steering but good for a plot. This is a cheap way to handle domain-dependent sentiment in any lexicon-based system.
  • I3. Block leakage from the test source. Excluding the evaluation website from every corpus query keeps unsupervised statistics from seeing the test data. This check is worth copying whenever web or crawled corpora supply features.
  • I4. Cache the constant queries. The seed-word counts are identical for every phrase, so caching them cut the cost to just under two queries per phrase. The same approach suits any rate-limited external resource.
  • I5. Use author-written pros and cons as a free reference. Turney proposes evaluating a pro/con summarizer by how well its most extreme phrases overlap with the pros and cons reviewers already supply. This turns existing site metadata into evaluation data with no extra annotation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/1073083.1073153

@inproceedings{turney2002thumbs,
  title = {{Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews}},
  author = {Turney, Peter},
  booktitle = {Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics},
  year = {2002},
  doi = {10.3115/1073083.1073153},
  url = {https://doi.org/10.3115/1073083.1073153}
}

References

14 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agresti, A. 1996. An introduction to categorical data analysis. New York: Wiley. find
  2. Brill, E. 1994. Some advances in transformation-based part of speech tagging. Proceedings of the Twelfth National Conference on Artificial Intelligence (pp. 722-727). Menlo Park, CA: AAAI Press. find
  3. Church, K.W., & Hanks, P. 1989. Word association norms, mutual information and lexicography. Proceedings of the 27th Annual Conference of the ACL (pp. 76-83). New Brunswick, NJ: ACL. find our card
  4. Frank, E., & Hall, M. 2001. A simple approach to ordinal classification. Proceedings of the Twelfth European Conference on Machine Learning (pp. 145-156). Berlin: Springer-Verlag. find
  5. Hatzivassiloglou, V., & McKeown, K.R. 1997. Predicting the semantic orientation of adjectives. Proceedings of the 35th Annual Meeting of the ACL and the 8th Conference of the European Chapter of the ACL (pp. 174-181). New Brunswick, NJ: ACL. find
  6. Hatzivassiloglou, V., & Wiebe, J.M. 2000. Effects of adjective orientation and gradability on sentence subjectivity. Proceedings of 18th International Conference on Computational Linguistics. New Brunswick, NJ: ACL. find
  7. Hearst, M.A. 1992. Direction-based text interpretation as an information access refinement. In P. Jacobs (Ed.), Text-Based Intelligent Systems: Current Research and Practice in Information Extraction and Retrieval. Mahwah, NJ: Lawrence Erlbaum Associates. find
  8. Landauer, T.K., & Dumais, S.T. 1997. A solution to Plato’s problem: The latent semantic analysis theory of the acquisition, induction, and representation of knowledge. Psychological Review, 104, 211-240. find
  9. Santorini, B. 1995. Part-of-Speech Tagging Guidelines for the Penn Treebank Project (3rd revision, 2nd printing). Technical Report, Department of Computer and Information Science, University of Pennsylvania. find
  10. Spertus, E. 1997. Smokey: Automatic recognition of hostile messages. Proceedings of the Conference on Innovative Applications of Artificial Intelligence (pp. 1058-1065). Menlo Park, CA: AAAI Press. find
  11. Tong, R.M. 2001. An operational system for detecting and tracking opinions in on-line discussions. Working Notes of the ACM SIGIR 2001 Workshop on Operational Text Classification (pp. 1-6). New York, NY: ACM. find
  12. Turney, P.D. 2001. Mining the Web for synonyms: PMI-IR versus LSA on TOEFL. Proceedings of the Twelfth European Conference on Machine Learning (pp. 491-502). Berlin: Springer-Verlag. find
  13. Wiebe, J.M. 2000. Learning subjective adjectives from corpora. Proceedings of the 17th National Conference on Artificial Intelligence. Menlo Park, CA: AAAI Press. find
  14. Wiebe, J.M., Bruce, R., Bell, M., Martin, M., & Wilson, T. 2001. A corpus study of evaluative and speculative language. Proceedings of the Second ACL SIG on Dialogue Workshop on Discourse and Dialogue. Aalborg, Denmark. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/turney-2002-thumbs-up-or-thumbs-down/