Elamur Research⌕
☰
home / papers / 2005 / Seeing Stars: Exploiting Class Relationships for Sentiment Categorization with Respect to Rating Scales

Seeing Stars: Exploiting Class Relationships for Sentiment Categorization with Respect to Rating Scales

Summary

Earlier sentiment classification work mostly separated positive from negative reviews. This paper addresses rating inference: predicting where a review falls on a multi-point scale such as star ratings. In this setting, labels are ordered, and some labels are closer to each other than others. The authors first ran a small study in which two human judges compared pairs of reviews by the same critic. People reliably spotted large rating gaps but became less accurate as the gap shrank. This motivated a three-class version and a four-class version of the task. The authors then released the scale dataset v1.0, built from movie reviews by four individual critics, with explicit rating markers and objective sentences stripped out. They compared three SVM-based approaches: one-vs-all, ε-insensitive regression, and a metric-labeling meta-algorithm. The meta-algorithm adjusts a base classifier's output so that similar documents end up with nearby labels. When paired with a new item similarity based on the share of positive sentences, metric labeling often beat both SVM baselines significantly. The study shows that using label distances together with a task-specific similarity measure helps fine-grained ordinal text classification.

Key points

  • Human pilot study (Table 1):
    • Both judges were perfect when two reviews differed by 3 or more "notches." A notch is a half star on a 4–5 star scale, or 10 points on a 100-point scale.
    • At a one-notch difference, pooled accuracy fell to 55%. This is still well above the 33% random baseline, though the two judges differed considerably.
  • Corpora: The scale dataset v1.0 contains four single-author movie-review corpora with 1770, 902, 1307 and 1027 documents. Single-author corpora sidestep differences in how critics calibrate their scales.
  • Label schemes:
    • The three-class scheme is negative / middling / positive.
    • The four-class scheme came from a five-class one. In each corpus, one extreme class held only about 5% of the documents, so it was merged into its neighbouring class.
  • Shared base learner: All methods use SVMs from Joachims' SVMlight package, which keeps the comparisons fair.
    • One-vs-all uses no label relations.
    • Regression uses them implicitly.
    • Metric labeling (following Kleinberg and Tardos) encodes them explicitly through a distance metric on labels.
  • Main finding: The metric-labeling approach was strongest when combined with the positive-sentence-percentage similarity. The results partly carried over to mixed-author data.

Interesting ideas

  • I1. Relative judgments for human studies: Ask annotators which of two texts is more positive instead of asking for absolute scores. Varying the true rating gap between pairs lets one experiment test several scale granularities at once. This is a cheap way to decide how fine-grained a labeling scheme can realistically be before annotating a new corpus.
  • I2. Classifier-agnostic post-processing: Treat a classifier's per-label scores as input to an optimization that also rewards giving nearby labels to similar items. Because it wraps an existing model, the trick could be reused for any ordinal target, such as reading level, urgency or quality grades.
  • I3. Label metrics for non-scale categories: The authors suggest encoding structure among classes as distances. For example, neutral can be placed closer to both positive and negative than those two are to each other, and hierarchical label sets can be expressed the same way. This is useful when classes are not a clean scale but are not unrelated either.
  • I4. Task-specific item similarity: Rather than relying on generic word overlap, the authors defined document similarity through a sentiment-relevant property (the proportion of positive sentences). Choosing a similarity measure that reflects the target dimension can matter more than the learning algorithm itself.
  • I5. Controlling annotator calibration: Keeping each corpus to a single author removes noise from differing personal rating scales. Collecting a few labeled samples from each new author is a practical way to handle calibration in real systems.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

593 papers of our library cite it (of the 140884 whose bibliographies we read); 52 of them have a page here, 541 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/1219840.1219855

@inproceedings{pang2005seeing,
  title = {{Seeing Stars: Exploiting Class Relationships for Sentiment Categorization with Respect to Rating Scales}},
  author = {Pang, Bo and Lee, Lillian},
  booktitle = {Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05)},
  year = {2005},
  doi = {10.3115/1219840.1219855},
  url = {https://doi.org/10.3115/1219840.1219855}
}

References

30 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Atkeson, Christopher G., Andrew W. Moore, and Stefan Schaal. 1997. Locally weighted learning. Artificial Intelligence Review, 11(1):11–73. find
  2. Boykov, Yuri, Olga Veksler, and Ramin Zabih. 1999. Fast approximate energy minimization via graph cuts. In Proceedings of the International Conference on Computer Vision (ICCV), pages 377–384. Journal version in IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) 23(11):1222–1239,… find
  3. Collins-Thompson, Kevyn and Jamie Callan. 2004. A language modeling approach to predicting reading difficulty. In HLT-NAACL: Proceedings of the Main Conference, pages 193– 200. find
  4. Das, Sanjiv and Mike Chen. 2001. Yahoo! for Amazon: Extracting market sentiment from stock message boards. In Proceedings of the Asia Pacific Finance Association Annual Conference (APFA). find
  5. Dave, Kushal, Steve Lawrence, and David M. Pennock. 2003. Mining the peanut gallery: Opinion extraction and semantic classification of product reviews. In Proceedings of WWW, pages 519–528. find
  6. Engström, Charlotta. 2004. Topic dependence in sentiment classification. Master’s thesis, University of Cambridge. find
  7. Herbrich, Ralf, Thore Graepel, and Klaus Obermayer. 2000. Large margin rank boundaries for ordinal regression. In Alexander J. Smola, Peter L. Bartlett, Bernhard Schölkopf, and Dale Schuurmans, editors, Advances in Large Margin Classifiers, Neural Information Processing Systems. MIT Press, pages 11… find
  8. Horvitz, Eric, Andy Jacobs, and David Hovel. 1999. Attention-sensitive alerting. In Proceedings of the Conference on Uncertainty and Artificial Intelligence, pages 305–313. find
  9. Ishikawa, Hiroshi. 2003. Exact optimization for Markov random fields with convex priors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(10). find
  10. Ishikawa, Hiroshi and Davi Geiger. 1998. Occlusions, discontinuities, and epipolar lines in stereo. In Proceedings of the 5th European Conference on Computer Vision (ECCV), volume I, pages 232–248, London, UK. Springer-Verlag. find
  11. Joachims, Thorsten. 1999. Making large-scale SVM learning practical. In Bernhard Schölkopf and Alexander Smola, editors, Advances in Kernel Methods - Support Vector Learning. MIT Press, pages 44–56. find
  12. Kleinberg, Jon and Éva Tardos. 2002. Approximation algorithms for classification problems with pairwise relationships: Metric labeling and Markov random fields. Journal of the ACM, 49(5):616–639. find
  13. Koppel, Moshe and Jonathan Schler. 2005. The importance of neutral examples for learning sentiment. In Workshop on the Analysis of Informal and Formal Information Exchange during Negotiations (FINEXIN). find
  14. Liu, Hugo, Henry Lieberman, and Ted Selker. 2003. A model of textual affect sensing using real-world knowledge. In Proceedings of Intelligent User Interfaces (IUI), pages 125–132. find
  15. McCallum, Andrew. 1999. Multi-label text classification with a mixture model trained by EM. In AAAI Workshop on Text Learning. find
  16. McCullagh, Peter. 1980. Regression models for ordinal data. Journal of the Royal Statistical Society, 42(2):109–42. find
  17. Pang, Bo and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the ACL, pages 271–278. find our card
  18. Pang, Bo, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up? Sentiment classification using machine learning techniques. In Proceedings of EMNLP, pages 79–86. find our card
  19. Qu, Yan, James Shanahan, and Janyce Wiebe, editors. 2004. Proceedings of the AAAI Spring Symposium on Exploring Attitude and Affect in Text: Theories and Applications. AAAI Press. AAAI technical report SS-04-07. find
  20. Rifkin, Ryan M. and Aldebaro Klautau. 2004. In defense of one-vs-all classification. Journal of Machine Learning Research, 5:101–141. find
  21. Schapire, Robert E. and Yoram Singer. 2000. BoosTexter: A boosting-based system for text categorization. Machine Learning, 39(2/3):135–168. find
  22. Smola, Alex J. and Bernhard Schölkopf. 1998. A tutorial on support vector regression. Technical Report Neuro-COLT NC-TR-98-030, Royal Holloway College, University of London. find
  23. Subasic, Pero and Alison Huettner. 2001. Affect analysis of text using fuzzy semantic typing. IEEE Transactions on Fuzzy Systems, 9(4):483–496. find
  24. Takamura, Hiroya, Yuji Matsumoto, and Hiroyasu Yamada. 2004. Modeling category structures with a kernel function. In Proceedings of CoNLL, pages 57–64. find
  25. Tong, Richard M. 2001. An operational system for detecting and tracking opinions in on-line discussion. SIGIR Workshop on Operational Text Classification. find
  26. Turney, Peter. 2002. Thumbs up or thumbs down? Semantic orientation applied to unsupervised classification of reviews. In Proceedings of the ACL, pages 417–424. find our card
  27. Vapnik, Vladimir. 1995. The Nature of Statistical Learning Theory. Springer. find
  28. Wilson, Theresa, Janyce Wiebe, and Rebecca Hwa. 2004. Just how mad are you? Finding strong and weak opinion clauses. In Proceedings of AAAI, pages 761–769. find
  29. Yu, Hong and Vasileios Hatzivassiloglou. 2003. Towards answering opinion questions: Separating facts from opinions and identifying the polarity of opinion sentences. In Proceedings of EMNLP. find our card
  30. Zhu, Xiaojin (Jerry). 2005. Semi-Supervised Learning with Graphs. Ph.D. thesis, Carnegie Mellon University. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pang-2005-seeing-stars-exploiting-class-relationships/