Elamur Research⌕
☰
home / papers / 2014 / NRC-Canada-2014: Detecting Aspects and Sentiment in Customer Reviews

NRC-Canada-2014: Detecting Aspects and Sentiment in Customer Reviews

Summary

The paper describes the NRC-Canada entry to SemEval-2014 Task 4, Aspect Based Sentiment Analysis, which covers restaurant and laptop reviews. The task has four parts: finding explicit aspect terms, assigning one of five restaurant aspect categories, and judging the sentiment toward terms and toward categories. Sentiment is one of positive, negative, neutral or conflict.

Aspect terms are found with an in-house semi-Markov tagger. The other three subtasks use supervised SVM classifiers. A large share of the gains comes from resources built from unlabeled review collections: about 184K Yelp restaurant reviews and about 125K Amazon laptop reviews. From these, the authors derived PMI-based sentiment lexicons, a word–aspect association lexicon and Brown word clusters.

The system ranked first in aspect category detection and in category sentiment, and third in term extraction. For term sentiment it placed first on laptops and second on restaurants. The work shows that cheap, automatically induced in-domain lexicons can substantially improve fine-grained sentiment classifiers.

Key points

  • Unlabeled corpora:
    • The Yelp restaurant reviews corpus has 183,935 reviews from 58 food-related business categories.
    • The Amazon laptop reviews corpus has 124,712 reviews that mention "laptop" or "notebook".
    • Reviews with 1–2 stars were treated as negative and those with 4–5 stars as positive.
  • Aspect term extraction:
    • A semi-Markov tagger with O/T tags labels phrases of up to 5 tokens and is trained with structured Passive-Aggressive learning.
    • It reached F1 80.19 on restaurants and 68.57 on laptops, third of 24 teams in both domains.
    • Lower-cased back-off features mattered more than the choice of PA over the perceptron.
  • Aspect term polarity:
    • A linear SVM (LibSVM) uses surface, lexicon and dependency-parse features from Stanford CoreNLP.
    • Accuracy was 70.49 on laptops and 80.16 on restaurants.
    • Removing the sentiment lexicons cut laptop accuracy to 63.61.
  • Aspect category detection:
    • Five one-vs-all SVMs reached F1 88.58, first among 21 submissions.
    • The Word–Aspect Association Lexicon added 4.5 F1 points.
  • Aspect category polarity:
    • The system reached 82.93 accuracy, first among 23 submissions.
    • Lexical resources contributed 8.78 points, mostly through the in-domain Yelp sentiment lexicons.
    • Tweet lexicons and word clusters did not help on this subtask.

Interesting ideas

  • I1. Star ratings as distant labels: The authors treated review ratings as noisy polarity labels and scored each word by the difference of its PMI with positive and with negative reviews. This yields large in-domain lexicons without any annotation, and it suits any domain that has rated user content.
  • I2. Separate negated and affirmative lexicons: Words are scored separately depending on whether they appear between a negation word and the next punctuation mark. For example, "good" scores 1.2 normally but −1.4 when negated. This simple way to model negation carries over to lexicon building in other languages.
  • I3. Bootstrapped word–aspect lexicon: The trained category classifier labels the large unlabeled corpus, and PMI then links words to categories. This turns a small labeled set into a broad association resource.
  • I4. Feature duplication for multiple categories: Every feature gets a general copy and a category-specific copy, following Daumé III's domain adaptation trick. A single classifier can then share common sentiment cues while still learning category-specific ones, which is useful whenever classes or subdomains have little data each.
  • I5. Fallback for empty predictions: If no binary classifier fires for a sentence, it receives the most probable category when that probability is at least 0.4. This cheaply recovers recall in multi-label setups.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

91 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 84 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/s14-2076

@inproceedings{kiritchenko2014nrc,
  title = {{NRC-Canada-2014: Detecting Aspects and Sentiment in Customer Reviews}},
  author = {Kiritchenko, Svetlana and Zhu, Xiaodan and Cherry, Colin and Mohammad, Saif},
  booktitle = {Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)},
  year = {2014},
  doi = {10.3115/v1/s14-2076},
  url = {https://doi.org/10.3115/v1/s14-2076}
}

References

21 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Peter F Brown, Peter V Desouza, Robert L Mercer, Vincent J Della Pietra, and Jenifer C Lai. 1992. Class-based n-gram models of natural language. Computational linguistics, 18(4):467–479. find
  2. Chih-Chung Chang and Chih-Jen Lin. 2011. LIB-SVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2(3):27:1–27:27. find
  3. Michael Collins. 2002. Discriminative training methods for hidden markov models: Theory and experiments with perceptron algorithms. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find our card
  4. Koby Crammer and Yoram Singer. 2002. On the algorithmic implementation of multiclass kernel-based vector machines. The Journal of Machine Learning Research, 2:265–292. find
  5. Koby Crammer, Ofer Dekel, Joseph Keshet, Shai Shalev-Shwartz, and Yoram Singer. 2006. Online passive-aggressive algorithms. Journal of Machine Learning Research, 7:551–585. find
  6. Hal Daumé III. 2007. Frustratingly easy domain adaptation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, ACL ’07, pages 256 – 263. find our card
  7. Berry de Bruijn, Colin Cherry, Svetlana Kiritchenko, Joel Martin, and Xiaodan Zhu. 2011. Machine-learned solutions for three stages of clinical information extraction: the state of the art at i2b2 2010. Journal of the American Medical Informatics Association, 18(5):557–562. find
  8. Marie-Catherine de Marneffe, Bill MacCartney, and Christopher D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In Proceedings of the International Conference on Language Resources and Evaluation, LREC ’06. find our card
  9. Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A. Smith. 2011. Part-of-speech tagging for Twitter: Annotation, features, and experiments. In Proceedings of the Annual Meeting of the Associatio… find our card
  10. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’04, pages 168–177, New York, NY, USA. ACM. find
  11. Svetlana Kiritchenko, Xiaodan Zhu, and Saif M. Mohammad. 2014. Sentiment analysis of short informal texts. Journal of Artificial Intelligence Research (to appear). find
  12. Julian McAuley and Jure Leskovec. 2013. Hidden factors and hidden topics: understanding rating dimensions with review text. In Proceedings of the 7th ACM conference on Recommender systems, pages 165–172. ACM. find
  13. Saif M. Mohammad and Peter D. Turney. 2010. Emotions evoked by common words and phrases: Using Mechanical Turk to create an emotion lexicon. In Proceedings of the NAACL-HLT Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, LA, California. find our card
  14. Saif M. Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu. 2013. NRC-Canada: Building the state-of-the-art in sentiment analysis of tweets. In Proceedings of the International Workshop on Semantic Evaluation, SemEval ’13, Atlanta, Georgia, USA, June. find our card
  15. Olutobi Owoputi, Brendan OConnor, Chris Dyer, Kevin Gimpel, Nathan Schneider, and Noah A Smith. 2013. Improved part-of-speech tagging for online conversational text with word clusters. In Proceedings of NAACL-HLT, pages 380–390. find our card
  16. Maria Pontiki, Dimitrios Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. SemEval-2014 Task 4: Aspect based sentiment analysis. In Proceedings of the International Workshop on Semantic Evaluation, SemEval ’14, Dublin, Ireland, August. find our card
  17. M.F. Porter. 1980. An algorithm for suffix stripping. Program, 3:130–137. find
  18. Sunita Sarawagi and William W Cohen. 2004. Semi-markov conditional random fields for information extraction. In Advances in Neural Information Processing Systems, volume 17, pages 1185–1192. find
  19. Peter Turney and Michael L Littman. 2003. Measuring praise and criticism: Inference of semantic orientation from association. ACM Transactions on Information Systems, 21(4). find
  20. Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, HLT ’05, pages 347–354, Stroudsburg, PA, USA. find our card
  21. Xiaodan Zhu, Hongyu Guo, Saif M. Mohammad, and Svetlana Kiritchenko. 2014. An empirical study on the effect of negation words on sentiment. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, ACL ’14. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kiritchenko-2014-nrc-canada-2014-detecting-aspects/