Enhanced Sentiment Learning Using Twitter Hashtags and Smileys
Summary
The paper tackles fine-grained sentiment classification of very short texts. It moves beyond positive/negative polarity to dozens of mood and opinion types, and it avoids manual annotation. The authors treat Twitter users' own markup as free supervision: 50 sentiment-bearing hashtags and 15 ASCII smileys serve as class labels for tweets. Each tweet becomes one feature vector combining four feature groups: single words, 2–5-word n-grams, surface patterns built from high-frequency words with content-word slots, and simple punctuation and length statistics. A kNN-style classifier with outlier removal assigns the label. The authors measure each feature group's contribution and confirm that every group helps. Mechanical Turk judges agreed with the predicted labels for most untagged tweets, and agreement was higher for smileys than for hashtags. Two techniques, tag co-occurrence and feature overlap, are used to explore how sentiment types relate to each other. The work shows that noisy user-generated tags can replace costly hand-labelled sentiment lexicons or datasets.
Key points
- Labels come from 50 Twitter hashtags and 15 smileys. Tweets carrying more than one label are excluded from training to reduce ambiguity.
- Features:
- Words and n-grams are weighted by inverse corpus count.
- Features found in under 0.5% of training sentences are dropped.
- Patterns contain 2–6 HFWs and 1–5 CW slots and must begin and end with an HFW. The HFW/CW frequency thresholds are 100 and 1000 words per million.
- Partial pattern matches get reduced scores (α = γ = 0.1).
- Classifier: kNN with k = 10 and Euclidean distance. Only training vectors sharing at least one feature are compared. Neighbours farther than twice the mean distance are discarded before the majority vote.
- Human evaluation: Four MTurk judges rated each task set, and five control items filtered out unreliable workers.
- Smiley labels were judged correct 84% of the time, versus 77% for hashtags.
- Judges found no fitting tag in 6% (smileys) and 10% (hashtags) of cases.
- Inter-judge agreement was κ = 0.41.
- Tags with opposite meanings (e.g., #sad and #happy) co-occur surprisingly often. Feature-overlap statistics show the same trends as direct co-occurrence, though the shared-feature percentages are higher.
Interesting ideas
- I1. User markup as distant supervision. Hashtags, emoji, or forum tags can act as noisy labels for training a multi-class classifier cheaply. This transfers to any language or platform where such labels exist, including Ukrainian social media.
- I2. Graded partial pattern matching. Exact, sparse, and incomplete matches receive progressively smaller weights. This reduces sparsity for long patterns in short texts and could suit any short-text pattern classifier.
- I3. Staged feature selection. Frequent words are found first, then only n-grams built from them, then only patterns built from those. This keeps feature extraction affordable on large corpora.
- I4. Control questions in crowdsourcing. Each task set includes a few unambiguous gold items, and workers who miss them are excluded. This is a simple quality filter for any annotation campaign.
- I5. Feature overlap as a label-similarity measure. Label relatedness is estimated from shared features rather than direct co-occurrence. This works even when co-occurrence data is sparse and could support clustering of categories in other taxonomies.
Cited by
- Open Domain Targeted Sentiment
- Learning Sentiment-Specific Word Embedding for Twitter Sentiment Classification
Cite
@inproceedings{davidov2010enhanced,
title = {{Enhanced Sentiment Learning Using Twitter Hashtags and Smileys}},
author = {Davidov, Dmitry and Tsur, Oren and Rappoport, Ari},
booktitle = {Coling 2010: Posters},
year = {2010},
url = {https://aclanthology.org/C10-2028/}
}References
- Akkaya, Cem, Janyce Wiebe, and Rada Mihalcea. 2009. Subjectivity word sense disambiguation. In EMNLP. find
- Andreevskaia, A. and S. Bergler. 2006. Mining wordnet for fuzzy sentiment: Sentiment tag extraction from wordnet glosses. In EACL. find
- Balog, Krisztian, Gilad Mishne, and Maarten de Rijke. 2006. Why are they excited? identifying and explaining spikes in blog mood levels. In EACL. find
- Bloom, Kenneth, Navendu Garg, and Shlomo Argamon. 2007. Extracting appraisal expressions. In HLT/NAACL. find
- Davidov, D. and A. Rappoport. 2006. Efficient unsupervised discovery of word categories using symmetric patterns and high frequency words. In COLING-ACL. find
- Davidov, D. and A. Rappoport. 2008. Unsupervised discovery of generic relationships using pattern clusters and its evaluation by automatically generated sat analogy questions. In ACL. find
- Davidov, D., O. Tsur, and A. Rappoport. 2010. Semi-supervised recognition of sarcastic sentences in twitter and amazon. In CoNLL. find our card
- Esuli, Andrea and Fabrizio Sebastiani. 2006. Sentiwordnet: A publicly available lexical resource for opinion mining. In LREC. find our card
- Jansen, B.J., M. Zhang, K. Sobel, and A. Chowdury. 2009. Twitter power: Tweets as electronic word of mouth. Journal of the American Society for Information Science and Technology. find
- Kim, S.M. and E. Hovy. 2004. Determining the sentiment of opinions. In COLING. find our card
- McDonald, Ryan, Kerry Hannan, Tyler Neylon, Mike Wells, and Jeff Reynar. 2007. Structured models for fine-to-coarse sentiment analysis. In ACL. find
- Melville, Prem, Wojciech Gryc, and Richard D. Lawrence. 2009. Sentiment analysis of blogs by combining lexical knowledge with text classification. In KDD. ACM. find
- Mihalcea, Rada and Hugo Liu. 2006. A corpus-based approach to finding happiness. In In AAAI 2006 Symposium on Computational Approaches to Analysing Weblogs. AAAI Press. find
- Mishne, Gilad. 2005. Experiments with mood classification in blog posts. In Proceedings of the 1st Workshop on Stylistic Analysis Of Text. find
- Riloff, Ellen. 2003. Learning extraction patterns for subjective expressions. In EMNLP. find our card
- Strapparava, Carlo and Rada Mihalcea. 2008. Learning to identify emotions in text. In SAC. find
- Titov, Ivan and Ryan McDonald. 2008a. A joint model of text and aspect ratings for sentiment summarization. In ACL/HLT, June. find our card
- Titov, Ivan and Ryan McDonald. 2008b. Modeling online reviews with multi-grain topic models. In WWW, pages 111–120, New York, NY, USA. ACM. find
- Tsur, Oren, Dmitry Davidov, and Ari Rappoport. 2010. Icwsm – a great catchy name: Semi-supervised recognition of sarcastic sentences in product reviews. In AAAI-ICWSM. find
- Turney, Peter D. 2002. Thumbs up or thumbs down? semantic orientation applied to unsupervised classification of reviews. In ACL ’02, volume 40. find our card
- Whitelaw, Casey, Navendu Garg, and Shlomo Argamon. 2005. Using appraisal groups for sentiment analysis. In CIKM. find
- Wiebe, Janyce and Rada Mihalcea. 2006. Word sense and subjectivity. In COLING/ACL, Sydney, AUS. find
- Wiebe, Janyce M. 2000. Learning subjective adjectives from corpora. In AAAI. find
- Wilson, Theresa, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In HLT/EMNLP. find our card
- Wilson, Theresa, Janyce Wiebe, and Paul Hoffmann. 2009. Recognizing contextual polarity: An exploration of features for phrase-level sentiment analysis. Computational Linguistics, 35(3):399–433. find our card
- Yu, Hong and Vasileios Hatzivassiloglou. 2003. Towards answering opinion questions: Separating facts from opinions and identifying the polarity of opinion sentences. In EMNLP. find our card
This page in Markdown → https://md.research.elamur.ai/free/en/papers/davidov-2010-enhanced-sentiment-learning-twitter-hashtags/