Elamur Research⌕
☰
home / papers / 2010 / Robust Sentiment Detection on Twitter from Biased and Noisy Data

Robust Sentiment Detection on Twitter from Biased and Noisy Data

Summary

Barbosa and Feng address sentiment classification of tweets. Tweets are too short for the classic n-gram models built for long texts such as reviews and news, and hand-labelled training data is expensive. They use a two-stage pipeline: first separate subjective from objective tweets, then label subjective ones as positive or negative. For training data they crawled tweets with automatic sentiment labels from three public websites (Twendz, Twitter Sentiment, TweetFeel) and analysed how good, biased and complementary these labels are.

Instead of raw words, the features are abstract:

  • word meta-information: POS tags, plus prior subjectivity and polarity from the MPQA lexicon extended with slang;
  • tweet-specific syntax: retweets, hashtags, replies, links, punctuation, emoticons and capitalisation.

Their system, TwitterSA, combines classifiers trained on each source. It beats unigram baselines on polarity, and its advantage grows when training data is small, biased or noisy. The work matters because it shows that cheap, imperfect labels can produce a usable classifier when the features generalise beyond specific words.

Key points

  • Data collection: tweets were gathered over three weeks with the generic query "of".
    • Twendz: 254,081 tweets, 3 classes.
    • Twitter Sentiment: 79,696 tweets, 3 classes.
    • TweetFeel: 13,122 tweets, positive/negative only.
  • Label quality and bias:
    • Estimated polarity label quality was 0.77 (Twendz), 0.82 (Twitter Sentiment) and 0.89 (TweetFeel).
    • TweetFeel was heavily biased: 4 words cover 95% of its positive examples and 6 words cover 96% of its negatives.
  • Polarity error rates:
    • TwitterSA(maxconf): 18.7%.
    • Unigrams: 20.9%.
    • ReviewSA: 21.7%.
    • Unigrams-TS: 24.3%.
    • Majority voting was the weakest way to combine the per-source classifiers.
  • Small training sets: with 2,000 training tweets, polarity error was 46% for Unigrams versus 23.8% for TwitterSA.
  • Top features for subjectivity (by information gain): the most informative were positive-polarity words, the presence of a link, strongly subjective words, capitalised text and verbs.

Interesting ideas

  • I1.

    Treat each automatic labeller as a separate annotator. Measure each source's quality and bias, then combine them rather than trusting any single one. Different sources make different mistakes, so ensembles over weak labels can cancel out errors. This fits any project that relies on silver-standard data.

  • I2.

    Clean noisy training data with heuristics:

    • drop items on which the sources disagree;
    • keep only one message per user, so that prolific accounts posting ads or job offers do not flood the data;
    • remove tweets containing the strongest opinion words from the objective set.

    These rules are cheap and can transfer to other weakly supervised corpora.

  • I3.

    Prefer abstract features to lexical ones when data is scarce or skewed. Representing text via POS tags, lexicon polarity and formatting cues makes the model less sensitive to the narrow vocabulary of a biased source. This could help in low-resource languages such as Ukrainian, where labelled data is limited.

  • I4.

    Use entropy and word coverage to diagnose bias. Checking how few words cover most examples of a class quickly reveals a labelling source that keys on a handful of trigger words.

  • I5.

    Flip prior polarity after a preceding negation. This simple rule adds some compositional awareness to lexicon features without any parsing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Part-of-speech tagging

Cited by

87 papers of our library cite it (of the 140884 whose bibliographies we read); 8 of them have a page here, 79 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{barbosa2010robust,
  title = {{Robust Sentiment Detection on Twitter from Biased and Noisy Data}},
  author = {Barbosa, Luciano and Feng, Junlan},
  booktitle = {Coling 2010: Posters},
  year = {2010},
  url = {https://aclanthology.org/C10-2005/}
}

References

17 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Cohen, J. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37. find
  2. Glance, N., M. Hurst, K. Nigam, M. Siegler, R. Stockton, and T. Tomokiyo. 2005. Deriving marketing intelligence from online discussion. In Proceedings of the eleventh ACM SIGKDD, pages 419–428. ACM. find
  3. Go, A., R. Bhayani, and L. Huang. 2009. Twitter sentiment classification using distant supervision. Technical report, Stanford Digital Library Technologies Project. find
  4. Landis, J.R. and G.G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, pages 159–174. find
  5. LingPipe 3.9.1. http://alias-i.com/lingpipe . ↗
  6. Pang, B. and L. Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the ACL, volume 2004. find our card
  7. Pang, B., L. Lee, and S. Vaithyanathan. 2002. Thumbs up?: sentiment classification using machine learning techniques. In Proceedings of the ACL, pages 79–86. Association for Computational Linguistics. find our card
  8. Polikar, R. 2006. Ensemble based systems in decision making. IEEE Circuits and systems magazine, 6(3):21–45. find
  9. Prentice, S. and E. Huffman. 2008. Social Medias New Role In Emergency Management. Idaho National Laboratory, pages 1–5. find
  10. Riloff, E. and J. Wiebe. 2003. Learning extraction patterns for subjective expressions. In Proceedings of the 2003 conference on Empirical methods in natural language processing, pages 105–112. find our card
  11. Riloff, E., J. Wiebe, and T. Wilson. 2003. Learning subjective nouns using extraction pattern bootstrapping. In Proceedings of the 7th Conference on Natural Language Learning, pages 25–32. find our card
  12. Riloff, E., S. Patwardhan, and J. Wiebe. 2006. Feature subsumption for opinion analysis. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 440–448. Association for Computational Linguistics. find
  13. Sheng, V.S., F. Provost, and P.G. Ipeirotis. 2008. Get another label? Improving data quality and data mining using multiple, noisy labelers. In Proceeding of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 614– 622. ACM. find
  14. Wiebe, J. and E. Riloff. 2005. Creating subjective and objective sentence classifiers from unannotated texts. Computational Linguistics and Intelligent Text Processing, pages 486–497. find
  15. Wiebe, J.M., RF Brace, and T.P. O’Hara. 1999. Development and use of a gold-standard data set for subjectivity classifications. In Proceedings of the ACL, pages 246–253. Association for Computational Linguistics. find
  16. Wilson, T., J. Wiebe, and P. Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In EMNLP, page 354. Association for Computational Linguistics. find our card
  17. Witten, Ian H. and Eibe Frank. 2005. Data Mining: Practical machine learning tools and techniques. Morgan Kaufmann. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/barbosa-2010-robust-sentiment-detection-twitter-biased/