Elamur Research⌕
☰
home / papers / 2010 / Semi-Supervised Recognition of Sarcasm in Twitter and Amazon

Semi-Supervised Recognition of Sarcasm in Twitter and Amazon

Summary

Sarcasm says one thing and means another, so standard sentiment analysis often misreads it. This affects review summarization, dialogue systems, review ranking and brand monitoring. Davidov, Tsur and Rappoport apply SASI, a semi-supervised sarcasm identification algorithm, to two very different sources: 5.9 million short, informal tweets and 66,000 longer, structured Amazon product reviews. SASI starts from a small seed of sentences labeled on a 1–5 sarcasm scale. It automatically extracts surface patterns, which are sequences of high-frequency words with slots for content words. These patterns, plus a few punctuation-based features, are used to score sentences with a k-NN-style classifier. Partial pattern matches still contribute to the score, which makes the method more robust. Evaluated against crowd-annotated gold data, SASI clearly beats a strong baseline and also works when trained on one domain and tested on the other. Tweets turned out to be easier than reviews, even though reviews offer more structure. The study helped establish sarcasm detection as a practical NLP task. It also shows that user-added #sarcasm hashtags make an unreliable gold standard.

Key points

  • Data: 5.9M unique tweets (average 14.2 words; URLs, user mentions and hashtags replaced with meta tags) and 66,000 Amazon reviews of 120 books and electronics products (product, company, title and author names replaced with tags).
  • Gold standard: Mechanical Turk, with 3 annotators per sentence. F-score was 0.83 on Twitter and 0.78 on Amazon.
  • Patterns: each pattern has 2–6 high-frequency words and 1–6 content-word slots. Inexact matches get partial credit through two weights, α and γ (both 0.1). The learned feature space has more than 300 dimensions.
  • Punctuation: punctuation features were the weakest predictors. The exception was "...", which helped when combined with other features.
  • #sarcasm tag: tested against hashtag-labeled tweets, the model reached precision 0.727, recall 0.436 and F 0.545. Users often add the tag as a search anchor, misuse it, or rely on it to mark very subtle sarcasm.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

131 papers of our library cite it (of the 140673 whose bibliographies we read); 56 of them have a page here, 75 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{davidov2010semi,
  title = {{Semi-Supervised Recognition of Sarcasm in Twitter and Amazon}},
  author = {Davidov, Dmitry and Tsur, Oren and Rappoport, Ari},
  booktitle = {Proceedings of the Fourteenth Conference on Computational Natural Language Learning},
  year = {2010},
  url = {https://aclanthology.org/W10-2914/}
}

References

27 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. L. Brown. 1980. The pragmatics of verbal irony. In R. W. Shuy and A. Snukal, editors, Language use and the uses of language, pages 111–127. Georgetown University Press. find
  2. Clint Burfoot and Timothy Baldwin. 2009. Automatic satire detection: Are you having a laugh? In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, pages 161–164, Suntec, Singapore, August. Association for Computational Linguistics. find
  3. H. Clark and R. Gerrig. 1984. On the pretence theory of irony. Journal of Experimental Psychology: General, 113:121–126. find
  4. Cristian Danescu-Niculescu-Mizil, Gueorgi Kossinets, Jon Kleinberg, and Lillian Lee. 2009. How opinions are received by online communities: A case study on amazon.com helpfulness votes. Jun. ↗
  5. D. Davidov and A. Rappoport. 2006. Efficient unsupervised discovery of word categories using symmetric patterns and high frequency words. In COLING-ACL. find
  6. Macmillan English Dictionary. 2007. Macmillan English Dictionary. Macmillan Education, 2 edition. find
  7. Raymond W Gibbs and Herbert L. Colston, editors. 2007. Irony in Language and Thought. Routledge (Taylor and Francis), New York. find
  8. R. W. Gibbs and J. E. O’Brien. 1991. Psychological aspects of irony understanding. Journal of Pragmatics, 16:523–530. ↗
  9. R. Gibbs. 1986. On the psycholinguistics of sarcasm. Journal of Experimental Psychology: General, 105:3–15. ↗
  10. H. P. Grice. 1975. Logic and conversation. In Peter Cole and Jerry L. Morgan, editors, Syntax and semantics, volume 3. New York: Academic Press. ↗
  11. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In KDD ’04: Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 168–177, New York, NY, USA. ACM. find
  12. A. Katz. 2005. Discourse and social-cultural factors in understanding non literal language. In Colston H. and Katz A., editors, Figurative language comprehension: Social and cultural influences, pages 183– 208. Lawrence Erlbaum Associates. find
  13. Jingjing Liu, Yunbo Cao, Chin-Yew Lin, Yalou Huang, and Ming Zhou. 2007. Low-quality product review detection in opinion summarization. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 334–342. find
  14. Rada Mihalcea and Stephen G. Pulman. 2007. Characterizing humour: An exploration of features in humorous texts. In CICLing, pages 337–347. find
  15. Rada Mihalcea and Carlo Strapparava. 2005. Making computers laugh: Investigations in automatic humor recognition. pages 531–538, Vancouver, Canada. find
  16. D.C. Muecke. 1982. Irony and the ironic. Methuen, London, New York. find
  17. Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the ACL, pages 271–278. ↗
  18. Bo Pang and Lillian Lee. 2008. Opinion Mining and Sentiment Analysis. Now Publishers Inc, July. ↗
  19. Ana-Maria Popescu and Oren Etzioni. 2005. Extracting product features and opinions from reviews. In HLT ’05: Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, pages 339–346, Morristown, NJ, USA. Association for Computational Linguistics. find
  20. Frank Jr. Stingfellow. 1994. The Meaning of Irony. State University of NY, New York. find
  21. J. Tepperman, D. Traum, and S. Narayanan. 2006. Yeah right: Sarcasm recognition for spoken dialogue systems. In InterSpeech ICSLP, Pittsburgh, PA. find
  22. Oren Tsur and Ari Rappoport. 2009. Revrank: A fully unsupervised algorithm for selecting the most helpful book reviews. In International AAAI Conference on Weblogs and Social Media. find
  23. Oren Tsur, Dmitry Davidiv, and Ari Rappoport. 2010. Icwsm – a great catchy name: Semi-supervised recognition of sarcastic sentences in product reviews. In International AAAI Conference on Weblogs and Social Media. find
  24. Akira Utsumi. 1996. A unified theory of irony and its computational formalization. In COLING, pages 962–967. find our card
  25. Akira Utsumi. 2000. Verbal irony as implicit display of ironic environment: Distinguishing ironic utterances from nonirony. Journal of Pragmatics, 32(12):1777–1806. ↗
  26. Janyce Wiebe, Theresa Wilson, Rebecca Bruce, Matthew Bell, and Melanie Martin. 2004. Learning subjective language. Computational Linguistics, 30(3):277– 308, January. ↗
  27. D. Wilson and D. Sperber. 1992. On verbal irony. Lingua, 87:53–76. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/davidov-2010-semi-supervised-recognition-sarcasm-twitter/