Elamur Research⌕
☰
home / papers / 2011 / Identifying Sarcasm in Twitter: A Closer Look

Identifying Sarcasm in Twitter: A Closer Look

Summary

This paper looks at automatic sarcasm detection on Twitter. Progress on the task has been held back by a shortage of reliably labelled, naturally occurring examples. To fix this, the authors build a corpus where the tweet authors mark sarcasm themselves with #sarcasm or #sarcastic. These tweets are compared with tweets tagged with plain positive or negative emotion hashtags. The data is cleaned automatically and then by hand, keeping only tweets where the hashtag ends the message. That leaves 900 English tweets per class. The authors then test two kinds of cues with SMO and logistic regression classifiers. Lexical cues are unigrams, LIWC categories, WordNet Affect, interjections and punctuation. Pragmatic cues are emoticons and whether the tweet replies to another user (ToUser). The systems are also compared with human judges. Telling sarcasm apart from plain sentiment turns out to be hard: the best three-way accuracy is about 57%, and the human judges do no better. The study matters for two reasons. It shows that labels from the authors make a more reliable gold standard than labels from outside coders. It also suggests that surface word features are not enough and that context and shared knowledge are needed.

Key points

  • Corpus: 2,700 tweets, 900 each for sarcastic, positive and negative, labelled by the authors' own hashtags. Retweets, URLs, non-English tweets and tweets where the hashtag was not at the end were removed.
  • Best classifier accuracies with 5-fold cross-validation: S-P-N 57.22% (SMO, unigrams); sarcastic vs. non-sarcastic 65.44%; S-P and S-N close to 70%; positive vs. negative 75.89%.
  • In the χ² ranking, the three pragmatic features (ToUser, smiley, frown) are among the top 10 most useful features. Sarcastic tweets have positive-emotion words like positive tweets and more negation like negative tweets.
  • Human judges did poorly. In the three-way task their mean accuracy was 62.59%, falling to 43.33% when only unanimous labels were counted, and agreement was moderate (Fleiss' κ 0.4788). In the two-way task the classifiers (up to 68.33%) beat the humans.
  • On a 100-tweet subset that all contain emoticons, humans reached 73% and SMO 71%. Judges said missing context and missing world knowledge were the main obstacles.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

121 papers of our library cite it (of the 140884 whose bibliographies we read); 58 of them have a page here, 63 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{gonzalezibanez2011identifying,
  title = {{Identifying Sarcasm in Twitter: A Closer Look}},
  author = {González-Ibáñez, Roberto and Muresan, Smaranda and Wacholder, Nina},
  booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies},
  year = {2011},
  url = {https://aclanthology.org/P11-2102/}
}

References

14 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Carvalho, P., Sarmento, S., Silva, M. J., and de Oliveira, E. 2009. Clues for detecting irony in user-generated contents: oh...!! it's "so easy" ;-). In Proceeding of the 1st international CIKM workshop on Topic-sentiment analysis for mass opinion (TSA '09). ACM, New York, NY, USA, 53-56. find
  2. Clark, H. and Gerrig, R. 1984. On the pretence theory of irony. Journal of Experimental Psychology: General, 113:121–126. D.C. find
  3. Davidov, D., Tsur, O., and Rappoport, A. 2010. Semi-Supervised Recognition of Sarcastic Sentences in Twitter and Amazon, Dmitry Proceeding of Computational Natural Language Learning (ACL-CoNLL). find
  4. Derks, D., Bos, A. E. R., and Grumbkow, J. V. 2008. Emoticons and Online Message Interpretation. Soc. Sci. Comput. Rev., 26(3), 379-388. find
  5. Gibbs, R. 1986. On the psycholinguistics of sarcasm. Journal of Experimental Psychology: General, 105:3–15. find
  6. Gibbs, R. W. and Colston H. L. eds. 2007. Irony in Language and Thought. Routledge (Taylor and Francis), New York. find
  7. Kreuz, R. J. and Glucksberg, S. 1989. How to be sarcastic: The echoic reminder theory of verbal irony. Journal of Experimental Psychology: General, 118:374-386. find
  8. Kreuz, R. J. and Caucci, G. M. 2007. Lexical influences on the perception of sarcasm. In Proceedings of the Workshop on Computational Approaches to Figurative Language (pp. 1-4). Rochester, New York: Association for Computational. find our card
  9. LIWC Inc. 2007. The LIWC application. Retrieved May 10, 2010, from http://www.liwc.net/liwcdescription.php. ↗
  10. Nigam, K. and Hurst, M. 2006. Towards a Robust Metric of Polarity. In Computing Attitude and Affect in Text: Theory and Applications (pp. 265-279). Retrieved February 22, 2010, from http://dx.doi.org/10.1007/1-4020-4102-0_20. ↗
  11. Pak, A. and Paroubek, P. 2010. Twitter as a Corpus for Sentiment Analysis and Opinion Mining, in 'Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC'10)' , European Language Resources Association (ELRA), Valletta, Malta Pang, B. and Lee, L. 2008. Opinion Mining and Sentiment Analysis. Now Publishers Inc, July. find our card
  12. Pennebaker, J.W., Francis, M.E., & Booth, R.J. (2001). Linguistic Inquiry and Word Count (LIWC): LIWC2001 (this includes the manual only). Mahwah, NJ: Erlbaum Publishers Strapparava, C. and Valitutti, A. 2004. Wordnet-affect: an affective extension of wordnet. In Proceedings of the 4th International Conference on Language Resources and Evaluation, Lisbon. find our card
  13. Tepperman, J., Traum, D., and Narayanan, S. 2006. Yeah right: Sarcasm recognition for spoken dialogue systems. In InterSpeech ICSLP, Pittsburgh, PA. find
  14. Utsumi, A. 2000. Verbal irony as implicit display of ironic environment: Distinguishing ironic utterances from nonirony. Journal of Pragmatics, 32(12):1777– 1806. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ibanez-2011-identifying-sarcasm-twitter-closer-look/