Elamur Research⌕
☰
home / papers / 2018 / #SarcasmDetection is soooo general! Towards a Domain-Independent Approach for Detecting Sarcasm

#SarcasmDetection is soooo general! Towards a Domain-Independent Approach for Detecting Sarcasm

Paper · 2018 · Natalie Parde and Rodney D. Nielsen

Summary

Sarcasm detectors are usually tuned to a single source, most often Twitter, and they tend to fail when moved to text with different conventions. Parde and Nielsen build a feature set meant to work across domains. It is centred on sentiment polarity, subjectivity and PMI cues drawn from general lexicons. They compare it with Twitter-specific and Amazon-specific features, using a newly collected and publicly released Twitter corpus and Filatova's Amazon review sarcasm corpus. They test several training regimes: in-domain only, out-of-domain only, simple pooling, and pooling plus the EasyAdapt domain adaptation step. On their own, Twitter data is almost useless for classifying Amazon reviews. After EasyAdapt, however, the same tweets combined with a small Amazon set reach F1 0.780 on Amazon. This beats the baselines and cuts error by 14% relative to the previous best result on that corpus. The work matters because it shows that plentiful but mismatched data can help a target domain, provided the features generalize and the adaptation is explicit.

Key points

  • Twitter data was collected in February and March 2016:
    • tweets with a trailing #sarcasm were labeled sarcastic (1959);
    • tweets with a trailing Ekman-emotion hashtag (#happiness, #anger, etc.) were labeled non-sarcastic (3039);
    • the Amazon corpus has 437 sarcastic and 817 non-sarcastic reviews;
    • both datasets were split 80/20 into training and test sets.
  • Retweets, @replies and tweets with links were removed. Only tweets with exactly one target hashtag at the end were kept, and that hashtag was stripped before training.
  • The lexical resources were Liu05, MPQA, AFINN and Google Web1T. Five Weka classifiers were compared with 5-fold cross-validation, and Naïve Bayes performed best.
  • Results on Amazon test data:
    • Twitter-only training gave F1 0.276;
    • combined training with EasyAdapt gave F1 0.780, compared with baselines of 0.515 and 0.345;
    • error fell from 0.256 to 0.220 compared with the prior best (Buschmeier et al., F1 0.744).
  • Results on Twitter: in-domain F1 was 0.583, comparable to earlier reports of 0.51 and 0.545.
  • Among the general features, polarity and subjectivity features worked well and PMI features were weaker. The authors suspect hashtag word-joining and unusual acronyms hurt PMI.

Interesting ideas

  • I1. Pooling a large off-domain set with a small target set only paid off once an explicit domain adaptation step (EasyAdapt) was added; naive merging or source-only training failed, so the regime matters as much as data volume.
  • I2. The non-sarcastic class was built from emotionally charged texts, so the model cannot win by simply detecting strong sentiment; this is a built-in hard negative control.
  • I3. Labels were harvested only when the marker hashtag trailed the text, because a hashtag inside the body usually means the writer is talking about sarcasm rather than being sarcastic (mention versus use).
  • I4. Instead of learning tweet-specific positive-phrase/negative-situation pairs, the contrast between positive sentiment and a negative situation was approximated with general lexicons, keeping the core irony cue portable across domains.
  • I5. The error analysis found that whole reviews were labeled sarcastic although only one sentence was, and the most polar word often sat outside that sentence, which misleads document-level features.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when silver data from mismatched genres (news, tweets, fables) is mixed, mark each example's genre so training can separate shared from genre-specific patterns rather than letting abundant genres swamp the target.
  • I2 (learn): irony and evaluation tests should include sincere emotional sentences as negatives, so we can check that the irony detector is not just a sentiment detector in disguise.
  • I3 (frames): the speech-act layer could separate mentioning a stance word ("this is sarcasm") from performing it, which also helps when cheaply harvesting silver labels.
  • I4 (frames): an irony rule could compare the polarity of evaluative words with the state the world model assigns to the situation (e.g., "great" plus a broken item), giving an explainable irony signal.
  • I5 (answer): irony judgments should be tied to the specific clause, so that answers about a text's attitude point to the exact sarcastic span rather than the whole document.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of North TexasDenton · United States18 works · 4,513 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Psychology · 798 works Experimental and Cognitive Psychology · 463 works

🧭 Topics 1 Sentiment & emotion · 608 works

Cite

Cite as Natalie Parde and Rodney D. Nielsen. 2018. #SarcasmDetection is soooo general! Towards a Domain-Independent Approach for Detecting Sarcasm. arXiv:1806.03369.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1806.03369 · arXiv 1806.03369

@misc{parde2018sarcasmdetection,
  title = {{\#SarcasmDetection is soooo general! Towards a Domain-Independent Approach for Detecting Sarcasm}},
  author = {Natalie Parde and Rodney D. Nielsen},
  year = {2018},
  doi = {10.48550/arxiv.1806.03369},
  eprint = {1806.03369},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1806.03369}
}

References

24 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amir, S.; Wallace, B. C.; Lyu, H.; Carvalho, P.; and Silva, M. J. 2016. Modelling context with user embeddings for sarcasm detection in social media. In Proc. of CoNLL. our card
  2. Bamman, D., and Smith, N. 2015. Contextualized sarcasm detection on twitter. In Proc. of ICWSM. find
  3. Brants, T., and Franz, A. 2006. Web 1t 5-gram v1. find
  4. Burfoot, C., and Baldwin, T. 2009. Automatic satire detection: Are you having a laugh? In Proc. of ACL-IJCNLP. find
  5. Buschmeier, K.; Cimiano, P.; and Klinger, R. 2014. An impact analysis of features in a classification approach to irony detection in product reviews. In Proc. of WASSA. our card
  6. Daumé III, H. 2007. Frustratingly easy domain adaptation. In Proc. of ACL. our card
  7. Davidov, D.; Tsur, O.; and Rappoport, A. 2010. Semi-supervised recognition of sarcasm in twitter and amazon. In Proc. of CoNLL. our card
  8. Ekman, P. 1992. An argument for basic emotions. Cognition and Emotion 6(3-4). ↗ our card
  9. Filatova, E. 2012. Irony and sarcasm: Corpus generation and analysis using crowdsourcing. In Proc. of LREC. our card
  10. Ghosh, A., and Veale, T. 2016. Fracking sarcasm using neural network. In Proc. of WASSA. ↗ our card
  11. Ghosh, D.; Guo, W.; and Muresan, S. 2015. Sarcastic or not: Word embeddings to predict the literal or sarcastic meaning of words. In Proc. of EMNLP. [González-Ibáñez, Muresan, and Wacholder 2011] González-Ibáñez, R.; Muresan, S.; and Wacholder, N. 2011. Identifying sarcasm in twitter: A closer look. In Proc. of ACL-HLT. our card
  12. Hansen, L. K.; Arvidsson, A.; Nielsen, F. A.; Colleoni, E.; and Etter, M. 2011. Good friends, bad news - affect and virality in twitter. In Proc. of FutureTech. ↗
  13. Justo, R.; Corcoran, T.; Lukin, S.; Walker, M.; and Torres, M. 2014. Extracting relevant knowledge for the detection of sarcasm and nastiness in the social web. Knowl.-Based Sys. ↗
  14. Kipps, C.; Nestor, P.; Acosta-Cabronero, J.; Arnold, R.; and Hodges, J. 2009. Understanding social dysfunction in the behavioural variant of frontotemporal dementia: the role of emotion and sarcasm processing. Brain 132(3). [Liebrecht, Kunneman, and van den Bosch 2013] Liebrecht, C.; Kunneman, F.; and van den Bosch, A. 2013. The perfect solution for detecting sarcasm in tweets# not. In Proc. of WASSA. ↗ our card
  15. Liu, B.; Hu, M.; and Cheng, J. 2005. Opinion observer: Analyzing and comparing opinions on the web. In WWW’05. find
  16. Manning, C. D.; Surdeanu, M.; Bauer, J.; Finkel, J.; Bethard, S. J.; and McClosky, D. 2014. The Stanford CoreNLP natural language processing toolkit. In ACL Sys. Demos. our card
  17. Maynard, D., and Greenwood, M. 2014. Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In Proc. of LREC. our card
  18. Owoputi, O.; Dyer, C.; Gimpel, K.; Schneider, N.; and Smith, N. 2013. Improved part-of-speech tagging for online conversational text with word clusters. In Proc. of NAACL. our card
  19. Rajadesingan, A.; Zafarani, R.; and Liu, H. 2015. Sarcasm detection on twitter: A behavioral modeling approach. In Proc. of WSDM. find
  20. Reyes, A.; Rosso, P.; and Veale, T. 2013. A multidimensional approach for detecting irony in twitter. Language Resources and Evaluation 47(1). ↗
  21. Riloff, E.; Qadir, A.; Surve, P.; De Silva, L.; Gilbert, N.; and Huang, R. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In Proc. of EMNLP. our card
  22. Walker, M.; Tree, J. F.; Anand, P.; Abbott, R.; and King, J. 2012. A corpus for research on deliberation and debate. In Proc. of LREC. our card
  23. Wilson, T.; Wiebe, J.; and Hoffmann, P. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In Proc. of HLT-EMNLP. our card
  24. Yosinski, J.; Clune, J.; Bengio, Y.; and Lipson, H. 2014. How transferable are features in deep neural networks? In Proc. of NIPS. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/parde-2018-sarcasmdetection-soooo-general-towards-domain/