Elamur Research⌕
☰
home / papers / 2018 / Detecting Sarcasm is Extremely Easy ;-)

Detecting Sarcasm is Extremely Easy ;-)

Paper · 2018 · Natalie Parde and Rodney Nielsen

Summary

Parde and Nielsen examine how a sarcasm classifier built to work across domains behaves on two very different kinds of text: short tweets and long Amazon product reviews. The model comes from their 2017 work. It was trained on 3998 tweets and 1003 reviews, and every instance gets the same hand-crafted feature set; features that cannot be computed, such as star rating for tweets, stay empty. The feature space is expanded with Daumé III's domain adaptation scheme, and Naïve Bayes does the classification. On both test sets the system matches or beats earlier results. The bigger contribution is a manual review of all 402 misclassified instances, grouped into error categories. False positives outnumber false negatives in both domains, mostly because of surface cues like heavy punctuation or mixed sentiment. Many false negatives can only be resolved with world knowledge. The authors also find noisy gold labels, since hashtag self-labeling is unreliable. They recommend commonsense and frame-semantic knowledge, careful text normalization, and sentiment/situation lexicons that are refreshed over time. The work matters because it shows where shallow cues stop working for sarcasm.

Key points

  • Test sets:
    • AMAZON has 251 reviews (87 sarcastic, 164 not), drawn from Filatova (2012).
    • TWITTER has 1000 tweets (391 sarcastic, 609 not). Labels come from #sarcasm versus emotion hashtags (#happiness, #sadness, etc.), and the hashtags were removed before use.
  • Results on the sarcastic class:
    • TWITTER: P 0.53, R 0.68, F1 0.59, against 0.58 F1 for their earlier Twitter-only model.
    • AMAZON: P 0.75, R 0.82, F1 0.78, against 0.74 F1 for Buschmeier et al. (2014).
  • Feature groups include:
    • Twitter indicators from Maynard & Greenwood.
    • Positive predicate / negative situation phrases from Riloff et al. (2013), including a ≤5-token ordering feature.
    • Star rating, laughter terms and interjections, punctuation marks.
    • Polarity, subjectivity and PMI.
    • Character repetition, all-caps counts, and bag-of-words.
  • Error distribution:
    • Amazon: 24 false positives, 16 false negatives.
    • Twitter: 235 false positives, 127 false negatives.
    • 109 tweet false positives had excessive punctuation, and 29 tweets plus 5 reviews were confused by mixed sentiment.
  • Label and data problems:
    • 23 tweets were not sarcastic without #sarcasm, 8 carried sarcasm only in other hashtags, and 13 merely talked about sarcasm.
    • Riloff's phrases barely appeared in this Twitter data.
    • 27 tweets without #sarcasm and 4 reviews labeled non-sarcastic were actually sarcastic.

Interesting ideas

  • I1. Daumé-style augmentation makes three copies of each feature (source-only, target-only, shared), so a simple learner can decide per feature whether a cue transfers across domains.
  • I2. A cheap but strong irony cue is a positive-sentiment predicate followed within a few tokens by a negative-situation phrase, i.e., a clash between evaluation and event.
  • I3. Many missed cases were sarcastic only because the described situation conflicts with common scripts or world facts; the authors suggest frame-semantic script violations and commonsense bases like NELL.
  • I4. Normalization should be selective: letter elongation ("sooooo") carries pragmatic meaning, while a transposition typo does not, so the two should not be collapsed alike.
  • I5. Checking every misclassified instance by hand exposed both model weaknesses and wrong gold labels (sarcastic items labeled negative, #sarcasm tweets that are not sarcastic).

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): The same trick could help train parsers or rule weights on a mix of treebank genres and silver data, keeping domain-specific signals apart from general ones.
  • I2 (frames): The irony module in prag could detect this clash structurally, as a positive evaluative frame whose theme is a negatively valued state, instead of over a token window.
  • I3 (world): Once the world model holds expected states for a scenario, an explicit contradiction between stated evaluation and expected outcome could become an explainable sarcasm signal passed up to prag.
  • I4 (tokens): The tokenizer could map typos to lemmas but keep elongation and compound-hashtag splitting as separate annotated attributes that later stages can read.
  • I5 (learn): Error audits on silver or self-labeled data should include a category for gold-label errors, so they do not get counted as model failures or turned into bad repair rules.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of North TexasDenton · United States18 works · 4,513 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Sentiment & emotion · 608 works

Cite

Cite as Natalie Parde and Rodney Nielsen. 2018. Detecting Sarcasm is Extremely Easy ;-). Proceedings of the Workshop on Computational Semantics beyond Events and Roles.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-1303

@inproceedings{parde2018detecting,
  title = {{Detecting Sarcasm is Extremely Easy ;-)}},
  author = {Natalie Parde and Rodney Nielsen},
  booktitle = {Proceedings of the Workshop on Computational Semantics beyond Events and Roles},
  year = {2018},
  doi = {10.18653/v1/w18-1303},
  url = {https://doi.org/10.18653/v1/w18-1303}
}

References

18 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Silvio Amir, Byron C Wallace, Hao Lyu, Paula Carvalho, and Mario J Silva. 2016. Modelling context with user embeddings for sarcasm detection in social media. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 167–177.
  2. Raquel Justo, Thomas Corcoran, Stephanie M. Lukin, Marilyn Walker, and M. Ins Torres. 2014. Extracting relevant knowledge for the detection of sarcasm and nastiness in the social web. Knowledge-Based Systems, 69:124 – 133. ↗
  3. David Bamman and Noah A Smith. 2015. Contextualized sarcasm detection on twitter. In Proceedings of the Ninth International AAAI Conference on Web and Social Media, pages 574–577. find
  4. Christine Liebrecht, Florian Kunneman, and Antal Van den Bosch. 2013. The perfect solution for detecting sarcasm in tweets #not. In Proceedings of the 4th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 29–37, Atlanta, Georgia. Association for Computational Linguistics.
  5. Clint Burfoot and Timothy Baldwin. 2009. Automatic satire detection: Are you having a laugh? In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, pages 161–164, Suntec, Singapore. Association for Computational Linguistics. find
  6. Diana Maynard and Mark Greenwood. 2014. Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In Proceedings of the Ninth International Conference on Language Resources and Evaluation, Reykjavik, Iceland. European Language Resources Association.
  7. Konstantin Buschmeier, Philipp Cimiano, and Roman Klinger. 2014. An impact analysis of features in a classification approach to irony detection in product reviews. In Proceedings of the 5th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 42–49, Baltimore, Maryland. Association for Computational Linguistics. ↗
  8. Tom Mitchell, William Cohen, Estevam Hruschka, Partha Talukdar, Justin Betteridge, Andrew Carlson, Bhavana D. Mishra, Matthew Gardner, Bryan Kisiel, Jayant Krishnamurthy, Ni Lao, Kathryn Mazaitis, Thahir Mohamed, Ndapa Nakashole, Emmanouil A. Platanios, Alan Ritter, Mehdi Samadi, Burr Settles, Richard Wang, Derry Wijaya, Abhinav Gupta, Xinlei Chen, Abulhair Saparov, Malcolm Greaves, and Joel Welling. 2015. Never-ending learning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI-15), pages 2302–2310. ↗
  9. Hal Daumé III. 2007. Frustratingly easy domain adaptation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 256–263, Prague, Czech Republic. Association for Computational Linguistics.
  10. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised recognition of sarcasm in twitter and amazon. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, pages 107–116, Uppsala, Sweden. Association for Computational Linguistics.
  11. Natalie Parde and Rodney Nielsen. 2017. #sarcas-mdetection is soooo general! towards a domain-independent approach for detecting sarcasm. In Proceedings of the 30th International Florida Artificial Intelligence Research Society Conference, pages 276–281. find
  12. Elena Filatova. 2012. Irony and sarcasm: Corpus generation and analysis using crowdsourcing. In Proceedings of the Eight International Conference on Language Resources and Evaluation, Istanbul, Turkey. European Language Resources Association.
  13. Ashwin Rajadesingan, Reza Zafarani, and Huan Liu. 2015. Sarcasm detection on twitter: A behavioral modeling approach. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, pages 97–106. ACM. find
  14. Aniruddha Ghosh and Tony Veale. 2016. Fracking sarcasm using neural network. In Proceedings of the 7th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 161–169, San Diego, California. Association for Computational Linguistics. ↗
  15. Antonio Reyes, Paolo Rosso, and Tony Veale. 2013. A multidimensional approach for detecting irony in twitter. Lang. Resour. Eval., 47(1):239–268. ↗
  16. Debanjan Ghosh, Weiwei Guo, and Smaranda Muresan. 2015. Sarcastic or not: Word embeddings to predict the literal or sarcastic meaning of words. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1003–1012, Lisbon, Portugal. Association for Computational Linguistics. ↗
  17. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalin-dra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 704–714, Seattle, Washington, USA. Association for Computational Linguistics.
  18. Roberto González-Ibáñez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying sarcasm in twitter: A closer look. In Proceedings of the 49th Annual Meeting of the Association for Computational

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/parde-2018-detecting-sarcasm-is-extremely-easy/