Elamur Research
☰
home / papers / 2016 / How Challenging is Sarcasm versus Irony Classification?: A Study With a Dataset from English Literature

How Challenging is Sarcasm versus Irony Classification?: A Study With a Dataset from English Literature

What it is about

The paper asks whether sarcasm (irony with a target of ridicule, so it changes sentiment towards that target) can be told from plain irony, using 4,306 quotes from literature with user tags (753 sarcasm, 677 irony, 2,876 philosophy). Three experienced annotators labelled 501 of them: their agreement with the tags is lower for sarcasm vs irony than for sarcasm vs philosophy (for one annotator kappa 0.62 vs 0.65, the same trend for all three); confusions come from self-deprecation and from contempt expressed without positive words. An SVM with the incongruity features of Joshi et al. (2015) reaches average F 0.846 for sarcasm vs philosophy (0.80 class-balanced) but only 0.654 for sarcasm vs irony. The authors note that a sarcastic quote can become a sincere one by changing a single word, and vice versa.

Ideas

For Mova

languages/en/34-irony-and-figurative-language.md §2 (verbal irony and sarcasm), §10 (Mova's detector), §12 (figurative language in the tales). Code: prag/src/irony.rs. Measure: the minimal-pair gate (all pairs must flip).

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/U16-1013.pdf

Summary

The paper asks whether telling sarcasm apart from irony is harder than separating sarcasm from an unrelated category. The authors distinguish the two by ridicule: sarcasm mocks a target and therefore affects sentiment, while irony does not. Much prior work treats them as the same thing. The authors built a corpus of book quotes from Goodreads, labelled by the user tags sarcasm, irony and philosophy. They used it to compare the sarcasm-vs-irony task with a sarcasm-vs-philosophy control.

The comparison has two parts:

  • Human: three experienced annotators labelled a random sample of quotes, and their agreement with the original tags was measured.
  • Computational: an SVM was trained on features from earlier sarcasm-vs-non-sarcasm work.

In both parts, sarcasm-vs-irony turned out clearly harder. Humans agreed less with the gold tags on that pair, and the classifier's F-score was roughly 12–15 points lower. The authors conclude that standard sarcasm features do not capture the sarcasm–irony boundary. They argue for new features, possibly built on identifying the target of ridicule. The results also serve as a baseline for this task.

Key points

  • Dataset: 4,306 Goodreads quotes, each carrying exactly one of three tags: sarcasm (753), irony (677) or philosophy (2,876). It is available on request.
  • Annotation: 501 randomly chosen quotes were labelled by annotators A1–A3, each with over 8k hours of experience, using four options (the three tags plus "cannot say").
    • Cohen's Kappa with the gold tags was 0.624, 0.537 and 0.451 for sarcasm–irony.
    • It was 0.654, 0.615 and 0.578 for sarcasm–philosophy.
  • Classifier setup: LibSVM with default parameters and five-fold cross-validation, using the feature set of Joshi et al. (2015):
    • unigrams;
    • pragmatic cues;
    • implicit sentiment phrases (Riloff et al. 2013b);
    • explicit sentiment features.
  • Overall results (average F-score):
    • sarcasm vs irony: 65.4%;
    • sarcasm vs philosophy: 84.6%;
    • class-balanced sarcasm vs philosophy, with the philosophy class undersampled: 80%.
  • Sarcasm class alone: F-score of 67.2% against irony versus 80.2% in the balanced philosophy setting. This suggests the difficulty lies specifically in recognising sarcastic quotes.

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{joshi2016how,
  title = {{How Challenging is Sarcasm versus Irony Classification?: A Study With a Dataset from English Literature}},
  author = {Joshi, Aditya and Tripathi, Vaibhav and Bhattacharyya, Pushpak and Carman, Mark and Singh, Meghna and Saraswati, Jaya and Shukla, Rajita},
  booktitle = {Proceedings of the Australasian Language Technology Association Workshop 2016},
  year = {2016},
  url = {https://aclanthology.org/U16-1013/}
}

References

22 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rehan Akbani, Stephen Kwek, and Nathalie Japkow-icz, 2004. Machine Learning: ECML 2004: 15th European Conference on Machine Learning, Pisa, Italy, September 20-24, 2004. Proceedings, chapter Applying Support Vector Machines to Imbalanced Datasets, pages 39–50. Springer Berlin Heidelberg, Berlin, Heidelberg. find
  2. David Bamman and Noah A Smith. 2015. Contextualized sarcasm detection on twitter. find
  3. Konstantin Buschmeier, Philipp Cimiano, and Roman Klinger. 2014. An impact analysis of features in a classification approach to irony detection in product reviews. find
  4. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised recognition of sarcastic sentences in twitter and amazon. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, pages 107–116. Association for Computational Linguistics. find our card
  5. David K Elson and Kathleen McKeown. 2010. Automatic attribution of quoted speech in literary narrative. In AAAI. Citeseer. find
  6. Roberto González-Ibáñez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying sarcasm in twitter: A closer look. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Short Papers - Volume 2, HLT ’11, pages 581–586, Stroudsburg, PA, USA. Association for Computational Linguistics. find our card
  7. Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing context incongruity for sarcasm detection. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, volume 2, pages 757–762. find our card
  8. Aditya Joshi, Kevin Patel, Vaibhav Tripathi, Pushpak Bhattacharyya, and Mark J Carman. 2016. Are word embedding-based features useful for sarcasm detection? In EMNLP. find our card
  9. Roger J. Kreuz and Gina M. Caucci. 2007. Lexical influences on the perception of sarcasm. In Proceedings of the Workshop on Computational Approaches to Figurative Language, FigLanguages ’07, pages 1– 4, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  10. Christopher J Lee and Albert N Katz. 1998. The differential role of ridicule in sarcasm and irony. Metaphor and Symbol, 13(1):1–15. find
  11. Jennifer Ling and Roman Klinger. 2016. An empirical, quantitative analysis of the differences between sarcasm and irony. In Semantic Web: ESWC 2016 Satellite Events. find
  12. Stephanie Lukin and Marilyn Walker. 2013. Really? well. apparently bootstrapping improves the performance of sarcasm and nastiness classifiers for online dialogue. find
  13. Diana Maynard and Mark A Greenwood. 2014. Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. find our card
  14. Soujanya Poria, Erik Cambria, Devamanyu Hazarika, and Prateek Vij. 2016. A deeper look into sarcastic tweets using deep convolutional neural networks. arXiv preprint arXiv:1610.08815. ↗ our card
  15. Antonio Reyes, Paolo Rosso, and Davide Buscaldi. 2012. From humor recognition to irony detection: The figurative language of social media. Data Knowl. Eng., 74:1–12, April. find
  16. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalin-dra De Silva, Nathan Gilbert, and Ruihong Huang. 2013a. Sarcasm as contrast between a positive sentiment and negative situation. find our card
  17. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalin-dra De Silva, Nathan Gilbert, and Ruihong Huang. 2013b. Sarcasm as contrast between a positive sentiment and negative situation. In EMNLP, volume 13, pages 704–714. find our card
  18. Andrew Skabar and Khaled Abdalgader. 2010. Improving sentence similarity measurement by incorporating sentential word importance. In AI 2010: Advances in Artificial Intelligence, pages 466–475. Springer. find
  19. Anders Søgaard. 2012. Mining wisdom. NAACL-HLT 2012, page 54. find
  20. Yuchun Tang, Yan-Qing Zhang, Nitesh V Chawla, and Sven Krasser. 2009. Svms modeling for highly imbalanced classification. Systems, Man, and Cybernetics, Part B: Cybernetics, IEEE Transactions on, 39(1):281–288. find
  21. Joseph Tepperman, David R Traum, and Shrikanth Narayanan. 2006. ” yeah right”: sarcasm recognition for spoken dialogue systems. find
  22. Byron C Wallace and Laura Kertz Do Kook Choe. 2014. Humans require context to infer ironic intent (so computers probably do, too). find