Elamur Research⌕
☰
home / papers / 2024 / Generalizable Sarcasm Detection is Just Around the Corner, of Course!

Generalizable Sarcasm Detection is Just Around the Corner, of Course!

Summary

Most sarcasm detection work trains and tests on a single dataset and assumes one narrow idea of sarcasm: saying the opposite of what is meant, usually to wound. Jang and Frassinelli ask whether such models still work on sarcasm from other sources, tones and labelling perspectives. They fine-tune pretrained language models such as BERT on four English corpora. These differ in who supplied the labels (authors or third-party annotators), in domain (online and social media versus offline conversation or dialogue) and in style (aggressive versus playful mocking). Each model is evaluated on its own dataset and on the other three.

One corpus, the Conversational Sarcasm Corpus (CSC), is new and publicly released. Prolific participants wrote replies to everyday situational prompts addressed to a friend or colleague.

The main findings:

  • Models score best on the data they were tuned on and drop sharply elsewhere.
  • Third-party labels are easier to learn than author labels.
  • CSC-trained models transfer best, even to other domains.

A LIWC-based analysis links these gaps to genuinely different styles of sarcasm. The authors argue the field needs a broader working definition.

Key points

  • Datasets compared:
    • MUStARD: 690 scenes from Friends and The Big Bang Theory, 345 sarcastic, low Kappa of 0.23/0.59.
    • Sarcasm Corpus V2: 9,386 sentences from online debate sites in the Internet Argument Corpus, half sarcastic, 9 annotators each.
    • iSarcasmEval: 3,801 tweets with author labels, 1,067 sarcastic.
    • CSC: the new corpus.
  • Motivation: earlier studies saw F-scores fall from 0.75 to 0.33 when models trained on third-party labels were tested on author-labelled data.
  • Intra-dataset results: fine-tuning on third-party labels consistently gave better performance than fine-tuning on author labels.
  • Cross-dataset results: most models generalized poorly. CSC-trained models achieved the highest cross-dataset generalizability.
  • Post-hoc LIWC analysis (17 categories):
    • Sarcasm Corpus V2 is rich in anger and negative emotion, swearwords and online register.
    • MUStARD leans toward family and drives vocabulary.
    • CSC leans toward agreement words and religious exclamations such as "oh my god."
    • iSarcasmEval had no correct predictions unique to models tuned on it.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.naacl-long.238 · arXiv 2404.06357

@inproceedings{jang2024generalizable,
  title = {{Generalizable Sarcasm Detection is Just Around the Corner, of Course!}},
  author = {Jang, Hyewon and Frassinelli, Diego},
  booktitle = {Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.238},
  eprint = {2404.06357},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2024.naacl-long.238}
}

References

55 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rob Abbott, Brian Ecker, Pranav Anand, and Marilyn Walker. 2016. Internet argument corpus 2.0: An SQL schema for dialogic social media and the corpora to go with it. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 4445–4452, Portorož, Slovenia. European Language Resources Association (ELRA). find
  2. Ibrahim Abu Farha, Silviu Vlad Oprea, Steven Wilson, and Walid Magdy. 2022. SemEval-2022 task 6: iSarcasmEval, intended sarcasm detection in English and Arabic. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 802–814, Seattle, United States. Association for Computational Linguistics. find our card
  3. Muhammad Abulaish and Ashraf Kamal. 2018. Self-Deprecating Sarcasm Detection: An Amalgamation of Rule-Based and Machine Learning Approach. In 2018 IEEE/WIC/ACM International Conference on Web Intelligence (WI), pages 574–579. IEEE. Event-place: Santiago. find
  4. Francesco Barbieri, Horacio Saggion, and Francesco Ronzano. 2014. Modelling sarcasm in twitter, a novel approach. In proceedings of the 5th workshop on computational approaches to subjectivity, sentiment and social media analysis, pages 50–58. find our card
  5. Alexandru-Costin Băroiu and S , tefan Trăus , an-Matu. 2022. Automatic sarcasm detection: Systematic literature review. Information, 13(8):399. find
  6. Andrea Bowes and Albert Katz. 2011. When Sarcasm Stings. Discourse Processes, 48(4):215–236. ↗
  7. Penelope Brown and Stephen Levinson. 1978. Universals in language usage: Politeness phenomena. In Esther Goody, editor, Questions and Politeness: Strategies in Social Interaction, pages 56–310. Cambridge University Press, Cambridge. find
  8. Yitao Cai, Huiyu Cai, and Xiaojun Wan. 2019. Multimodal sarcasm detection in Twitter with hierarchical fusion model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2506–2515, Florence, Italy. Association for Computational Linguistics. find our card
  9. Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. 2019. Towards Multimodal Sarcasm Detection (An _obviously_ Perfect Paper). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4619–4629. Association for Computational Linguistics. find our card
  10. Tuhin Chakrabarty, Arkadiy Saakyan, Debanjan Ghosh, and Smaranda Muresan. 2022. FLUTE: Figurative language understanding through textual explanations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7139–7159, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  11. Herbert L Colston. 1997. Salting a wound or sugaring a pill: The pragmatic functions of ironic criticism. Discourse processes, 23(1):25–45. ↗
  12. Sourav Das, Soumitra Ghosh, Anup Kumar Kolya, and Asif Ekbal. 2023. Un paralleled sarcasm: a framework of parallel deep lstms with cross activation functions towards detection and generation of sarcastic statements. Language Resources and Evaluation, 57(2):765–802. find
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  14. Shelly Dews, Joan Kaplan, and Ellen Winner. 1995. Why not say it directly? The social functions of irony. Discourse Processes, 19(3):347–367. find
  15. Elena Filatova. 2012. Irony and sarcasm: Corpus generation and analysis using crowdsourcing. In Lrec, pages 392–398. Citeseer. find our card
  16. Paula Fortuna, Juan Soler-Company, and Leo Wanner. 2021. How well do hate speech, toxicity, abusive and offensive language classification models generalize across datasets? Information Processing Management, 58(3):102524. ↗
  17. Jean E. Fox Tree, J. Trevor D’Arcey, Alicia A. Hammond, and Alina S. Larson. 2020. The sarchasm: Sarcasm production and identification in spontaneous conversation. Discourse Processes, 57(5-6):507– 533. ↗
  18. Simona Frenda, Alessandra Teresa Cignarella, Valerio Basile, Cristina Bosco, Viviana Patti, and Paolo Rosso. 2022. The unbearable hurtfulness of sarcasm. Expert Systems with Applications, 193:116398. ↗
  19. Aniruddha Ghosh and Tony Veale. 2017. Magnets for Sarcasm: Making Sarcasm Detection Timely, Contextual and Very Personal. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 482–491. Association for Computational Linguistics. Event-place: Copenhagen, Denmark. find our card
  20. Debanjan Ghosh, Alexander R. Fabbri, and Smaranda Muresan. 2018. Sarcasm Analysis Using Conversation Context. Computational Linguistics, 44(4):755– 792. ↗ our card
  21. Raymond W. Gibbs. 2000. Irony in Talk Among Friends. Metaphor and Symbol, 15(1-2):5–27. ↗
  22. Sam Glucksberg. 1995. Commentary on nonliteral language: Processing and use. Metaphor and Symbol, 10(1):47–57. ↗
  23. Herbert P Grice. 1975. Logic and conversation. In Speech acts, pages 41–58. Brill. ↗
  24. Alon Halevy, Peter Norvig, and Fernando Pereira. 2009. The unreasonable effectiveness of data. IEEE intelligent systems, 24(2):8–12. ↗
  25. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. CoRR, abs/2111.09543. ↗ our card
  26. Nikhil Jaiswal. 2020. Neural sarcasm detection using conversation context. In Proceedings of the second workshop on figurative language processing, pages 77–82. find our card
  27. Hyewon Jang, Bettina Braun, and Diego Frassinelli. 2023. Intended and perceived sarcasm between close friends: What triggers sarcasm and what gets conveyed? In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 45. find
  28. Julia Jorgensen. 1996. The functions of sarcastic irony in speech. Journal of Pragmatics, 26(5):613–634. ↗
  29. Aditya Joshi, Pushpak Bhattacharyya, and Mark J. Carman. 2017. Automatic Sarcasm Detection: A Survey. ACM Computing Surveys, 50(5):1–22. ↗ our card
  30. Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing Context Incongruity for Sarcasm Detection. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 757–762. Association for Computational Linguistics. Event-place: Beijing, China. find our card
  31. Thomas R Keenan and Kathleen Quigley. 1999. Do young children use echoic information in their comprehension of sarcastic speech? a test of echoic mention theory. British Journal of Developmental Psychology, 17(1):83–96. ↗
  32. Mikhail Khodak, Nikunj Saunshi, and Kiran Vodrahalli. 2018. A Large Self-Annotated Corpus for Sarcasm. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), page 6. European Language Resources Association (ELRA). find our card
  33. Roger J Kreuz and Sam Glucksberg. 1989. How to be sarcastic: The echoic reminder theory of verbal irony. Journal of experimental psychology: General, 118(4):374. ↗
  34. Jacqueline K. Matthews, Jeffrey T. Hancock, and Philip J. Dunham. 2006. The Roles of Politeness and Humor in the Asymmetry of Affect in Verbal Irony. Discourse Processes, 41(1):3–24. ↗
  35. Silviu Oprea and Walid Magdy. 2019. Exploring Author Context for Detecting Intended vs Perceived Sarcasm. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2854–2859, Florence, Italy. Association for Computational Linguistics. ↗ our card
  36. Silviu Oprea and Walid Magdy. 2020. iSarcasm: A Dataset of Intended Sarcasm. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online. Association for Computational Linguistics. find our card
  37. Silviu Oprea, Steven Wilson, and Walid Magdy. 2021. Chandler: An explainable sarcastic response generator. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 339–349, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗ our card
  38. Shereen Oraby, Vrindavan Harrison, Lena Reed, Ernesto Hernandez, Ellen Riloff, and Marilyn Walker. 2016. Creating and characterizing a diverse corpus of sarcasm in dialogue. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 31–41, Los Angeles. Association for Computational Linguistics. find our card
  39. Jordan Painter, Helen Treharne, and Diptesh Kanojia. 2022. Utilizing weak supervision to create S3D: A sarcasm annotated dataset. In Proceedings of the Fifth Workshop on Natural Language Processing and Computational Social Science (NLP+CSS), pages 197–206, Abu Dhabi, UAE. Association for Computational Linguistics. find
  40. Hongliang Pan, Zheng Lin, Peng Fu, Yatao Qi, and Weiping Wang. 2020. Modeling intra and inter-modality incongruity for multi-modal sarcasm detection. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1383–1392, Online. Association for Computational Linguistics. find our card
  41. James Pennebaker, Ryan Boyd, Kayla Jordan, and Kate Blackburn. 2015. The development and psychometric properties of LIWC2015. Technical report, University of Texas at Austin. ↗
  42. John S. Leggitt and Raymond W. Gibbs. 2000. Emotional Reactions to Verbal Irony. Discourse Processes, 29(1):1–24. ↗
  43. Penny M. Pexman and Kara M. Olineck. 2002. Does Sarcasm Always Sting? Investigating the Impact of Ironic Insults and Ironic Compliments. Discourse Processes, 33(3):199–217. find
  44. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  45. Tomáš Ptáček, Ivan Habernal, and Jun Hong. 2014. Sarcasm detection on czech and english twitter. In Proceedings of COLING 2014, the 25th international conference on computational linguistics: Technical papers, pages 213–223. find our card
  46. Anupama Ray, Shubham Mishra, Apoorva Nunna, and Pushpak Bhattacharyya. 2022. A multimodal corpus for emotion recognition in sarcasm. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6992–7003, Marseille, France. European Language Resources Association. find our card
  47. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalindra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as Contrast between a Positive Sentiment and Negative Situation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 704–714. Association for Computational Linguistics. find our card
  48. Dan Sperber and Deirdre Wilson. 1981. Irony and the use-mention distinction. find
  49. Dan Sperber and Deirdre Wilson. 1986. Relevance: Communication and cognition, volume 142. Citeseer. find
  50. Martin Sykora, Suzanne Elayan, and Thomas W Jackson. 2020. A qualitative analysis of sarcasm, irony and related# hashtags on twitter. Big Data & Society, 7(2):2053951720972735. ↗
  51. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. SemEval-2018 task 3: Irony detection in English tweets. In Proceedings of the 12th International Workshop on Semantic Evaluation, pages 39–50, New Orleans, Louisiana. Association for Computational Linguistics. ↗ our card
  52. Marilyn Walker, Jean Fox Tree, Pranav Anand, Rob Abbott, and Joseph King. 2012. A corpus for research on deliberation and debate. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 812– 817, Istanbul, Turkey. European Language Resources Association (ELRA). find our card
  53. Jennifer Woodland and Daniel Voyer. 2011. Context and Intonation in the Perception of Sarcasm. Metaphor and Symbol, 26(3):227–239. ↗
  54. Wenjie Yin and Arkaitz Zubiaga. 2021. Towards generalisable hate speech detection: a review on obstacles and solutions. PeerJ Computer Science, 7:e598. ↗
  55. Yazhou Zhang, Yaochen Liu, Qiuchi Li, Prayag Tiwari, Benyou Wang, Yuhua Li, Hari Mohan Pandey, Peng Zhang, and Dawei Song. 2021. Cfn: A complex-valued fuzzy network for sarcasm detection in conversations. IEEE Transactions on Fuzzy Systems, 29(12):3696–3710. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/jang-2024-generalizable-sarcasm-detection-is-just/