Elamur Research⌕
☰
home / papers / 2023 / Personalized Intended and Perceived Sarcasm Detection on Twitter

Personalized Intended and Perceived Sarcasm Detection on Twitter

Paper · 2023 · Joan Plepi, Magdalena Buski and Lucie Flek

Summary

Sarcasm is hard to detect because its meaning often depends on who is speaking and on how listeners react. Most earlier datasets also cover only one side: either what the author meant as sarcastic, or what readers took as sarcastic. This paper builds on reactive supervision (Shmueli et al., 2020), which finds sarcastic tweets through replies that say someone is "being sarcastic". The grammatical person of each reply is used to label the tweet as intended (1st person) or perceived (2nd/3rd person). The authors found that many such replies actually express doubt or ask for clarification. They therefore add a fifth, regex-based filter step that removes these false cues.

The result is a Twitter dataset of about 35K tweets from 2022. It comes with conversational context (eliciting and oblivious tweets) and roughly 89M historical tweets from the authors. Text-only models (SBERT) are compared with author-aware models and a GNN. Adding conversational context helps most with perceived sarcasm, which remains the harder class.

Key points

  • The extra filter dropped 26.6% of 100K collected cue tweets. On 500 hand-labelled cues (Fleiss' Kappa 0.94), the false-positive rate fell from 46.6% (4-step pipeline) to about 5% (5-step pipeline).
  • The dataset has 35,250 tweets: 12,574 intended (1st person), 3,295 perceived (2nd person), 846 perceived (3rd person) and 18,535 non-sarcastic. All are English tweets from January to November 2022, with retweets excluded.
  • Non-sarcastic tweets were sampled at random, excluding tweets with sarcasm-related words or hashtags. Author history totals about 88.7M tweets across about 33K authors.
  • Two binary tasks were evaluated: sarcastic vs. non-sarcastic, and intended vs. perceived. Models: SBERT, Priming, A-SBERT, AA and GNN, each also run with oblivious/eliciting ("o/e") context.
  • Perceived sarcasm is mislabelled as intended far more often than the reverse (e.g., 59.1% vs. 7.5% for SBERT on 3,343 test tweets). Context cuts this error noticeably (e.g., 38.3% for o/e AA). The GNN benefited least from context, gaining only 1.3%.

Interesting ideas

  • I1. The audience's reaction can serve as a free label: a reply that calls out sarcasm marks the earlier tweet, and the pronoun in that reply tells whether the label reflects intent or perception.
  • I2. Before trusting weak labels, the authors hand-labelled a small random sample and measured the false-positive rate of each pipeline variant, which made the value of the extra filter visible.
  • I3. The authors treat "unknown" (a reply asking for clarification) as its own annotation class rather than forcing it into yes/no, which exposed the main source of noise.
  • I4. The authors separate intended from perceived sarcasm, i.e. speaker meaning from hearer interpretation, and find the hearer side much harder without context.
  • I5. Tweets that triggered the sarcastic one (eliciting) and replies that missed it (oblivious) are kept as context, and this mostly helped the perceived class.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): The irony module in prag could mine conversation-like data, such as dialogues in tales, for reply cues that mark a previous utterance as non-literal, giving labelled irony examples tied to the speaker's perspective.
  • I2 (learn): Silver data agreed by witness parsers could be audited the same way, with a small gold sample per pipeline version reporting precision before and after each new filter or repair rule.
  • I3 (learn): Evaluation sets for pragmatics could include an explicit "undecidable from text" class, acting as a negative control so the system is not rewarded for confidently guessing irony.
  • I4 (world): The world of a text could store separate belief states for what a character means and what other characters take them to mean, so answers to questions about misunderstanding in fables can be explained.
  • I5 (frames): Speech-act and implicature decisions could look at the preceding and following turns, using another character's failure to react as evidence for or against an ironic reading.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Arts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Sentiment & emotion · 972 works

Cite

Cite as Joan Plepi, Magdalena Buski and Lucie Flek. 2023. Personalized Intended and Perceived Sarcasm Detection on Twitter. Proceedings of the 3rd Workshop on Computational Linguistics for the Political and Social Sciences.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{plepi2023personalized,
  title = {{Personalized Intended and Perceived Sarcasm Detection on Twitter}},
  author = {Joan Plepi, Magdalena Buski and Lucie Flek},
  booktitle = {Proceedings of the 3rd Workshop on Computational Linguistics for the Political and Social Sciences},
  year = {2023},
  url = {https://aclanthology.org/2023.cpss-1.2/}
}

References

39 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Brian P. Ackerman. 1982. Contextual integration and utterance interpretation: The ability of children and adults to interpret sarcastic. Child Development, Volume 53, pages 1075–1083. find
  2. Debanjan Ghosh, Avijit Vajpayee, and Smaranda Muresan. 2020. A report on the 2020 sarcasm detection shared task. arXiv preprint arXiv:2005.05814. ↗
  3. Silvio Amir, Byron C. Wallace, Hao Lyu, Paula Carvalho, and Mário J. Silva. 2016. Modelling context with user embeddings for sarcasm detection in social media. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 167–177, Berlin, Germany. Association for Computational Linguistics.
  4. Raymond W Gibbs. 1986. On the psycholinguistics of sarcasm. Journal of experimental psychology: general, 115(1):3. ↗
  5. David Bamman and Noah Smith. 2015. Contextualized sarcasm detection on twitter. In Proceedings of the International AAAI Conference on Web and Social Media, volume 9. find
  6. Roberto González-Ibánez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying sarcasm in twitter: a closer look. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 581–586.
  7. Francesco Barbieri, Horacio Saggion, and Francesco Ronzano. 2014. Modelling sarcasm in Twitter, a novel approach. In Proceedings of the 5th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 50–58, Baltimore, Maryland. Association for Computational Linguistics.
  8. Devamanyu Hazarika, Soujanya Poria, Sruthi Gorantla, Erik Cambria, Roger Zimmermann, and Rada Mihalcea. 2018. Cascade: Contextual sarcasm detection in online discussion forums. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1837–1848.
  9. Claire Hewson and Tom Buchanan. 2013. Ethics guidelines for internet-mediated research. The British Psychological Society. find
  10. Alexandru-Costin Băroiu and Stefan Trăusan-Matu. 2022. Automatic sarcasm detection: Systematic literature review. Information 2022, 13, 399. ↗
  11. Aditya Joshi, Pushpak Bhattacharyya, and Mark James Carman. 2016. Automatic sarcasm detection: A survey. ACM Computing Surveys (CSUR), 50:1 – 22. ↗
  12. Paula Carvalho, Luís Sarmento, Mário J Silva, and Eu-génio De Oliveira. 2009. Clues for detecting irony in user-generated contents: oh...!! it’s" so easy";-. In Proceedings of the 1st international CIKM workshop on Topic-sentiment analysis for mass opinion, pages 53–56. find
  13. Endri Kacupaj, Joan Plepi, Kuldeep Singh, Harsh Thakkar, Jens Lehmann, and Maria Maleshkova. 2021. Conversational question answering over knowledge graphs with transformer and graph attention networks. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 850–862, Online. Association for Computational Linguistics. find
  14. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010a. Semi-supervised recognition of sarcasm in twitter and amazon. In Proceedings of the fourteenth conference on computational natural language learning, pages 107–116.
  15. Sandeepa Kannangara. 2018. Mining twitter for fine-grained political opinion polarity classification, ideology detection and sarcasm detection. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, pages 751–752. find
  16. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010b. Semi-supervised recognition of sarcasm in Twitter and Amazon. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, pages 107–116, Uppsala, Sweden. Association for Computational Linguistics.
  17. Anupam Khattri, Aditya Joshi, Pushpak Bhattacharyya, and Mark Carman. 2015. Your sentiment precedes you: Using an author’s historical tweets to predict sarcasm. In Proceedings of the 6th workshop on computational approaches to subjectivity, sentiment and social media analysis, pages 25–30. ↗
  18. Jenny L Davis, Tony P Love, and Gemma Killen. 2018. Seriously funny: The political work of humor on social media. New Media & Society, 20(10):3898– 3916. ↗
  19. Mikhail Khodak, Nikunj Saunshi, and Kiran Vodrahalli. 2017. A large self-annotated corpus for sarcasm. arXiv preprint arXiv:1704.05579. ↗
  20. Joan Plepi, Béla Neuendorf, and Lucie Flek. 2022a. Unifying data perspectivism and personalization: An application to social norms. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7391–7402. Association for Computational Linguistics. find
  21. Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32, ICML’14, page II–1188–II–1196. JMLR.org. ↗
  22. Joan Plepi, Flora Sakketou, Henri-Jacques Geiss, and Lucie Flek. 2022b. Temporal graph analysis of misinformation spreaders in social media. In Proceedings of TextGraphs-16: Graph-based Methods for Natural Language Processing, pages 89–104, Gyeongju, Republic of Korea. Association for Computational Linguistics. find
  23. Pushkar Mishra, Marco Del Tredici, Helen Yannakoudakis, and Ekaterina Shutova. 2019a. Abusive Language Detection with Graph Convolutional Networks. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota. ↗
  24. Tomás Ptácek, Ivan Habernal, and Jun Hong. 2014. Sarcasm detection on czech and english twitter. In COL-ING 2014, pages 213–223.
  25. Rohan Mishra, Pradyumn Prakhar Sinha, Ramit Sawhney, Debanjan Mahata, Puneet Mathur, and Rajiv Ratn Shah. 2019b. SNAP-BATNET: Cascading author profiling and social network graphs for suicide ideation detection on social media. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 147–156, Minneapolis, Minnesota. Association for Computational Linguistics. find
  26. Ashwin Rajadesingan, Reza Zafarani, and Huan Liu. 2015. Sarcasm detection on twitter: A behavioral modeling approach. In Proceedings of the eighth ACM international conference on web search and data mining, pages 97–106. ↗
  27. Nils Reimers and Iryna Gurevych. 2019. entence- bert: Sentence embeddings using siamese bert- networks. In In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982– 3992. Association for Computational Linguistics. find
  28. Chris Norval and Tristan Henderson. 2017. Contextual consent: Ethical mining of social media for health research. CoRR, abs/1701.07765. ↗
  29. Antonio Reyes, Paolo Rosso, and Tony Veale. 2013. A multidimensional approach for detecting irony in twitter. Language resources and evaluation, 47(1):239–268. ↗
  30. Silviu Oprea and Walid Magdy. 2019. Exploring author context for detecting intended vs perceived sarcasm. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2854– 2859, Florence, Italy. Association for Computational Linguistics. ↗
  31. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalindra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 704–714.
  32. Silviu Oprea and Walid Magdy. 2020a. iSarcasm: A dataset of intended sarcasm. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1279–1289, Online. Association for Computational Linguistics.
  33. Patricia Rockwell and Evelyn M. Theriot. 2001. Culture, gender, and gender mix in encoders of sarcasm: A self-assessment analysis. Communication Research Reports, 18(1):44–52. ↗
  34. Silviu Vlad Oprea and Walid Magdy. 2020b. The effect of sociocultural variables on sarcasm communication online. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW1):1–22. ↗
  35. Laurie A Rudman and Peter Glick. 2008. The social psychology of gender: How power and intimacy shape gender relations. find
  36. Hongliang Pan, Zheng Lin, Peng Fu, Yatao Qi, and Weiping Wang. 2020. Modeling intra and inter-modality incongruity for multi-modal sarcasm detection. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1383–1392, Online. Association for Computational Linguistics. ↗
  37. Flora Sakketou, Joan Plepi, Riccardo Cervero, Henri Jacques Geiss, Paolo Rosso, and Lucie Flek. 2022. FACTOID: A new dataset for identifying misinformation spreaders and political bias. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3231–3241, Marseille, France. European Language Resources Association. find
  38. Joan Plepi and Lucie Flek. 2021. Perceived and intended sarcasm detection with graph attention networks. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4746–4753, Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  39. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. Regular expressions Boaz Shmueli, Lun-Wei Ku, and Soumya Ray. 2020. Reactive supervision: A new method for collecting sarcasm data. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2553–2559. r"not being sarcastic" r"not(\s*[A-Za-z,;’\\\/s@])* \s*sarcastic") r"(sarcastic)\s*(\?)+" r"wasn’t being sarcastic" Oren Tsur, Dmitry Davidov, and Ari Rappoport. 2010. Icwsm—a great catchy name: Semi-supervised recognition of sarcastic sentences in online product reviews. In Proceedings of the International AAAI Conference on Web and Social Media, volume 4. r"wasnt being sarcastic" r"wasn’t being sarcastic" r"was not being sarcastic" r"weren’t being sarcastic" r"weren’t being sarcastic" Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph Attention Networks. International Conference on Learning Representations. Accepted as poster. r"werent being sarcastic" r"were not being sarcastic" r"(sarcastic)\s*(\?)+" r"sarcastic\sor" r"hope(\s*[A-Za-z,;’\\s@])*\s*being sarcastic" Byron C. Wallace, Do Kook Choe, Laura Kertz, and Eugene Charniak. 2014. Humans require context to infer ironic intent (so computers probably do, too). In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 512–516, Baltimore, Maryland. Association for Computational Linguistics. r"hope(\s*[A-Za-z,;’\\s@])*\s*being(\s* A-Za-z,;’\\s@ )*\s*sarcastic" r"hope you’re being sarcastic" r"pray(\s*[A-Za-z,;’\\s@])*\s*being sarcastic" Changsong Wen, Guoli Jia, and Jufeng Yang. 2023. Dip: Dual incongruity perceiving network for sarcasm detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2540–2550. r"if(\s*[A-Za-z,;’\\s@])*\s*being sarcastic" r"sarcastic[A-Za-z,;’\\s@]*\s*correct" r"sarcastic\s*([A-Za-z,;’\\s@]\s)0,2 right" Nan Xu, Zhixiong Zeng, and Wenji Mao. 2020. Reasoning with multimodal sarcastic tweets via modeling cross-modality contrast and semantic association. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3777–3786, Online. Association for Computational Linguistics. r"are you being sarcastic" *Table 10: Compound regular expression used to filter tweets incorrectly identified as cue tweets.* A Meishan Zhang, Yue Zhang, and Guohong Fu. 2016. Tweet sarcasm detection using deep neural network. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: technical papers, pages 2449–2460. Regular Expressions Table 10, shows a list of curated regular expressions that we used to filter out false positive cue tweets. The main target class that was fixed from the regular expressions, was the perceived sarcasm, where the number of false positive rate was significantly reduced.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/plepi-2023-personalized-intended-perceived-sarcasm-detection/