Elamur Research⌕
☰
home / papers / 2021 / Perceived and Intended Sarcasm Detection with Graph Attention Networks

Perceived and Intended Sarcasm Detection with Graph Attention Networks

Paper · 2021 · Joan Plepi and Lucie Flek

Summary

The paper argues that sarcasm detection should model the relationship between a writer and their audience, not only the wording of a post. The authors use the SPIRS Twitter corpus (about 30K labeled tweets from roughly 19k users) and add more than 10M historical tweets from the same authors. Each author gets a user2vec vector built from their past posts, and each tweet gets a SentenceBERT vector. Users and tweets become nodes in one heterogeneous graph, linked by user interactions, by shared conversation threads, and by authorship. A Graph Attention Network refines these node vectors, and a small feed-forward classifier labels the tweet. With about 500K trainable parameters, the model reaches 84.2% F1, compared with 69.9% for a fine-tuned BERT baseline. A further analysis splits sarcasm into intended (the author flags it) and perceived (someone else flags it). The intended kind turns out to be much easier to recognize, which suggests that modeling the audience matters for future work.

Key points

  • Data: SPIRS (Shmueli et al., 2020), with 15,000 sarcastic tweets (10,000 self-reported, 5,000 perceived), 15,000 non-sarcastic, 10,000 oblivious and 9,156 elicit tweets, extended with over 10M past tweets for user embeddings.

  • Graph structure:

    • Users and tweets are nodes, with three edge types: user–user (quotes, mentions, replies), tweet–tweet (all tweets in one thread fully connected), and tweet–author.
    • The graph has 108K nodes, density 0.00002 and 32% label homophily.
    • Cue tweets are left out to avoid label leakage.

  • F1 scores, averaged over 10 seeds:

    ModelF1
    BERT69.9%
    BERT + user2vec73.4%
    BERT + tweet-tweet GAT70.1%
    BERT + user-only GAT76.1%
    Full user+tweet GAT84.2%
    Full GAT without elicit tweets82.0%
    Full GAT without oblivious tweets81.4%
    Full GAT with cue tweets added94.5%

  • Intended vs perceived: when the model must separate the two, perceived sarcasm reaches only 56% F1 versus 84% for self-reported, and the GAT gains 2.2% over BERT. Most errors push perceived cases into the more frequent self-reported class.

  • Attention analysis: edges from a tweet to its author receive the highest weights, while oblivious and elicit edges receive less. Thread context mostly helps when user-level priors are weak or inconsistent. Early GCN trials were weaker and less interpretable than GAT.

Interesting ideas

  • I1. Speakers and utterances can share one graph with typed edges (who wrote what, who talked to whom, which posts belong to one exchange), so a statement's reading is shaped by both its author and its discussion.
  • I2. Separating whether a statement was meant ironically from whether a listener took it that way gives two distinct targets with very different difficulty.
  • I3. Cue posts that openly name the label were removed from the input graph, and the paper shows how much they would inflate the score (94.5% vs 84.2%).
  • I4. Removing one type of context at a time (elicit, oblivious, thread edges) shows how much each source contributes.
  • I5. Learned attention weights over typed edges were read as an explanation of which context drove a decision, and GAT was chosen over GCN partly for interpretability.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): The irony and implicature module in prag could link each utterance to a speaker node carrying that speaker's earlier statements in the text, and flag likely irony when the literal content clashes with the speaker's established stance.
  • I2 (frames): Irony frames could store intent and uptake as separate attributes, so a fable reader can explain cases where a character misses another's sarcasm.
  • I3 (learn): Evaluation could include a matching leak-check control, rerunning with and without any input that restates the gold answer to measure how much of a gain is real.
  • I4 (learn): The same per-source ablation could be applied to the witness parsers or knowledge sources behind silver data, ranking which ones actually move accuracy.
  • I5 (world): When the world model resolves an entity's state from several facts, recording a weight per supporting edge would give a readable trace of which facts decided the answer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Philipps University of MarburgMarburg · Germany4 works · 225 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Sentiment & emotion · 608 works

Cite

Cite as Joan Plepi and Lucie Flek. 2021. Perceived and Intended Sarcasm Detection with Graph Attention Networks. Findings of the Association for Computational Linguistics: EMNLP 2021.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.findings-emnlp.408

@inproceedings{plepi2021perceived,
  title = {{Perceived and Intended Sarcasm Detection with Graph Attention Networks}},
  author = {Joan Plepi and Lucie Flek},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2021},
  year = {2021},
  doi = {10.18653/v1/2021.findings-emnlp.408},
  url = {https://doi.org/10.18653/v1/2021.findings-emnlp.408}
}

References

40 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Raymond W. Gibbs. 2000. Irony in talk among friends. Metaphor and Symbol, 15(1-2):5–27. ↗
  2. Silvio Amir, Byron C. Wallace, Hao Lyu, Paula Carvalho, and Mário J. Silva. 2016. Modelling context with user embeddings for sarcasm detection in social media. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 167–177, Berlin, Germany. Association for Computational Linguistics.
  3. Roberto González-Ibáñez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying sarcasm in Twitter: A closer look. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 581–586, Portland, Oregon, USA. Association for Computational Linguistics.
  4. Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015. ↗
  5. Aditya Grover and Jure Leskovec. 2016. Node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 855–864, New York, NY, USA. Association for Computing Machinery. find
  6. David Bamman and Noah Smith. 2015. Contextualized sarcasm detection on twitter. In Proceedings of the International AAAI Conference on Web and Social Media, volume 9. find
  7. Tanja Bänziger and Klaus R. Scherer. 2005. The role of intonation in emotional expressions. Speech Communication, 46(3):252–267. Quantitative Prosody Modelling for Natural Speech Description and Generation. ↗
  8. Paula Carvalho, Luís Sarmento, Mário J. Silva, and Eu-génio de Oliveira. 2009. Clues for detecting irony in user-generated contents: Oh...!! it’s "so easy";-). In Proceedings of the 1st International CIKM Workshop on Topic-Sentiment Analysis for Mass Opinion, TSA ’09, page 53–56, New York, NY, USA. Association for Computing Machinery. find
  9. Shantanu Chandra, Pushkar Mishra, Helen Yannakoudakis, Madhav Nimishakavi, Marzieh Saeidi, Devamanyu Hazarika, Soujanya Poria, Sruthi Gorantla, Erik Cambria, Roger Zimmermann, and Rada Mihalcea. 2018. CASCADE: Contextual sarcasm detection in online discussion forums. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1837–1848, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  10. Claire Hewson and Tom Buchanan. 2013. Ethics guidelines for internet-mediated research. The British Psychological Society. find
  11. Anupam Khattri, Aditya Joshi, Pushpak Bhattacharyya, and Mark Carman. 2015. Your sentiment precedes you: Using an author’s historical tweets to predict sarcasm. In Proceedings of the 6th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 25–30, Lisboa, Portugal. Association for Computational Linguistics. ↗
  12. Diederik P Kingma and Jimmy Ba. 2015. Adam, a method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR), volume 1412. ↗
  13. Corina Koolen and Andreas van Cranenburgh. 2017. These are not the stereotypes you are looking for: Bias and fairness in authorial gender attribution. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing, pages 12–22, Valencia, Spain. Association for Computational Linguistics. find
  14. Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32, ICML’14, page II–1188–II–1196. JMLR.org. ↗
  15. Alice E. Marwick and Danah Boyd. 2011. I tweet honestly, i tweet passionately: Twitter users, context collapse, and the imagined audience. New Media & Society, 13(1):114–133. ↗
  16. Binny Mathew, Ritam Dutt, Pawan Goyal, and Animesh Mukherjee. 2019. Spread of hate speech in online social media. In Proceedings of the 10th ACM Conference on Web Science, WebSci ’19, page 173–182, New York, NY, USA. Association for Computing Machinery. find
  17. Diana Maynard and Mark Greenwood. 2014. Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 4238–4243, Reykjavik, Iceland. European Language Resources Association (ELRA). ↗
  18. Rohan Mishra, Pradyumn Prakhar Sinha, Ramit Sawhney, Debanjan Mahata, Puneet Mathur, and Rajiv Ratn Shah. 2019. SNAP-BATNET: Cascading author profiling and social network graphs for suicide ideation detection on social media. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 147–156, Minneapolis, Minnesota. Association for Computational Linguistics. find
  19. Chris Norval and Tristan Henderson. 2017. Contextual consent: Ethical mining of social media for health research. CoRR, abs/1701.07765. ↗
  20. Silviu Oprea and Walid Magdy. 2019. Exploring author context for detecting intended vs perceived sarcasm. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2854–2859, Florence, Italy. Association for Computational Linguistics. ↗
  21. Jing Qian, Mai ElSherief, Elizabeth Belding, and William Yang Wang. 2018. Leveraging intra-user and inter-user representation learning for automated hate speech detection. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 118–123, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  22. Jiezhong Qiu, Yuxiao Dong, Hao Ma, Jian Li, Kuansan Wang, and Jie Tang. 2018. Network embedding as matrix factorization: Unifying deepwalk, line, pte, and node2vec. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM ’18, pages 459–467, New York, NY, USA. ACM. find
  23. Ashwin Rajadesingan, Reza Zafarani, and Huan Liu. 2015. Sarcasm detection on twitter: A behavioral modeling approach. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, WSDM ’15, page 97–106, New York, NY, USA. Association for Computing Machinery. find
  24. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics. ↗
  25. Antonio Reyes, Paolo Rosso, and Tony Veale. 2013. A multidimensional approach for detecting irony in twitter. Lang. Resour. Eval., 47(1):239–268. ↗
  26. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalindra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 704–714, Seattle, Washington, USA. Association for Computational Linguistics. ↗
  27. Patricia Rockwell and Evelyn M. Theriot. 2001. Culture, gender, and gender mix in encoders of sarcasm: A self-assessment analysis. Communication Research Reports, 18(1):44–52. ↗
  28. Laurie A Rudman and Peter Glick. 2008. The social psychology of gender: How power and intimacy shape gender relations. find
  29. Deven Santosh Shah, H. Andrew Schwartz, and Dirk Hovy. 2020. Predictive biases in natural language processing models: A conceptual framework and overview. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5248–5264, Online. Association for Computational Linguistics.
  30. Katie Shilton and Sheridan Sayles. 2016. " we aren’t all going to be on the same page about ethics": Ethical practices and challenges in research on digital and social media. In 2016 49th Hawaii International Conference on System Sciences (HICSS), pages 1909–1918. IEEE. find
  31. Boaz Shmueli, Lun-Wei Ku, and Soumya Ray. 2020. Reactive Supervision: A New Method for Collecting Sarcasm Data. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2553–2559, Online. Association for Computational Linguistics. ↗
  32. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958. find
  33. Yi Tay, Anh Tuan Luu, Siu Cheung Hui, and Jian Su. 2018. Reasoning with sarcasm by reading in-between. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1010–1020, Melbourne, Australia. Association for Computational Linguistics. ↗
  34. Leanne Townsend and Claire Wallace. 2016. Social media research: A guide to ethics. University of Aberdeen, 1:16. find
  35. Oren Tsur, Dmitry Davidov, and Ari Rappoport. 2010. Icwsm — a great catchy name: Semi-supervised recognition of sarcastic sentences in online product reviews. Proceedings of the International AAAI Conference on Web and Social Media, 4(1):162– 169. find
  36. Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11). find
  37. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6000–6010, Red Hook, NY, USA. Curran Associates Inc.
  38. Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph Attention Networks. International Conference on Learning Representations. Accepted as poster. ↗
  39. Byron C. Wallace, Do Kook Choe, Laura Kertz, and Eugene Charniak. 2014. Humans require context to infer ironic intent (so computers probably do, too). In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 512–516, Baltimore, Maryland. Association for Computational Linguistics. ↗
  40. Matthew L Williams, Pete Burnap, and Luke Sloan. 2017. Towards an ethical framework for publishing twitter data in social research: Taking into account users’ views, online context and algorithmic estimation. Sociology, 51(6):1149–1168. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/plepi-2021-perceived-intended-sarcasm-detection-graph-2021-findings-emnlp-408/