Elamur Research⌕
☰
home / papers / 2020 / A Report on the 2020 Sarcasm Detection Shared Task

A Report on the 2020 Sarcasm Detection Shared Task

Summary

This paper describes the sarcasm detection shared task held at FigLang 2020, co-located with ACL 2020. Its central question is how much prior conversation helps a classifier decide whether a reply is sarcastic. The task had two tracks built from self-labeled data. The Reddit track drew training material from the self-annotated Reddit Corpus of Khodak et al. (2017), where authors flag sarcasm with "/s". The Twitter track used tweets tagged #sarcasm or #sarcastic, contrasted with tweets carrying sentiment hashtags. Every instance came with at least two earlier dialogue turns. Systems were scored on CodaLab by F1 averaged over the sarcastic and non-sarcastic classes. Nearly all participants used Transformer models such as BERT and RoBERTa, most often by simply appending the context to the response. The winning team, miroblog, stood out because it combined ensembles with data augmentation. The organizers found that context beyond roughly three turns added almost nothing. The task provides a shared benchmark for context-aware sarcasm detection and shows that subtle humor still defeats strong pretrained models.

Key points

  • Data: Reddit training used 4,400 balanced posts. Twitter training used 5,000 balanced tweets. Each track had a balanced test set of 1,800 instances, and the Reddit test data was newly collected rather than taken from Khodak et al. (2017).
  • Participation: Reddit received 655 submissions from 39 participants, and Twitter received 1,070 submissions from 38 participants. Fourteen teams wrote system papers.
  • Winning approach (miroblog):
    • It ensembled models that used different context depths.
    • It created extra negative examples from context threads alone.
    • It pseudo-labeled large unlabeled dialogue threads using BERT next-sentence-prediction (NSP) confidence.
  • Other context strategies:
    • salokr summarized the full thread with a CNN-LSTM.
    • taha treated the immediate context as an aspect in an aspect-based sentiment model.
    • tanvidadu inserted explicit separators between context and response.
  • Error analysis: The top three systems all missed 80 Reddit instances (30 sarcastic) and 20 Twitter instances (10 sarcastic). Many of these involved niche topics or sarcasm accompanied by cues usually linked to non-sarcastic text, such as "haha" or :).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

35 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 30 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.figlang-1.1 · arXiv 2005.05814

@inproceedings{ghosh2020report,
  title = {{A Report on the 2020 Sarcasm Detection Shared Task}},
  author = {Ghosh, Debanjan and Vajpayee, Avijit and Muresan, Smaranda},
  booktitle = {Proceedings of the Second Workshop on Figurative Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.figlang-1.1},
  eprint = {2005.05814},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.figlang-1.1}
}

References

65 references, 41 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. kalaivani A and Thenmozhi D. 2020. Sarcasm identification and detection in conversion context using BERT. In Proceedings of the Second Workshop on Figurative Language Processing, Seattle, WA, USA. find our card
  2. Silvio Amir, Byron C Wallace, Hao Lyu, and Paula Carvalho Mário J Silva. 2016. Modelling context with user embeddings for sarcasm detection in social media. arXiv preprint arXiv:1607.00976. ↗ our card
  3. Taha Shangipour ataei, Soroush Javdan, and Behrouz Minaei-Bidgoli. 2020. Applying Transformers and aspect-based sentiment analysis approaches on sarcasm detection. In Proceedings of the Second Workshop on Figurative Language Processing, Seattle, WA, USA. find our card
  4. Adithya Avvaru, Sanath Vobilisetty, and Radhika Mamidi. 2020. Detecting sarcasm in conversation context using Transformer based model. In Proceedings of the Second Workshop on Figurative Language Processing, Seattle, WA, USA. find
  5. Aniruddha Ghosh and Tony Veale. 2016. Fracking sarcasm using neural network. In Proceedings of NAACL-HLT, pages 161–169. ↗ our card
  6. Arup Baruah, Kaushik Das, Ferdous Barbhuiya, and Kuntal Dey. 2020. Context-aware sarcasm detection using BERT. In Proceedings of the Second Workshop on Figurative Language Processing, Seattle, WA, USA. find our card
  7. Aniruddha Ghosh and Tony Veale. 2017. Magnets for sarcasm: Making sarcasm detection timely, contextual and very personal. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 482–491. Association for Computational Linguistics. find our card
  8. Yitao Cai, Huiyu Cai, and Xiaojun Wan. 2019. Multimodal sarcasm detection in twitter with hierarchical fusion model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2506–2515. find our card
  9. Debanjan Ghosh, Alexander R Fabbri, and Smaranda Muresan. 2018. Sarcasm analysis using conversation context. arXiv preprint arXiv:1808.07531. ↗ our card
  10. Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. 2019. Towards multimodal sarcasm detection (an obviously perfect paper). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4619–4629. find our card
  11. Debanjan Ghosh, Alexander Richard Fabbri, and Smaranda Muresan. 2017. The role of conversation context for sarcasm detection in online interactions. arXiv preprint arXiv:1707.06226. ↗ our card
  12. Debanjan Ghosh, Weiwei Guo, and Smaranda Muresan. 2015. Sarcastic or not: Word embeddings to predict the literal or sarcastic meaning of words. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1003– 1012, Lisbon, Portugal. Association for Computational Linguistics. find our card
  13. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175. ↗ our card
  14. Debanjan Ghosh and Smaranda Muresan. 2018. ” with 1 follower i must be awesome: P”. exploring the role of irony markers in irony recognition. arXiv preprint arXiv:1804.05253. ↗
  15. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364. ↗ our card
  16. Tanvi Dadu and Kartikey Pant. 2020. Sarcasm detection using context separators in online discourse. In Proceedings of the Second Workshop on Figurative Language Processing, Seattle, WA, USA. ↗ our card
  17. Debanjan Ghosh, Elena Musi, Kartikeya Upasani, and Smaranda Muresan. 2019. Interpreting verbal irony: Linguistic strategies and the connection to the type of semantic incongruity. arXiv preprint arXiv:1911.00891. ↗
  18. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised recognition of sarcastic sentences in twitter and amazon. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, CoNLL ’10. find our card
  19. Roberto González-Ibáñez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying sarcasm in twitter: A closer look. In ACL (Short Papers), pages 581–586. Association for Computational Linguistics. find our card
  20. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  21. Hunter Gregory, Steven Li, Pouya Mohammadi, Natalie Tarn, Rachel Ballantyne, and Cynthia Rudin. 2020. A Transformer approach to contextual sarcasm detection in twitter. In Proceedings of the Second Workshop on Figurative Language Processing, Seattle, WA, USA. find
  22. Xiangjue Dong, Changmao Li, and Jinho D. Choi. 2020. Transformer-based context-aware sarcasm detection in conversation threads from social media. In Proceedings of the Second Workshop on Figurative Language Processing, Seattle, WA, USA. find our card
  23. Devamanyu Hazarika, Soujanya Poria, Sruthi Gorantla, Erik Cambria, Roger Zimmermann, and Rada Mihalcea. 2018. Cascade: Contextual sarcasm detection in online discussion forums. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1837–1848. Association for Computational Linguistics. find our card
  24. Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1615–1625. Association for Computational Linguistics. find our card
  25. Nikhil Jaiswal. 2020. Neural sarcasm detection using conversation context. In Proceedings of the Second Workshop on Figurative Language Processing, Seattle, WA, USA. find our card
  26. Amit Kumar Jena, Aman Sinha, and Rohit Agarwal. 2020. C-net: Contextual network for sarcasm detection. In Proceedings of the Second Workshop on Figurative Language Processing, Seattle, WA, USA. of the Second Workshop on Figurative Language Processing, Seattle, WA, USA. find our card
  27. Chee Wee Leong, Beata Beigman Klebanov, and Ekaterina Shutova. 2018. A report on the 2018 vua metaphor detection shared task. In Proceedings of the Workshop on Figurative Language Processing, pages 56–66. find
  28. Aditya Joshi, Pushpak Bhattacharyya, and Mark J Carman. 2017. Automatic sarcasm detection: A survey. ACM Computing Surveys (CSUR), page 73. ↗ our card
  29. Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing context incongruity for sarcasm detection. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), volume 2, pages 757–762. find our card
  30. CC Liebrecht, FA Kunneman, and APJ van den Bosch. 2013. The perfect solution for detecting sarcasm in tweets# not. In Proceedings of the 4th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis. find our card
  31. Aditya Joshi, Vaibhav Tripathi, Pushpak Bhattacharyya, and Mark Carman. 2016. Harnessing sequence labeling for sarcasm detection in dialogue from tv series ‘friends’. CoNLL 2016, page 146. find our card
  32. Rongcheng Lin, Jing Xiao, and Jianping Fan. 2018. Nextvlad: An efficient neural network to aggregate frame-level features for large-scale video classification. In Proceedings of the European Conference on Computer Vision (ECCV), pages 0–0. find
  33. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64–77. ↗ our card
  34. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  35. Akshay Khatri and Pranav P. 2020. Sarcasm detection in tweets with BERT and GloVe embeddings. In Proceedings of the Second Workshop on Figurative Language Processing, Seattle, WA, USA. find
  36. Dehong Ma, Sujian Li, Xiaodong Zhang, and Houfeng Wang. 2017. Interactive attention networks for aspect-level sentiment classification. arXiv preprint arXiv:1709.00893. ↗
  37. Anupam Khattri, Aditya Joshi, Pushpak Bhattacharyya, and Mark Carman. 2015. Your sentiment precedes you: Using an author’s historical tweets to predict sarcasm. In Proceedings of the 6th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 25–30, Lisboa, Portugal. Association for Computational Linguistics. ↗ our card
  38. Navonil Majumder, Soujanya Poria, Haiyun Peng, Niyati Chhaya, Erik Cambria, and Alexander Gelbukh. 2019. Sentiment and sarcasm classification with multitask learning. IEEE Intelligent Systems, 34(3):38–43. ↗
  39. Diana Maynard and Mark A Greenwood. 2014. Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In Proceedings of LREC. find our card
  40. Mikhail Khodak, Nikunj Saunshi, and Kiran Vodrahalli. 2017. A large self-annotated corpus for sarcasm. arXiv preprint arXiv:1704.05579. ↗ our card
  41. Amardeep Kumar and Vivek Anand. 2020. Transformers on sarcasm detection with context. In Proceedings of the Second Workshop on Figurative Language Processing, Seattle, WA, USA. find our card
  42. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119. ↗
  43. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942. ↗
  44. Abhijit Mishra, Diptesh Kanojia, Seema Nagar, Kuntal Dey, and Pushpak Bhattacharyya. 2016. Harnessing cognitive features for sarcasm detection. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1095– 1104, Berlin, Germany. ↗ our card
  45. Hankyol Lee, Youngjae Yu, and Gunhee Kim. 2020. Augmenting data for sarcasm detection with unlabeled conversation context. In Proceedings of the Second Workshop on Figurative Language Processing, Seattle, WA, USA. find our card
  46. Smaranda Muresan, Roberto Gonzalez-Ibanez, Debanjan Ghosh, and Nina Wacholder. 2016. Identification of nonliteral language in social media: A case study on sarcasm. Journal of the Association for Information Science and Technology, 67(11):2725– 2737. ↗
  47. Jens Lemmens, Ben Burtenshaw, Ehsan Lotfi, Ilia Markov, and Walter Daelemans. 2020. Sarcasm detection using an ensemble approach. In Proceedings Tony Veale and Yanfen Hao. 2010. Detecting ironic intent in creative comparisons. In European Conference on Artificial Intelligence, volume 215, pages 765–770, Lisbon, Portugal. find
  48. Silviu Oprea and Walid Magdy. 2019. Exploring author context for detecting intended vs perceived sarcasm. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2854–2859. find our card
  49. Byron C Wallace. 2015. Computational irony: A survey and new perspectives. Artificial Intelligence Review, 43(4):467–483. find
  50. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014), 12. find
  51. Byron C Wallace, Do Kook Choe, Laura Kertz, and Eugene Charniak. 2014. Humans require context to infer ironic intent (so computers probably do, too). In ACL (2), pages 512–516. find
  52. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365. ↗
  53. Zelin Wang, Zhijian Wu, Ruimin Wang, and Yafeng Ren. 2015. Twitter sarcasm detection exploiting a context-based model. In International Conference on Web Information Systems Engineering, pages 77– 91, Miami, Florida. Springer. find
  54. Ashwin Rajadesingan, Reza Zafarani, and Huan Liu. 2015. Sarcasm detection on twitter: A behavioral modeling approach. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, pages 97–106. ACM. find
  55. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pages 5754–5764. ↗ our card
  56. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalindra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 704–714. find our card
  57. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of NAACL-HLT, pages 1480–1489. find our card
  58. Biqing Zeng, Heng Yang, Ruyang Xu, Wu Zhou, and Xuli Han. 2019. Lcf: A local context focus mechanism for aspect-based sentiment classification. Applied Sciences, 9(16):3389. ↗
  59. Rossano Schifanella, Paloma de Juan, Joel Tetreault, and Liangliang Cao. 2016. Detecting sarcasm in multimodal social platforms. In Proceedings of the 2016 ACM on Multimedia Conference, pages 1136– 1145. ACM. find our card
  60. Meishan Zhang, Yue Zhang, and Guohong Fu. 2016. Tweet sarcasm detection using deep neural network. In Proceedings of COLING 2016, The 26th International Conference on Computational Linguistics: Technical Papers, pages 2449–2460. find our card
  61. Youwei Song, Jiahai Wang, Tao Jiang, Zhiyue Liu, and Yanghui Rao. 2019. Attentional encoder network for targeted sentiment classification. arXiv preprint arXiv:1902.09314. ↗
  62. Himani Srivastava, Vaibhav Varshney, Surabhi Kumari, and Saurabh Srivastava. 2020. A novel hierarchical BERT architecture for sarcasm detection. In Proceedings of the Second Workshop on Figurative Language Processing, Seattle, WA, USA. find our card
  63. Yi Tay, Anh Tuan Luu, Siu Cheung Hui, and Jian Su. 2018. Reasoning with sarcasm by reading in-between. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1010–1020. Association for Computational Linguistics. find our card
  64. Oren Tsur, Dmitry Davidov, and Ari Rappoport. 2010. Icwsm-a great catchy name: Semi-supervised recognition of sarcastic sentences in online product reviews. In ICWSM. find
  65. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. Semeval-2018 task 3: Irony detection in english tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 39– 50. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ghosh-2020-report-2020-sarcasm-detection-shared/