Elamur Research⌕
☰
home / papers / 2020 / Detecting Sarcasm in Conversation Context Using Transformer-Based Models

Detecting Sarcasm in Conversation Context Using Transformer-Based Models

Summary

The paper tackles sarcasm detection when a reply's sarcastic meaning depends on the preceding dialogue rather than on one isolated sentence. The authors used the Twitter and Reddit datasets from the FigLang2020 shared task. Each sample pairs a response with an ordered chain of earlier context turns, and both datasets are balanced between sarcastic and non-sarcastic labels. They cleaned the social-media text with these steps:

  • collapsing repeated punctuation while keeping punctuation marks,
  • converting emojis to text,
  • keeping hashtag words instead of deleting them,
  • expanding contractions.

Analysis of conversation lengths guided how many of the most recent context turns to concatenate with the response. They then fine-tuned bert-base-uncased and compared it with XLNet and several LSTM variants. BERT reached F1 of 0.752 on Twitter, using the last seven turns, and 0.621 on Reddit, using five. Both scores beat the shared-task baselines. The work shows that pretrained Transformers handle multi-turn context better than recurrent models. It also shows that choosing how much context to include is a meaningful design decision.

Key points

  • Baselines from Ghosh et al. (2018) were F1 0.67 for Twitter and 0.6 for Reddit. The authors' BERT models reached 0.752 (BERT-7, Twitter) and 0.621 (BERT-5, Reddit).
  • On Twitter, the LSTM, BiLSTM, Stacked-LSTM and Conv-LSTM models scored only 0.66–0.67. XLNet scored 0.684 on Twitter and 0.541 on Reddit.
  • Context length mattered:
    • Using all turns lowered BERT's scores to 0.724 on Twitter and 0.592 on Reddit.
    • Too few turns lost information.
    • Too many turns added heavy zero-padding.
    • On Reddit, the training and test sets had mismatched length distributions.
  • The authors kept hashtag content because tags such as #poor can flip a sentence's sentiment. They kept exclamation marks as cues of emotion.
  • The best leaderboard systems reached 0.93 (Twitter) and 0.83 (Reddit). The authors note that their own models do not generalize across datasets.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.figlang-1.15

@inproceedings{avvaru2020detecting,
  title = {{Detecting Sarcasm in Conversation Context Using Transformer-Based Models}},
  author = {Avvaru, Adithya and Vobilisetty, Sanath and Mamidi, Radhika},
  booktitle = {Proceedings of the Second Workshop on Figurative Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.figlang-1.15},
  url = {https://doi.org/10.18653/v1/2020.figlang-1.15}
}

References

21 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gavin Abercrombie and Dirk Hovy. 2016. Putting Sarcasm Detection into Context: The Effects of Class Imbalance and Manual Labelling on Supervised Machine Classification of Twitter Conversations. In Proceedings of the ACL 2016 Student Research Workshop, pages 107–113. find our card
  2. Silvio Amir, Byron C. Wallace, Hao Lyu, Paula Carvalho, and Mário J. Silva. 2016. Modelling context with user embeddings for sarcasm detection in social media. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 167–177, Berlin, Germany. Association for Computational Linguistics. find our card
  3. David Bamman and Noah A Smith. 2015. Contextualized Sarcasm Detection on Twitter. In Ninth International AAAI Conference on Web and Social Media. find
  4. Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-Supervised Recognition of Sarcastic Sentences in Twitter and Amazon. In Proceedings of the fourteenth conference on computational natural language learning, pages 107–116. Association for Computational Linguistics. find our card
  5. Elnaz Davoodi and Leila Kosseim. 2017. Automatic identification of AltLexes using monolingual parallel corpora. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, pages 195–200, Varna, Bulgaria. INCOMA Ltd. ↗
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805. ↗
  7. Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. arXiv preprint arXiv:1708.00524. ↗ our card
  8. Debanjan Ghosh, Alexander R Fabbri, and Smaranda Muresan. 2018. Sarcasm analysis using conversation context. Computational Linguistics, 44(4):755– 792. ↗ our card
  9. Debanjan Ghosh, Alexander Richard Fabbri, and Smaranda Muresan. 2017. The Role of Conversation Context for Sarcasm Detection in Online Interactions. arXiv preprint arXiv:1707.06226. ↗ our card
  10. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural computation, 9(8):1735–1780. ↗
  11. Aditya Joshi, Pushpak Bhattacharyya, and Mark J Carman. 2017. Automatic Sarcasm Detection: A Survey. ACM Computing Surveys (CSUR), 50(5):1–22. ↗ our card
  12. Yash Kumar Lal, Vaibhav Kumar, Mrinal Dhar, Manish Shrivastava, and Philipp Koehn. 2019. De-mixing sentiment from code-mixed text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 371–377, Florence, Italy. Association for Computational Linguistics. find
  13. Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. 2019. Towards Multimodal Sarcasm Detection (An Obviously Perfect Paper). arXiv preprint arXiv:1906.01815. ↗ our card
  14. Ping Liu, Wen Li, and Liang Zou. 2019. NULI at SemEval-2019 task 6: Transfer learning for offensive language detection using bidirectional transformers. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 87– 91, Minneapolis, Minnesota, USA. Association for Computational Linguistics. find
  15. P. Chaudhari and C. Chandankhede. 2017. Literature survey of sarcasm detection. In 2017 International Conference on Wireless Communications, Signal Processing and Networking (WiSPNET), pages 2041–2046. find
  16. Ashwin Rajadesingan, Reza Zafarani, and Huan Liu. 2015. Sarcasm Detection on Twitter: A Behavioral Modeling Approach. In Proceedings of the eighth ACM international conference on web search and data mining, pages 97–106. find
  17. Marco Antonio Calijorne Soares and Fernando Silva Parreiras. 2018. A literature review on question answering techniques, paradigms and systems. Journal of King Saud University-Computer and Information Sciences. ↗
  18. Byron C Wallace, Laura Kertz, Eugene Charniak, et al. 2014. Humans require context to infer ironic intent (so computers probably do, too). In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 512–516. find
  19. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized Autoregressive Pretraining for Language Understanding. In Advances in neural information processing systems, pages 5754–5764. ↗ our card
  20. Meishan Zhang, Yue Zhang, and Guohong Fu. 2016. Tweet sarcasm detection using deep neural network. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2449–2460, Osaka, Japan. find our card
  21. The COLING 2016 Organizing Committee. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/avvaru-2020-detecting-sarcasm-conversation-context-transformer/