Elamur Research⌕
☰
home / papers / 2020 / Neural Sarcasm Detection using Conversation Context

Neural Sarcasm Detection using Conversation Context

Summary

This paper addresses sarcasm detection in Twitter conversations for the FigLang-2020 Sarcasm Shared Task. Its central question is how much preceding dialogue helps a classifier decide whether a reply is sarcastic. The author treats the problem as binary classification on the shared-task Twitter data (5,000 training and 1,800 test tweets). Inputs are built in five variants, ranging from the reply alone (RESP) to the reply preceded by one, two or three earlier turns (CON1–CON3) or by the full thread (CON). Four pretrained models are fine-tuned and compared: ELMo, USE, BERT LARGE and RoBERTa LARGE.

Context of limited length helps most models. Feeding the three most recent turns (CON3) gives the best results for BERT LARGE and RoBERTa LARGE. RoBERTa LARGE was fine-tuned twice with different initializations, and a majority-voting ensemble of these runs reaches F1 0.790. This placed the team (nclabj) second on the Twitter leaderboard. The study offers practical evidence that a short window of recent conversation is more useful than either no context or the entire thread.

Key points

  • Data: FigLang-2020 Twitter data with labels SARCASM / NOTSARCASM.
    • 10% of the training set was held out for validation.
    • Preprocessing removed usernames and split CamelCase hashtags into separate words.
  • Individual models: RoBERTa LARGE on CON3 was the strongest single model, with F1 0.772 (second run: 0.766).
    • BERT LARGE peaked at F1 0.737 on CON3.
    • USE scored best without context (RESP, 0.728).
    • ELMo trailed all others, at roughly 0.64–0.70.
  • Training setup:
    • RoBERTa LARGE was fine-tuned with fastai and HuggingFace Transformers, using gradual unfreezing, a 1cycle learning-rate policy and a learning rate of 1e-5.
    • ELMo, USE and BERT LARGE used TensorFlow Hub modules topped with dense layers sized 512, 256 and 1.
    • The decision threshold for these models was tuned on the validation set.
  • Best system: a majority vote over three RoBERTa LARGE models (IDs 4, 8, 9) reached Pr 0.790, Re 0.792 and F1 0.790, ranking 2nd in the shared task.
  • Reddit track: the same approach placed 8th with F1 0.702, though these results are not analysed in the paper.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

8 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 6 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.figlang-1.11

@inproceedings{jaiswal2020neural,
  title = {{Neural Sarcasm Detection using Conversation Context}},
  author = {Jaiswal, Nikhil},
  booktitle = {Proceedings of the Second Workshop on Figurative Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.figlang-1.11},
  url = {https://doi.org/10.18653/v1/2020.figlang-1.11}
}

References

25 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder. CoRR, abs/1803.11175. ↗ our card
  2. Jacob Devlin et al. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. find
  3. Aniruddha Ghosh and Tony Veale. 2016. Fracking sarcasm using neural network. ↗ our card
  4. Debanjan Ghosh, Alexander Richard Fabbri, and Smaranda Muresan. 2017. The role of conversation context for sarcasm detection in online interactions. CoRR, abs/1707.06226. 4238–4243, Reykjavik, Iceland. European Language Resources Association (ELRA). ↗ our card
  5. Debanjan Ghosh, Weiwei Guo, and Smaranda Muresan. 2015. Sarcastic or not: Word embeddings to predict the literal or sarcastic meaning of words. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1003– 1012, Lisbon, Portugal. Association for Computational Linguistics. find our card
  6. Matthew Peters, Mark Neumann, et al. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). find
  7. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalindra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 704–714, Seattle, Washington, USA. Association for Computational Linguistics. find our card
  8. Jeremy Howard and Sylvain Gugger. 2020. Fastai: A layered api for deep learning. Information, 11(2):108. ↗
  9. Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015. Deep unordered composition rivals syntactic methods for text classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1681–1691, Beijing, China. Association for Computational Linguistics. find
  10. Tim Rocktäschel, Edward Grefenstette, Karl Hermann, Tomáš Kočiský, and Phil Blunsom. 2015. Reasoning about entailment with neural attention. ↗ our card
  11. Rossano Schifanella, Paloma Juan, Joel Tetreault, and Liangliang Cao. 2016. Detecting sarcasm in multimodal social platforms. find our card
  12. Aditya Joshi, Pushpak Bhattacharyya, Mark Carman, and Vaibhav Tripathi. 2016a. Harnessing sequence labeling for sarcasm detection in dialogue from tv series ’friends’. find our card
  13. Leslie N. Smith. 2015. No more pesky learning rate guessing games. CoRR, abs/1506.01186. ↗
  14. Nitish Srivastava, Geoffrey E Hinton, et al. 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research. find
  15. Aditya Joshi, Pushpak Bhattacharyya, and Mark J. Carman. 2017. Automatic sarcasm detection: A survey. ACM Comput. Surv., 50(5). ↗ our card
  16. Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing context incongruity for sarcasm detection. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 757–762, Beijing, China. Association for Computational Linguistics. find our card
  17. Yu Sun, Shuohuan Wang, Yu-Kun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. ERNIE: enhanced representation through knowledge integration. CoRR, abs/1904.09223. ↗ our card
  18. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, undefine-dukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6000–6010, Red Hook, NY, USA. Curran Associates Inc. find
  19. Aditya Joshi, Vaibhav Tripathi, Kevin Patel, Pushpak Bhattacharyya, and Mark James Carman. 2016b. Are word embedding-based features useful for sarcasm detection? CoRR, abs/1610.00883. ↗ our card
  20. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. International Conference on Learning Representations. ↗
  21. Byron C. Wallace, Do Kook Choe, Laura Kertz, and Eugene Charniak. 2014. Humans require context to infer ironic intent (so computers probably do, too). In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 512–516, Baltimore, Maryland. Association for Computational Linguistics. find
  22. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. CoRR, abs/1901.07291. ↗
  23. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. ↗
  24. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtow-icz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ↗ our card
  25. Diana Maynard and Mark Greenwood. 2014. Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto. 2007. On early stopping in gradient descent learning. Constructive Approximation, 26:289–315. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jaiswal-2020-neural-sarcasm-detection-conversation-context/