Transformers on Sarcasm Detection with Context

Summary

This paper is a system description for the Sarcasm Detection with Context shared task at the Second Workshop on Figurative Language Processing (ACL 2020). It asks whether earlier turns of a social-media conversation help decide if a reply is sarcastic. The authors use two balanced English corpora: Twitter conversations (Ghosh et al., 2018) and Reddit threads (Khodak et al., 2018). They lightly clean the text and build the context by joining the earlier turns from newest to oldest. They then fine-tune BERT, RoBERTa and spanBERT in two setups: one where the reply and its context form a single input, and one where they are fed as a pair. They also test three custom designs:

an LSTM placed on top of the transformer's final hidden states
a Siamese Transformer
a Dual Transformer with a Bi-LSTM

The LSTM-over-Transformer model with roberta-large performs best on both corpora. Adding context clearly beats classifying the reply alone. The work matters because it shows that conversational context measurably improves sarcasm detection with pre-trained transformers.

Key points

Data: 5,000 tweets and 4,400 Reddit posts, each split evenly between SARCASM and NOT SARCASM. Every item comes in JSON with response, context and label fields.
Best model: LSTM over Transformer with roberta-large reached F1 77.22 on Twitter and 69.11 on Reddit.
Reply only, no context: the best F1 was 67.50 on Twitter and about 63.8 on Reddit. The paper's text gives 63.2 for Reddit, while its table shows 63.83.
Model comparisons: larger checkpoints consistently beat base ones. spanbert-large outperformed both BERT variants, and roberta-large was the strongest single-input baseline. The pair-input Siamese and Dual Transformer models did not beat the best model, possibly because of limited training data.
Setup: the best maximum sequence lengths were 126 tokens for Twitter and 80 for Reddit. Models were built with the Hugging Face transformers library. The authors made no comparison with prior systems, citing limited compute.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

6 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 5 not yet.

A Report on the 2020 Sarcasm Detection Shared Task

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.figlang-1.13

@inproceedings{kumar2020transformers,
  title = {{Transformers on Sarcasm Detection with Context}},
  author = {Kumar, Amardeep and Anand, Vivek},
  booktitle = {Proceedings of the Second Workshop on Figurative Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.figlang-1.13},
  url = {https://doi.org/10.18653/v1/2020.figlang-1.13}
}

References

12 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. *Table 2: Result on Reddit* Debanjan Ghosh, Alexander R. Fabbri, and Smaranda Muresan. 2018. Sarcasm analysis using conversation context. Computational Linguistics, 44(4):755– 792. find our card
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. ↗
Aditya Joshi, Vaibhav Tripathi, Pushpak Bhattacharyya, and Mark J. Carman. 2016. Harnessing sequence labeling for sarcasm detection in dialogue from TV series ‘Friends’. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 146–155, Berlin, Germany. Association for Computational Linguistics. find our card
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64–77. ↗ our card
Mikhail Khodak, Nikunj Saunshi, and Kiran Vodra-halli. 2018. A large self-annotated corpus for sarcasm. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018), Miyazaki, Japan. European Languages Resources Association (ELRA). find our card
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. ↗
Jonas Mueller and Aditya Thyagarajan. 2016. Siamese recurrent architectures for learning sentence similarity. In thirtieth AAAI conference on artificial intelligence. find
Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. IEEE transactions on Signal Processing, 45(11):2673–2681. ↗
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc. find
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtow-icz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. ↗ our card
Meishan Zhang, Yue Zhang, and Guohong Fu. 2016. Tweet sarcasm detection using deep neural network. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2449–2460, Osaka, Japan. find our card
The COLING 2016 Organizing Committee. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/kumar-2020-transformers-sarcasm-detection-context/

📄 Transformers on Sarcasm Detection with Context