An Evaluation of State-of-the-Art Large Language Models for Sarcasm Detection
Paper · 2023 · Juliann Zhou
Summary
The paper asks how well two strong but very different sarcasm detectors perform when run side by side on the same Reddit data. Sarcasm reverses the literal meaning of an utterance, so models that judge sentiment from the text alone often miss it. The author reimplements CASCADE (Hazarika et al., 2018), which adds context to a CNN text encoder. That context is a user embedding built from personality and writing-style features, plus a discourse vector that describes the forum. The second model is RCNN-RoBERTa (Potamias et al., 2020), which places a recurrent-convolutional layer and a BiLSTM on top of pretrained RoBERTa. Both are tested on the r/politics subsample of SARC (Khodak et al., 2017) and compared against several baselines and a human score. RCNN-RoBERTa performs best, CASCADE comes second, and both beat the baselines. The models still fall short of human accuracy. The paper concludes that author and discussion context help, and that transformers help more. Despite its title, the study covers BERT-era models, and the proposed BERT+CASCADE hybrid is left as future work.
Key points
- Dataset: the SARC r/politics subsample, about 17k sequences, chosen because it is the only part of SARC with a human evaluation. Average sarcastic share is 23.2%, although the paper treats the split as balanced.
- Accuracy / F1 scores:
- Average human: 0.82 / –
- Bag-of-Words SVM: 0.59 / 0.60
- CNN-SVM: 0.65 / 0.67
- CUE-SVM: 0.69 / 0.70
- CASCADE: 0.74 / 0.75
- RCNN-RoBERTa: 0.79 / 0.78
- CASCADE's user embedding has two parts:
- personality vectors from a CNN pretrained on essays labelled with Big Five traits, averaged over the user's posts;
- style vectors from ParagraphVector.
- The two are fused with CCA and concatenated with forum discourse vectors.
- RCNN-RoBERTa training setup: 5 epochs (about 3 hours), learning rate 2e-5, batch size 10, 64 LSTM units, dropout 0.1. Hyperparameters for both models were chosen by RandomSearch.
- CASCADE pads comments to 100 words and holds out 20% of the training data for validation. The author reports that both main models beat the baselines with statistical significance.
Interesting ideas
- I1. Several rule-based systems in the related work treat sarcasm as a mismatch, for example a hashtag whose sentiment contradicts the rest of the text. This gives an explicit, checkable signal instead of an opaque score.
- I2. CASCADE models each author's habitual tendency toward sarcasm from their post history. The paper's math-weekend example shows the same sentence can be sincere for one speaker and ironic for another.
- I3. Results are placed between a bag-of-words floor and a human ceiling, which shows how much of the gap a model actually closes.
- I4. The test subset was chosen where human judges have the background knowledge to understand the comments. Otherwise even the ceiling becomes unreliable.
- I5. The baselines form a ladder (CNN-SVM, then user embeddings, then personality and discourse features). This lets each kind of context be credited with its own gain.
For Mova
How these ideas could serve Mova, our small open language model — its files, stages and tests:
- I1 (frames): crate prag could flag irony when the literal polarity of a frame conflicts with the polarity expected from context, and record which two pieces disagreed.
- I2 (world): store a stance or attitude attribute on speaker entities, so the irony reading of a line in a story is conditioned on what the world already knows about that speaker.
- I3 (learn): every irony or implicature evaluation could report a lexical-only negative control and human agreement, so gains are measured relative to both ends.
- I4 (answer): when building QA or fable tests that require irony or world knowledge, keep items whose needed background is actually present in Mova's compiled knowledge, or tag them separately.
- I5 (learn): ablate context sources one at a time, such as coreference, speaker history and discourse, to see which actually drives pragmatic decisions.
Institutions
United StatesFields, topics and concepts
📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works
🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 972 worksEvaluation & benchmarks · 1,220 works
Cite
Cite as Juliann Zhou. 2023. An Evaluation of State-of-the-Art Large Language Models for Sarcasm Detection. arXiv:2312.03706.
@misc{zhou2023evaluation,
title = {{An Evaluation of State-of-the-Art Large Language Models for Sarcasm Detection}},
author = {Juliann Zhou},
year = {2023},
doi = {10.48550/arxiv.2312.03706},
eprint = {2312.03706},
archivePrefix = {arXiv},
url = {https://doi.org/10.48550/arxiv.2312.03706}
}References
- S. Amir, B. C. Wallace, H. Lyu, P. Carvalho, and M. J. Silva, “Modelling Context with User Embeddings for Sarcasm Detection in Social Media,” ACLWeb, Aug. 01, 2016. https://aclanthology.org/K16-1017/ (accessed May 31, 2023). Devamanyu Hazarika, Soujanya Poria, Sruthi Gorantla, E. Cambria, R. Zimmermann, and Rada Mihalcea, “CASCADE: Contextual Sarcasm Detection in Online Discussion Forums,” Association for Computational Linguistics, vol. C18, no. 1156, pp. 1837–1848, May 2018, Accessed: Sep. 16, 2023. [Online]. Available: https://arxiv.org/abs/1805.06413 F. Barbieri, H. Saggion, and F. Ronzano, “Modelling Sarcasm in Twitter, a Novel Approach,” Meeting of the Association for Computational Linguistics, vol. W14, no. 2609, pp. 50–58, Jun. 2014, doi: https://doi.org/10.3115/v1/w14-2609. D. Irazú, M. Montes-y-Gómez, Hugo Jair Escalante, P. Rosso, and V. Patti, “A Knowledge-Based Weighted KNN for Detecting Irony in Twitter,” Springer eBooks, vol. 11289, pp. 194–206, Jan. 2018, doi: https://doi.org/10.1007/978-3-030-04497-8_16. A. Ghosh et al., “SemEval-2015 Task 11: Sentiment Analysis of Figurative Language in Twitter,” Association for Computational Linguistics, vol. S15, no. 2080, Jan. 2015, doi: https://doi.org/10.18653/v1/s15-2080. Konstantin Buschmeier, Philipp Cimiano, and R. Klinger, “An Impact Analysis of Features in a Classification Approach to Irony Detection in Product Reviews,” Association for Computational Linguistics, vol. W14, no. 2608, Jan. 2014, doi: https://doi.org/10.3115/v1/w14-2608. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” CoRR, vol. abs/1810.04805, Oct. 2018, Accessed: Sep. 16, 2023. [Online]. Available: https://arxiv.org/abs/1810.04805 A. Khatri and P. P, “Sarcasm Detection in Tweets with BERT and GloVe Embeddings,” ACLWeb, Jul. 01, 2020. https://aclanthology.org/2020.figlang-1.7/ (accessed May 21, 2022). A. Kumar and G. Garg, “Empirical study of shallow and deep learning models for sarcasm detection using context in benchmark datasets,” Journal of Ambient Intelligence and Humanized Computing, vol. 14, no. 5327–5342, Aug. 2019, doi: https://doi.org/10.1007/s12652-019-01419-7. D. Maynard and M. Greenwood, “Who cares about Sarcastic Tweets? Investigating the Impact of Sarcasm on Sentiment Analysis.,” Language Resources and Evaluation, vol. Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pp. 4238– 4243, Mar. 2014, Accessed: Sep. 16, 2023. [Online]. Available: https://aclanthology.org/L14-1527/ Mikhail Khodak, Nikunj Saunshi, and Kiran Vodrahalli, “A Large Self-Annotated Corpus for Sarcasm,” European Language Resources Association (ELRA), vol. arXiv:1704.05579, no. 4, Apr. 2017, Accessed: Sep. 16, 2023. [Online]. Available: https://arxiv.org/abs/1704.05579 R. A. Potamias, G. Siolas, and A. - G. Stafylopatis, “A transformer-based approach to irony and sarcasm detection,” Neural Computing and Applications, vol. 32, no. 23, pp. 17309–17320, Jun. 2020, doi: https://doi.org/10.1007/s00521-020-05102-3. A. Reyes, P. Rosso, and T. Veale, “A multidimensional approach for detecting irony in Twitter,” Language Resources and Evaluation, vol. 47, no. 1, pp. 239–268, Jul. 2012, doi: https://doi.org/10.1007/s10579-012-9196-x. A. Rajadesingan, R. Zafarani, and H. Liu, “Sarcasm Detection on Twitter,” Proceedings of the Eighth ACM International Conference on Web Search and Data Mining -WSDM ’15, 2015, doi: https://doi.org/10.1145/2684822.2685316. Soujanya Poria, E. Cambria, Devamanyu Hazarika, and P. Vij, “A Deeper Look into Sarcastic Tweets Using Deep Convolutional Neural Networks,” International Conference on Computational Linguistics, pp. 1601–1612, Dec. 2016, Accessed: Sep. 16, 2023. [Online]. Available: https://arxiv.org/abs/1610.08815 Cynthia Van Hee, E. Lefever, and V. Hoste, “LT3: Sentiment Analysis of Figurative Tweets: piece of cake #NotReally,” Ghent University Academic Bibliography (Ghent University), vol. S15, no. 2115, Jan. 2015, doi: https://doi.org/10.18653/v1/s15-2115. B. C. Wallace, Do Kook Choe, and E. Charniak, “Sparse, Contextually Informed Models for Irony Detection: Exploiting User Communities, Entities and Sentiment,” Association for Computational Linguistics, vol. P15, no. 1100, Jan. 2015, doi: https://doi.org/10.3115/v1/p15-1100. ↗
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2023-evaluation-state-art-large-language/