Elamur Research⌕
☰
home / papers / 2024 / SarcasmBench: Towards Evaluating Large Language Models on Sarcasm Understanding

SarcasmBench: Towards Evaluating Large Language Models on Sarcasm Understanding

Paper · 2024 · Yazhou Zhang, Chunwang Zou, Zheng Lian et al.

Summary

The paper tests a common belief: that large language models have essentially solved fast, intuitive "System I" tasks such as sentiment classification. The authors argue that sarcasm is a harder case, because the intended meaning often runs against the literal wording. They built SarcasmBench to check this. It evaluates eleven LLMs (including GPT-4, ChatGPT, Claude 3, Mistral, Baichuan 2/3, ChatGLM 2/3, LLaMA 2/3 and Qwen 1.5) against eight supervised pre-trained language models on six common sarcasm datasets. Each LLM is run under three prompting regimes: zero-shot IO, few-shot IO and few-shot chain-of-thought.

The main results:

  • Prompted LLMs still score below fine-tuned PLM baselines.
  • GPT-4 is clearly the strongest LLM.
  • Plain few-shot prompting works better than both zero-shot and CoT.

The authors explain the CoT result by suggesting that sarcasm is grasped holistically rather than through explicit step-by-step logic. The benchmark matters because it gives a systematic, reproducible way to see where general-purpose LLMs fall short on figurative, intention-laden language.

Key points

  • Scale of the comparison: 11 LLMs and 8 supervised PLMs, evaluated on 6 widely used sarcasm benchmarks. Sarcasm detection is framed as label generation, with "sarcastic" or "non-sarcastic" as the only allowed outputs.
  • GPT-4 beats the other LLMs by about 14.0% on average across prompting methods. Claude 3 and ChatGPT come next.
  • Few-shot IO prompting gives roughly a 4.5% average gain over zero-shot IO and few-shot CoT. Few-shot demonstrations are chosen by KNN search for examples similar to the test item.
  • Multimodal models were also tested on the MMSD and CMMA datasets. GPT-4V, Qwen-VL-Plus and Wenxin 4 were compared with multimodal PLMs such as Multi-view CLIP and MILNet, and supervised systems again led overall.
  • Stated limitations:
    • No tree- or graph-style prompting was tried.
    • Performance across different contexts was not analysed.
    • Ensembling methods such as bagging and boosting are left for future work.

Interesting ideas

  • I1. Chain-of-thought did not help here and sometimes hurt, which suggests that irony shows up as a single clash between surface polarity and situation rather than as a reasoning chain; this is worth keeping in mind before building multi-step derivations for every pragmatic phenomenon.
  • I2. Choosing few-shot demonstrations by nearest-neighbour similarity to the test input is a cheap way to make in-context examples relevant, and it helps any setup where a model learns from a handful of shown cases.
  • I3. The study compares big prompted models against small supervised specialists, and the specialists still win on a task that looks easy; this is a useful reminder not to assume the larger general model is the better annotator.
  • I4. Sarcasm serves as a stress test showing that "solved" sentiment tasks are not solved once the literal reading is misleading, so evaluation sets should include items where taking the surface at face value gives the wrong answer.
  • I5. Forcing the model to return only a fixed label string makes the outputs of many different systems directly comparable and easy to parse automatically.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): The irony detector in prag could fire on a direct incongruity pattern, such as a positive-attitude verb whose complement frame describes an unpleasant event (the paper's "like to be reprimanded" example), and record that clash as its explanation.
  • I2 (learn): When an LLM teacher tier labels a hard sentence, it could be shown the most similar already-agreed silver examples instead of a fixed demo set.
  • I3 (learn): Teacher tiers could be scored per phenomenon (e.g. irony, negation) against a supervised baseline before their labels are allowed into silver data.
  • I4 (answer): Fable and tale QA could include sarcastic utterances as negative controls, checking that readers answer from intended meaning rather than literal polarity.
  • I5 (learn): Prompts to LLM teachers could require closed-vocabulary answers (frame labels, role names), so their votes can be compared mechanically with the witness parsers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️Chinese Academy of SciencesBeijing · China246 works · 24,952 citations🔬Institute of AutomationBeijing · China87 works · 14,883 citations🎓Hong Kong Polytechnic UniversityHong Kong · China106 works · 5,190 citations🎓Tianjin UniversityTianjin · China52 works · 1,128 citations🎓Halmstad UniversityHalmstad · Sweden4 works · 167 citations🎓Zhengzhou University of Light IndustryZhengzhou · China3 works · 78 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Sentiment & emotion · 1,035 works

Cite

Cite as Yazhou Zhang, Chunwang Zou, Zheng Lian, Prayag Tiwari and Jing Qin. 2024. SarcasmBench: Towards Evaluating Large Language Models on Sarcasm Understanding. arXiv:2408.11319.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2408.11319 · arXiv 2408.11319

@misc{zhang2024sarcasmbench,
  title = {{SarcasmBench: Towards Evaluating Large Language Models on Sarcasm Understanding}},
  author = {Yazhou Zhang, Chunwang Zou, Zheng Lian, Prayag Tiwari and Jing Qin},
  year = {2024},
  doi = {10.48550/arxiv.2408.11319},
  eprint = {2408.11319},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2408.11319}
}

References

51 references, 34 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Z. Shao, Z. Yu, M. Wang, J. Yu, Prompting large language models with answer heuristics for knowledge-based visual question answering, in: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, 2023, pp. 14974– 14983. find
  2. Y. Zhang, M. Wang, P. Tiwari, Q. Li, B. Wang, J. Qin, Dialoguellm: Context and emotion knowledge-tuned llama models for emotion recognition in conversations, arXiv preprint arXiv:2310.11374 (2023). ↗
  3. Y. Zhang, M. Wang, C. Ren, Q. Li, P. Tiwari, B. Wang, J. Qin, Pushing the limit of llm capacity for text classification, arXiv preprint arXiv:2402.07470 (2024). ↗
  4. J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al., Chain-of-thought prompting elicits reasoning in large language models, Advances in neural information processing systems 35 (2022) 24824–24837.
  5. S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y. Cao, K. Narasimhan, Tree of thoughts: Deliberate problem solving with large language models, Advances in Neural Information Processing Systems 36 (2024). find
  6. Y. Ren, Z. Wang, Q. Peng, D. Ji, A knowledge-augmented neural network model for sarcasm detection, Information Processing & Management 60 (6) (2023) 103521. find
  7. Y. Zhang, Y. Yu, D. Zhao, Z. Li, B. Wang, Y. Hou, P. Tiwari, J. Qin, Learning multi-task commonness and uniqueness for multi-modal sarcasm detection and sentiment analysis in conversation, IEEE Transactions on Artificial Intelligence (2023). find
  8. T. Yue, R. Mao, H. Wang, Z. Hu, E. Cambria, Knowlenet: Knowledge fusion network for multimodal sarcasm detection, Information Fusion 100 (2023) 101921. find
  9. Y. Zhang, D. Ma, P. Tiwari, C. Zhang, M. Masud, M. Shorfuzzaman, D. Song, Stance-level sarcasm detection with bert and stance-centered graph attention networks, ACM Transactions on Internet Technology 23 (2) (2023) 1–21. find
  10. J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al., Gpt-4 technical report, arXiv preprint arXiv:2303.08774 (2023). ↗
  11. A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al., Mistral 7b, arXiv preprint arXiv:2310.06825 (2023). ↗
  12. A. Yang, B. Xiao, B. Wang, B. Zhang, C. Bian, C. Yin, C. Lv, D. Pan, D. Wang, D. Yan, et al., Baichuan 2: Open large-scale language models, arXiv preprint arXiv:2309.10305 (2023). ↗
  13. T. GLM, A. Zeng, B. Xu, B. Wang, C. Zhang, D. Yin, D. Rojas, G. Feng, H. Zhao, H. Lai, et al., Chatglm: A family of large language models from glm-130b to glm-4 all tools, arXiv preprint arXiv:2406.12793 (2024). ↗
  14. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al., Llama: Open and efficient foundation language models, arXiv preprint arXiv:2302.13971 (2023). ↗
  15. A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, et al., Qwen2 technical report, arXiv preprint arXiv:2407.10671 (2024). ↗
  16. B. Yao, Y. Zhang, Q. Li, J. Qin, Is sarcasm detection a step-by-step reasoning process in large language models? (2024). arXiv:2407.12725. URL https://arxiv.org/abs/2407.12725 ↗
  17. Y. Zhang, Y. Yu, Q. Guo, B. Wang, D. Zhao, S. Uprety, D. Song, Q. Li, J. Qin, Cmma: Benchmarking multi-affection detection in chinese multi-modal conversations, Advances in Neural Information Processing Systems 36 (2024). find
  18. L. Zhou, X. Xu, X. Wang, Bns-net: A dual-channel sarcasm detection method considering behavior-level and sentence-level conflicts, arXiv preprint arXiv:2309.03658 (2023). ↗
  19. D. Jain, A. Kumar, G. Garg, Sarcasm detection in mash-up language using soft-attention based bi-directional lstm and feature-rich cnn, Applied Soft Computing 91 (2020) 106198. find
  20. D. Ghosh, A. R. Fabbri, S. Muresan, Sarcasm analysis using conversation context, Computational Linguistics 44 (4) (2018) 755–792.
  21. B. Liang, C. Lou, X. Li, M. Yang, L. Gui, Y. He, W. Pei, R. Xu, Multi-modal sarcasm detection via cross-modal graph convolutional network, in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vol. 1, Association for Computational Linguistics, 2022, pp. 1767–1777.
  22. Y. Liu, Y. Wang, A. Sun, X. Meng, J. Li, J. Guo, A dual-channel framework for sarcasm recognition by detecting sentiment conflict, in: M. Carpuat, M.-C. de Marneffe, I. V. Meza Ruiz (Eds.), Findings of the Association for Computational Linguistics: NAACL 2022, Association for Computational Linguistics, Seattle, United States, 2022, pp. 1670–1680. doi:10.18653/v1/2022.findings-naacl.126. URL https://aclanthology.org/2022. findings-naacl.126 ↗
  23. Y. Liu, R. Zhang, Y. Fan, J. Guo, X. Cheng, Prompt tuning with contradictory intentions for sarcasm recognition, in: Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, 2023, pp. 328–339.
  24. Y. Qiao, L. Jing, X. Song, X. Chen, L. Zhu, L. Nie, Mutual-enhanced incongruity learning network for multi-modal sarcasm detection, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, 2023, pp. 9507–9515. find
  25. Y. Tian, N. Xu, R. Zhang, W. Mao, Dynamic routing transformer network for multimodal sarcasm detection, in: A. Rogers, J. Boyd-Graber, N. Okazaki (Eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Toronto, Canada, 2023, pp. 2468–2480. doi: 10.18653/v1/2023.acl-long.139. URL https://aclanthology.org/2023.acl-long.139 ↗
  26. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al., Evaluating large language models trained on code, arXiv preprint arXiv:2107.03374 (2021). ↗
  27. W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, Y. Du, C. Yang, Y. Chen, Z. Chen, J. Jiang, R. Ren, Y. Li, X. Tang, Z. Liu, P. Liu, J.-Y. Nie, J.-R. Wen, A survey of large language models (2023). arXiv: 2303.18223. ↗
  28. X. Ren, P. Zhou, X. Meng, X. Huang, Y. Wang, W. Wang, P. Li, X. Zhang, A. Podolskiy, G. Arshinov, et al., Pangu-{\Sigma}: Towards trillion parameter language model with sparse heterogeneous computing, arXiv preprint arXiv:2303.10845 (2023). ↗
  29. B. Yao, Y. Zhang, Q. Li, J. Qin, Is sarcasm detection a step-by-step reasoning process in large language models?, arXiv preprint arXiv:2407.12725 (2024). ↗
  30. X. Sun, X. Li, J. Li, F. Wu, S. Guo, T. Zhang, G. Wang, Text classification via large language models, arXiv preprint arXiv:2305.08377 (2023). ↗
  31. S. Lukin, M. Walker, Really? well. apparently bootstrapping improves the performance of sarcasm and nastiness classifiers for online dialogue, arXiv preprint arXiv:1708.08572 (2017). ↗
  32. S. Oraby, V. Harrison, L. Reed, E. Hernandez, E. Riloff, M. Walker, Creating and characterizing a diverse corpus of sarcasm in dialogue, arXiv preprint arXiv:1709.05404 (2017). ↗
  33. A. Ghosh, T. Veale, Fracking sarcasm using neural network, in: Proceedings of the 7th workshop on computational approaches to subjectivity, sentiment and social media analysis, 2016, pp. 161–169.
  34. I. A. Farha, S. Oprea, S. Wilson, W. Magdy, Semeval-2022 task 6: isarcasmeval, intended sarcasm detection in english and arabic, in: The 16th International Workshop on Semantic Evaluation 2022, Association for Computational Linguistics, 2022, pp. 802–814.
  35. E. Riloff, A. Qadir, P. Surve, L. De Silva, N. Gilbert, R. Huang, Sarcasm as contrast between a positive sentiment and negative situation, in: Proceedings of the 2013 conference on empirical methods in natural language processing, 2013, pp. 704–714.
  36. C. Van Hee, E. Lefever, V. Hoste, Semeval-2018 task 3: Irony detection in english tweets, in: Proceedings of the 12th international workshop on semantic evaluation, 2018, pp. 39–50.
  37. S. Lukin, M. Walker, Really? well. apparently bootstrapping improves the performance of sarcasm and nastiness classifiers for online dialogue, in: C. Danescu-Niculescu-Mizil, A. Farzindar, M. Gamon, D. Inkpen, M. Nagarajan (Eds.), Proceedings of the Workshop on Language Analysis in Social Media, Association for Computational Linguistics, Atlanta, Georgia, 2013, pp. 30–40. URL https://aclanthology.org/W13-1104 ↗
  38. Y. Kim, Convolutional neural networks for sentence classification, arXiv preprint arXiv:1408.5882 (2014). ↗
  39. Y. Zhang, Q. Li, D. Song, P. Zhang, P. Wang, Quantum-inspired interactive networks for conversational sentiment analysis, in: Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, International Joint Conferences on Artificial Intelligence Organization, 2019, pp. 5436–5442. doi:10.24963/ijcai.2019/755. URL https://doi.org/10.24963/ijcai.2019/755 ↗
  40. Y. Wang, M. Huang, L. Zhao, et al., Attention-based lstm for aspect-level sentiment classification, in: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016, pp. 606–615.
  41. J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, in: J. Burstein, C. Doran, T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics, Minneapolis, Minnesota, 2019, pp. 4171–4186. doi:10.18653/v1/N19-1423. URL https://aclanthology.org/N19-1423 ↗
  42. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized bert pretraining approach (2019). arXiv:1907.11692. URL https://arxiv.org/abs/1907.11692 ↗
  43. P. He, X. Liu, J. Gao, W. Chen, Deberta: Decoding-enhanced bert with disentangled attention, in: International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=XPZIaotutsD ↗
  44. Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, Q. V. Le, Xlnet: Generalized autoregressive pretraining for language understanding, Advances in neural information processing systems 32 (2019).
  45. Y. Liu, Y. Wang, A. Sun, X. Meng, J. Li, J. Guo, A dual-channel framework for sarcasm recognition by detecting sentiment conflict, in: M. Carpuat, M.-C. de Marneffe, I. V. Meza Ruiz (Eds.), Findings of the Association for Computational Linguistics: NAACL 2022, Association for Computational Linguistics, Seattle, United States, 2022, pp. 1670–1680. doi:10.18653/v1/2022.findings-naacl.126. URL https://aclanthology.org/2022. findings-naacl.126 ↗
  46. Y. Cai, H. Cai, X. Wan, Multi-modal sarcasm detection in Twitter with hierarchical fusion model, in: A. Korhonen, D. Traum, L. Màrquez (Eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Florence, Italy, 2019, pp. 2506– 2515. doi:10.18653/v1/P19-1239. URL https://aclanthology.org/P19-1239 ↗
  47. B. Tang, B. Lin, H. Yan, S. Li, Leveraging generative large language models with visual instruction and demonstration retrieval for multimodal sarcasm detection, in: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 2024, pp. 1732–1742.
  48. J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, J. Zhou, Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond (2023). arXiv:2308.12966. URL https://arxiv.org/abs/2308.12966 ↗
  49. H. Pan, Z. Lin, P. Fu, Y. Qi, W. Wang, Modeling intra and inter-modality incongruity for multi-modal sarcasm detection, in: T. Cohn, Y. He, Y. Liu (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, Association for Computational Linguistics, Online, 2020, pp. 1383–1392. doi:10.18653/v1/2020.findings-emnlp.124. URL https://aclanthology.org/2020. findings-emnlp.124 ↗
  50. Y. Qiao, L. Jing, X. Song, X. Chen, L. Zhu, L. Nie, Mutual-enhanced incongruity learning network for multi-modal sarcasm detection, Proceedings of the AAAI Conference on Artificial Intelligence 37 (8) (2023) 9507–9515. doi:10.1609/aaai.v37i8.26138. URL https://ojs.aaai.org/index.php/AAAI/article/view/26138 ↗
  51. L. Qin, S. Huang, Q. Chen, C. Cai, Y. Zhang, B. Liang, W. Che, R. Xu, MMSD2.0: Towards a reliable multi-modal sarcasm detection system, in: A. Rogers, J. Boyd-Graber, N. Okazaki (Eds.), Findings of the Association for Computational Linguistics: ACL 2023, Association for Computational Linguistics, Toronto, Canada, 2023, pp. 10834–10845. doi:10.18653/v1/ 2023.findings-acl.689. URL https://aclanthology.org/2023.findings-acl.689 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2024-sarcasmbench-towards-evaluating-large-language/