Elamur Research⌕
☰
home / papers / 2024 / Is Sarcasm Detection A Step-by-Step Reasoning Process in Large Language Models?

Is Sarcasm Detection A Step-by-Step Reasoning Process in Large Language Models?

Summary

Step-by-step prompting such as CoT helps LLMs on logic and math tasks. This paper asks whether sarcasm recognition benefits from the same treatment, or whether it works more like intuitive, holistic judgement that merges many signals at once. The authors propose SarcasmCue, a prompting framework with four sub-methods. Two are sequential: chain of contradiction (CoC) contrasts surface sentiment with the real intent, and graph of cues (GoC) searches over linked cues. Two are non-sequential: bagging of cues (BoC) aggregates predictions from random cue subsets, and tensor of cues (ToC) fuses linguistic, contextual and emotional cue views via a tensor product. They test these on four benchmarks with GPT-4o, Claude 3.5 Sonnet, Llama 3-8B and Qwen 2-7B, against IO, CoT and ToT prompting. Sequential methods work best on strong models, while ToC dominates on small ones. This suggests sarcasm detection is not purely stepwise reasoning and that the best prompt design depends on model capacity.

Key points

  • The benchmarks are IAC-V1, IAC-V2, SemEval and MUStARD, all scored with F1 / Macro-F1. The framework also transfers to humor detection on CMMA and UR-FUNNY-V2 with a GPT-4o backbone.
  • With GPT-4o and Claude 3.5 Sonnet, CoC and GoC beat the strongest baseline by about 3.5%. The headline relative F1 gains over the best baseline are 4.2%, 2.0%, 29.7% and 58.2%.
  • On Llama 3-8B, ToC reaches an average F1 of 65.24, about 29.7% above ToT. On Qwen 2-7B, ToC improves on IO prompting by 58.2%.
  • In the ablations, removing any cue type (linguistic, emotional or contextual) generally lowers scores. Linguistic cues matter most for BoC.
  • In the error analysis, CoC, GoC and BoC tend to over-predict sarcasm, producing more false positives. ToC has the lowest and most balanced error rate. Code is available at github.com/qiuchili/llm_sarcasm_detection.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Natural language processing · AI & machine learning

Cited by

14 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 12 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2407.12725 · arXiv 2407.12725

@misc{yao2024is,
  title = {{Is Sarcasm Detection A Step-by-Step Reasoning Process in Large Language Models?}},
  author = {Yao, Ben and Zhang, Yazhou and Li, Qiuchi and Qin, Jing},
  year = {2024},
  doi = {10.48550/arxiv.2407.12725},
  eprint = {2407.12725},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.12725}
}

References

19 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609. ↗ our card
  2. Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. 2024. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17682–17690. find
  3. Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. 2019. Towards multimodal sarcasm detection (an _obviously_ perfect paper). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Florence, Italy. Association for Computational Linguistics. find our card
  4. Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, et al. 2024. A survey on multimodal large language models for autonomous driving. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 958–979. find our card
  5. Debanjan Ghosh, Alexander R Fabbri, and Smaranda Muresan. 2018. Sarcasm analysis using conversation context. Computational Linguistics, 44(4):755–792. ↗ our card
  6. Md Kamrul Hasan, Sangwu Lee, Wasifur Rahman, Amir Zadeh, Rada Mihalcea, Louis-Philippe Morency, and Ehsan Hoque. 2021. Humor knowledge enriched transformer for understanding multimodal humor. Proceedings of the AAAI Conference on Artificial Intelligence, 35(14):12972–12980. find
  7. Md Kamrul Hasan, Wasifur Rahman, Amir Zadeh, Jianyuan Zhong, Md Iftekhar Tanveer, Louis-Philippe Morency, et al. 2019. Ur-funny: A multimodal language dataset for understanding humor. arXiv preprint arXiv:1904.06618. ↗
  8. Yiyi Liu, Ruqing Zhang, Yixing Fan, Jiafeng Guo, and Xueqi Cheng. 2023. Prompt tuning with contradictory intentions for sarcasm recognition. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 328–339. find our card
  9. Stephanie Lukin and Marilyn Walker. 2013. Really? well. apparently bootstrapping improves the performance of sarcasm and nastiness classifiers for online dialogue. In Proceedings of the Workshop on Language Analysis in Social Media, pages 30–40, Atlanta, Georgia. Association for Computational Linguistics. find our card
  10. Shereen Oraby, Vrindavan Harrison, Lena Reed, Ernesto Hernandez, Ellen Riloff, and Marilyn Walker. 2016. Creating and characterizing a diverse corpus of sarcasm in dialogue. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 31–41, Los Angeles. Association for Computational Linguistics. find our card
  11. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. SemEval-2018 task 3: Irony detection in English tweets. In Proceedings of the 12th International Workshop on Semantic Evaluation, pages 39–50, New Orleans, Louisiana. Association for Computational Linguistics. ↗ our card
  12. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837. find our card
  13. Deepak Jain, Akshi Kumar, and Geetanjali Garg. 2020. Sarcasm detection in mash-up language using soft-attention based bi-directional lstm and feature-rich cnn. Applied Soft Computing, 91:106198. ↗
  14. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36. find
  15. Bin Liang, Chenwei Lou, Xiang Li, Min Yang, Lin Gui, Yulan He, Wenjie Pei, and Ruifeng Xu. 2022. Multi-modal sarcasm detection via cross-modal graph convolutional network. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1767–1777. Association for Computational Linguistics. find our card
  16. Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. Tensor fusion network for multimodal sentiment analysis. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1103–1114, Copenhagen, Denmark. Association for Computational Linguistics. find
  17. Yazhou Zhang, Dan Ma, Prayag Tiwari, Chen Zhang, Mehedi Masud, Mohammad Shorfuz-zaman, and Dawei Song. 2023. Stance-level sarcasm detection with bert and stance-centered graph attention networks. ACM Transactions on Internet Technology, 23(2):1–21. ↗
  18. Yazhou Zhang, Yang Yu, Qing Guo, Benyou Wang, Dongming Zhao, Sagar Uprety, Dawei Song, Qiuchi Li, and Jing Qin. 2024. Cmma: Benchmarking multi-affection detection in chinese multimodal conversations. Advances in Neural Information Processing Systems, 36. find
  19. Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493. ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yao-2024-sarcasm-detection-step-step-reasoning/