Elamur Research⌕
☰
home / papers / 2022 / Should a Chatbot be Sarcastic? Understanding User Preferences Towards Sarcasm Generation

Should a Chatbot be Sarcastic? Understanding User Preferences Towards Sarcasm Generation

Paper · 2022 · Silviu Vlad Oprea, Steven Wilson and Walid Magdy

Summary

Earlier sarcasm generation work mostly asked how to make machine output sound sarcastic. This paper asks whether and when a chatbot should be sarcastic at all, and which ingredients make sarcasm recognisable. The authors took 250 tweet-derived texts in which users describe their own actions, drawn from the Wilson and Mihalcea (2019) corpus, and split them into five sentiment bands. For each input they produced four responses from their Chandler generator and three from other systems. Chandler is built on the Implicit Display Theory (IDT), which allows individual linguistic devices to be switched on or off. Human judges then rated how appropriate sarcasm was, picked their preferred response, and scored each response on sarcasm, humour, specificity and coherence. Sarcasm was judged unsuitable for most inputs. When it was acceptable, the input was usually positive or joking. Even then, people tended to choose non-sarcastic replies, with coherence driving their choices. Pragmatic insincerity and emotional markers turned out to be key for perceived sarcasm. The work argues that sarcasm generation should be gated by appropriateness, which matters both ethically and for usability.

Key points

  • The input set has 250 texts in five partitions of 50: very negative, negative, neutral, positive and very positive. Sentiment was assigned with the RoBERTa-based tweet sentiment classifier of Barbieri et al. (2020).
  • Chandler follows IDT, under which sarcasm requires three things:
    • an allusion to a failed expectation,
    • pragmatic insincerity, meaning a deliberate violation of a pragmatic principle such as a Grice maxim,
    • an indirect signal of negative attitude, carried by emotional markers.
  • Chandler variants differ by if-then relation type (xAttr, xReact, xNeed, xEffect) and by ablations, for example removing insincerity (−i).
  • Removing emotional markers significantly lowered perceived sarcasm for every relation type. Removing insincerity lowered sarcasm for some relation types but made responses more specific and coherent. Responses that mention user attributes (xAttr) scored highest on sarcasm, humour, specificity and coherence.
  • Out of 750 survey instances, only 107 (about 14%) had a preferred response with a sarcasm score of at least 3.
    • Within those, SarcasmBot was often preferred, although it produced only 28 distinct replies across all 250 inputs.
    • For positive inputs, Ch-xNeed was preferred.
  • Significance was tested with t-tests using Bonferroni correction (p < 0.001). The results support IDT's claim that sarcasm strength grows with the number of devices present.

Interesting ideas

  • I1. Decide whether a rhetorical style fits the context before deciding how to produce it, and check this with human appropriateness ratings rather than assuming the style is always welcome.
  • I2. Turn a linguistic theory into a checklist of separable devices, then ablate each device to measure its effect on human perception.
  • I3. Model the speaker's expectation simply as the negation of the event the text describes, which makes a "failed expectation" concrete and checkable.
  • I4. Stratify evaluation inputs by sentiment, including separate extreme bands, because behaviour at "very positive" differed from behaviour at "positive".
  • I5. Count unique outputs alongside preference scores, since a template system can win short evaluations while being too repetitive for real dialogue.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): The generator could include an explicit, explainable gate that permits ironic or playful phrasing only when the world of the text shows a positive or already-joking context.
  • I2 (frames): The irony detector in prag could score IDT's three conditions separately (failed expectation, insincerity, negative attitude), so each irony label comes with a traceable justification and a graded degree.
  • I3 (world): The world builder could store a ¬P expectation state alongside each reported event, giving the irony and implicature modules something explicit to point to.
  • I4 (learn): Evaluation sets for pragmatic phenomena could be stratified by sentiment band, so that gains are not driven by a single easy slice.
  • I5 (learn): A diversity check such as distinct-output counts per input set could serve as a negative control when judging generation quality.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom255 works · 68,771 citations🔬The Alan Turing InstituteLondon · United Kingdom19 works · 3,608 citations🎓Oakland UniversityRochester · United States4 works · 180 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 2 Sentiment & emotion · 608 works Dialogue systems · 401 works

Cite

Cite as Silviu Vlad Oprea, Steven Wilson and Walid Magdy. 2022. Should a Chatbot be Sarcastic? Understanding User Preferences Towards Sarcasm Generation. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.acl-long.530

@inproceedings{oprea2022should,
  title = {{Should a Chatbot be Sarcastic? Understanding User Preferences Towards Sarcasm Generation}},
  author = {Silviu Vlad Oprea, Steven Wilson and Walid Magdy},
  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.530},
  url = {https://doi.org/10.18653/v1/2022.acl-long.530}
}

References

45 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ibrahim Abu Farha, Silviu Vlad Oprea, Steven R. Wilson, and Walid Magdy. 2022. SemEval-2022 task 6: iSarcasmEval, intended sarcasm detection in English and Arabic. In Proceedings of The 16th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  2. Jacopo Amidei, Paul Piwek, and Alistair Willis. 2018. Rethinking the agreement in human evaluation tasks. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3318–3329, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find
  3. Silvio Amir, Byron C. Wallace, Hao Lyu, Paula Carvalho, and Mario J. Silva. 2016. Modelling context with user embeddings for sarcasm detection in social media. In CoNLL, pages 167–177. ACL. ↗
  4. David Bamman and Noah A. Smith. 2015. Contextualized sarcasm detection on twitter. In ICWSM, pages 574–577. AAAI Press. find
  5. Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa Anke, and Leonardo Neves. 2020. TweetEval: Unified benchmark and comparative evaluation for tweet classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1644–1650, Online. ACL.
  6. Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: Commonsense transformers for automatic knowledge graph construction. In ACL, pages 4762–4779. ACL.
  7. Richard Bruntsch and Willibald Ruch. 2017. Studying irony detection beyond ironic criticism: Let’s include ironic praise. Frontiers in Psychology, 8:606. ↗
  8. Tuhin Chakrabarty, Debanjan Ghosh, Smaranda Muresan, and Nanyun Peng. 2020. Rˆ3: Reverse, retrieve, and rank for sarcasm generation with commonsense knowledge. In ACL, pages 7976–7986. ACL.
  9. Herbert H. Clark. 1996. Using Language. Cambridge University Press. find
  10. Herbert H. Clark and Richard J. Gerrig. 1984. On the pretense theory of irony. Journal of Experimental Psychology: General, 113(1):121–126. ↗
  11. Herbert L Colston and Jennifer O’Brien. 2000b. Contrast of Kind Versus Contrast of Magnitude: The Pragmatic Accomplishments of Irony and Hyperbole. Discourse Processes, 30(2):179–199. ↗
  12. J.A. Meaney, Steven R. Wilson, Luis Chiruzzo, Adam Lopez, and Walid Magdy. 2021. Semeval 2021 task 7, hahackathon, detecting and rating humor and offense. In ACL.
  13. Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. 2021. Dealing with disagreements: Looking beyond the majority vote in subjective annotations. arXiv preprint arXiv:2110.05719. ↗
  14. George A. Miller. 1995. Wordnet: A lexical database for english. Communications of the ACM, 38(11):39–41. ↗
  15. Shelly Dews and Ellen Winner. 1995. meaning: A social function of irony. Muting the H. P. Grice. 1975. Logic and conversation. In Peter Cole and Jerry L. Morgan, editors, Syntax and Semantics: Vol. 3: Speech Acts, pages 41–58. Academic Press, Cambridge, UK. find
  16. Devamanyu Hazarika, Soujanya Poria, Sruthi Gorantla, Erik Cambria, Roger Zimmermann, and Rada Mihalcea. 2018. Cascade: Contextual sarcasm detection in online discussion forums. In COLING, pages 1837–1848. ACL.
  17. Julia Jorgensen. 1996. The functions of sarcastic irony in speech. Journal of Pragmatics, 26(5):613–634. ↗
  18. Aditya Joshi, Anoop Kunchukuttan, Mark James Carman, and Pushpak Bhattacharyya. 2015. Sarcasmbot: An open-source sarcasm-generation module for chatbots. In WISDOM at KDD. ACM. find
  19. Aditya Joshi, Vaibhav Tripathi, Kevin Patel, Pushpak Bhattacharyya, and Mark Carman. 2016. Are word embedding-based features useful for sarcasm detection? In EMNLP, pages 1006–1011. ACL. ↗
  20. Mikhail Khodak, Nikunj Saunshi, and Kiran Vodrahalli. 2018. A large self-annotated corpus for sarcasm. In LREC. ELRA.
  21. Roger J. Kreuz and Sam Glucksberg. 1989. How to Be Sarcastic: The Echoic Reminder Theory of Verbal Irony. Journal of Experimental Psychology: General, 118(4):374–386. ↗
  22. Roger J Kreuz, Debra L Long, and Mary B Church. 1991. On Being Ironic: Pragmatic and Mnemonic Implications. Metaphor and Symbolic Activity, 6(3):149–162. ↗
  23. Sachi Kumon-Nakamura, Sam Glucksberg, and Mary Brown. 1995. How about another piece of pie: The allusional pretense theory of discourse irony. ↗
  24. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  25. William C. Mann and Sandra A. Thompson. 1987. Rhetorical Structure Theory: Description and Construction of Text Structures, pages 85–95. Springer Netherlands, Dordrecht. find
  26. Abhijit Mishra, Tarun Tater, and Karthik Sankaranarayanan. 2019. A modular architecture for unsupervised sarcasm generation. In EMNLP-IJCNLP, pages 6144–6154. ACL.
  27. Silviu Oprea and Walid Magdy. 2019. Exploring author context for detecting intended vs perceived sarcasm. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2854–2859, Florence, Italy. Association for Computational Linguistics. ↗
  28. Silviu Oprea and Walid Magdy. 2020. The effect of sociocultural variables on sarcasm communication online. Proceedings of The 23rd ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW). ↗
  29. Silviu Oprea, Steven Wilson, and Walid Magdy. 2021. Chandler: An explainable sarcastic response generator. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 339–349, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  30. Penny M. Pexman and Meghan T. Zvaigzne. 2004. Does irony go better with friends? Metaphor and Symbol, 19(2):143–163. ↗
  31. Ashwin Rajadesingan, Reza Zafarani, and Huan Liu. 2015. Sarcasm detection on twitter: A behavioral modeling approach. In WSDM, pages 97–106. ACM. ↗
  32. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalindra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In EMNLP, pages 704–714. ACL.
  33. Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. SemEval-2017 task 4: Sentiment analysis in Twitter. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 502–518, Vancouver, Canada. ACL.
  34. Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. 2019. Atomic: An atlas of machine commonsense for if-then reasoning. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):3027–3035. ↗
  35. Dan Sperber and Deirdre Wilson. 1981. Irony and the use-mention distinction. Philosophy, 3:143–184. find
  36. Dan Sperber and Deirdre Wilson. 1998. Irony and relevance: A reply to seto, hamamoto and yamanashi. In R. Carston and S. Uchida, editors, Relevance theory: Applications and implications, pages 289–293. Benjamins, Amsterdam. find
  37. Martin Sykora, Suzanne Elayan, and Thomas W Jackson. 2020. A qualitative analysis of sarcasm, irony and related #hashtags on twitter. Big Data & Society, 7(2):2053951720972735. ↗
  38. John W Tukey. 1949. Comparing individual means in the analysis of variance. Biometrics, pages 99–114. ↗
  39. Akira Utsumi. 1996. Implicit display theory of verbal irony: Towards a computational model of irony. In Proceedings of the International Workshop on Computational Humor (IWCH’96). find
  40. Akira Utsumi. 2000. Verbal irony as implicit display of ironic environment: Distinguishing ironic utterances from nonirony. Journal of Pragmatics, 32(12):1777–1806. ↗
  41. Byron C. Wallace, Do Kook Choe, and Eugene Charniak. 2015. Sparse, contextually informed models for irony detection: Exploiting user communities, entities and sentiment. In ACL, pages 1035–1044. ACL.
  42. Deirdre Wilson. 2006. The pragmatics of verbal irony: Echo or pretence? Lingua, 116(10):1722–1743. ↗
  43. Deirdre Wilson and Dan Sperber. 1992. On verbal irony. Lingua, 87(1):53–76. ↗
  44. Steven Wilson and Rada Mihalcea. 2019. Predicting human activities from user-generated content. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2572–2582, Florence, Italy. Association for Computational Linguistics. ↗
  45. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. DialoGPT : Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 270–278. ACL. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/oprea-2022-should-chatbot-be-sarcastic-understanding/