Elamur Research⌕
☰
home / papers / 2024 / I'm sure you're a real scholar yourself: Exploring Ironic Content Generation by Large Language Models

I'm sure you're a real scholar yourself: Exploring Ironic Content Generation by Large Language Models

Paper · 2024 · Pier Felice Balestrucci, Silvia Casola, Soda Marem Lo et al.

Summary

The paper asks whether LLMs can produce replies to social media posts that human readers actually find ironic. The authors repurpose EPIC, a perspectivist irony corpus of short post–reply conversations originally built for detection, as training data for generation. Zero-shot prompting of a base Mistral 7B gave poor output: long texts, numbered lists and repetitions. They therefore fine-tuned it with LoRA into two separate models, one learning from ironic pairs (IRO) and one from non-ironic pairs (NIRO). They compared the outputs linguistically with the source posts and with human replies, and ran a large crowdsourced evaluation. IRO's replies are linguistically poorer than human ones, but they often allude to implicit context and are sarcastic. Raters judged them more ironic than NIRO's in the vast majority of comparisons. A follow-up tried to build age-targeted irony generators, but young and old raters did not clearly prefer "their" model. The work matters because it shows that irony generation is learnable from modest data. It also releases generated replies and judgments for irony and human-vs-machine detection research.

Key points

  • EPIC has 3,000 conversations from Reddit and Twitter, collected across five English-speaking countries. It carries 14,172 disaggregated annotations (about 5 per item) plus annotator demographics such as age, gender and ethnicity.
  • Labels were aggregated by per-instance majority vote, and items with no majority were discarded. The data was split 80/20 for train/test, and Mistral 7B was LoRA-tuned for five epochs; Llama2-7B was tried only in preliminary runs.
  • The authors trained eight variants: ironic or non-ironic, combined with masking or not masking the prompt in the loss, and removing or anonymizing @-mentions. Three authors judged 100 outputs per variant, and the best setup masked the input and dropped mentions.
  • Generation was run on 523 cleaned test posts, with non-English or content-free posts removed. Crowdworkers were recruited on Prolific.
  • In the age experiment (Y-model vs O-model), young and old raters gave identical judgments on 188 items (45.09%). This showed no age-specific preference, and the authors blame the small, non-dedicated dataset.

Interesting ideas

  • I1. Train two twin generators on identical prompts that differ only in the label of their training data (ironic vs literal), and compare their outputs side by side to isolate what the label contributes.
  • I2. Screen outputs for severe defects (empty text, textual loops, verbatim copies of the input) before judging any subtler property. This keeps fluency failures from contaminating judgments of intent.
  • I3. Keep annotations disaggregated with annotator metadata, then aggregate by majority vote and drop items where no majority emerges. This preserves the subjectivity signal while giving a clean training set.
  • I4. Test small preprocessing choices factorially rather than fixing them by intuition, and pick the winner by human-judged quality. The choices here were loss masking of the prompt and how noisy tokens such as user mentions are handled.
  • I5. Evaluate pragmatic quality by asking raters which of two candidates is more ironic, with a "neither" option, instead of absolute scores. Relative judgments are more stable for subjective phenomena.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Pair ironic and literal replies to the same post to build contrastive test sets for the irony and implicature module. A correct detector must flip its reading between the two.
  • I2 (back): Add a cheap defect gate before generated text from the meaning graph is scored for faithfulness or style.
  • I3 (learn): When witness parsers or LLM teachers disagree without a majority, exclude the item from silver data and keep it in a separate "contested" pool for analysis.
  • I4 (learn): Apply the same small grid when fine-tuning LLM teacher tiers, for example prompt masking and handling of URLs and handles, and keep the winning configuration documented.
  • I5 (answer): Use pairwise "which reading fits better, or neither" checks when grading Mova's explanations of figurative passages in fables and tales.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Ludwig-Maximilians-Universität MünchenMunich · Germany107 works · 56,478 citations🎓University of TurinTurin · Italy33 works · 6,637 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Pier Felice Balestrucci, Silvia Casola, Soda Marem Lo, Valerio Basile and Alessandro Mazzei. 2024. I'm sure you're a real scholar yourself: Exploring Ironic Content Generation by Large Language Models. Findings of the Association for Computational Linguistics: EMNLP 2024.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.findings-emnlp.847

@inproceedings{balestrucci2024i,
  title = {{I'm sure you're a real scholar yourself: Exploring Ironic Content Generation by Large Language Models}},
  author = {Pier Felice Balestrucci, Silvia Casola, Soda Marem Lo, Valerio Basile and Alessandro Mazzei},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2024},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.847},
  url = {https://doi.org/10.18653/v1/2024.findings-emnlp.847}
}

References

30 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Miriam Amin and Manuel Burghardt. 2020. A survey on approaches to computational humor generation. In Proceedings of the 4th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, pages 29–41, Online. International Committee on Computational Linguistics. ↗
  2. Emily M Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604. ↗
  3. Julita Bielaniewicz, Kamil Kanclerz, Piotr Miłkowski, Marcin Gruza, Konrad Karanowski, Przemysław Kazienko, and Jan Kocoń. 2022. Deep-sheep: Sense of humor extraction from embeddings in the personalized context. In 2022 IEEE International Conference on Data Mining Workshops (ICDMW), pages 967– 974. find
  4. Silvia Casola, Simona Frenda, Soda Marem Lo, Erhan Sezerer, Antonio Uva, Valerio Basile, Cristina Bosco, Alessandro Pedrani, Chiara Rubagotti, Viviana Patti, and Davide Bernardi. 2024. MultiPICo: Multilingual perspectivist irony corpus. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16008–16021, Bangkok, Thailand. Association for Computational Linguistics.
  5. Silvia Casola, Soda Marem Lo, Valerio Basile, Simona Frenda, Alessandra Cignarella, Viviana Patti, and Cristina Bosco. 2023. Confidence-based ensembling of perspective-aware models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3496–3507, Singapore. Association for Computational Linguistics. find
  6. Tuhin Chakrabarty, Debanjan Ghosh, Smaranda Muresan, and Nanyun Peng. 2020. Rˆ3: Reverse, retrieve, and rank for sarcasm generation with commonsense knowledge. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7976–7986, Online. Association for Computational Linguistics. ↗
  7. Yi-Pei Chen, Noriki Nishida, Hideki Nakayama, and Yuji Matsumoto. 2024. Recent trends in personalized dialogue generation: A review of datasets, methodologies, and evaluations. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 13650–13665, Torino, Italia. ELRA and ICCL. find
  8. Alessandra Cignarella, Simona Frenda, Valerio Basile, Cristina Bosco, Viviana Patti, and Paolo Rosso. 2018. Overview of the evalita 2018 task on irony detection in italian tweets (ironita). pages 26–34.
  9. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20:37 – 46. ↗
  10. Simona Frenda, Alessandro Pedrani, Valerio Basile, Soda Marem Lo, Alessandra Teresa Cignarella, Raffaella Panizzon, Cristina Marco, Bianca Scarlini, Viviana Patti, Cristina Bosco, and Davide Bernardi. 2023. EPIC: Multi-perspective annotation of a corpus of irony. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13844–13857, Toronto, Canada. Association for Computational Linguistics.
  11. Aparna Garimella, Carmen Banea, Nabil Hossain, and Rada Mihalcea. 2020. “judge me by my size (noun), do you?” YodaLib: A demographic-aware humor generation framework. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2814–2825, Barcelona, Spain (Online). International Committee on Computational Linguistics. find
  12. Raymond W. Gibbs and Herbert L. Colston. 2007. Irony in Language and Thought: A Cognitive Science Reader. Lawrence Erlbaum Associates, New York. ↗
  13. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  14. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825. ↗
  15. Jihen Karoui, Farah Benamara, Véronique Moriceau, Viviana Patti, Cristina Bosco, and Nathalie Aussenac-Gilles. 2017. Exploring the impact of pragmatic phenomena on irony detection in tweets: A multilingual corpus study. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 262–272, Valencia, Spain. Association for Computational Linguistics. ↗
  16. Rensis Likert. 1932. A technique for the measurement of attitudes / by Rensis Likert. Archives of psychology ; no. 140. [s.n.], New York. find
  17. Tyler Loakman, Aaron Maladry, and Chenghua Lin. 2023. The iron(ic) melting pot: Reviewing human evaluation in humour, irony and sarcasm generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6676–6689, Singapore. Association for Computational Linguistics.
  18. Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In International Conference on Learning Representations. ↗
  19. Abhijit Mishra, Tarun Tater, and Karthik Sankaranarayanan. 2019. A modular architecture for unsupervised sarcasm generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6144–6154, Hong Kong, China. Association for Computational Linguistics. ↗
  20. D. C. Muecke. 1970. Irony and the Ironic. Methuen, London. find
  21. Silviu Vlad Oprea, Steven Wilson, and Walid Magdy. 2022. Should a chatbot be sarcastic? understanding user preferences towards sarcasm generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7686–7700, Dublin, Ireland. Association for Computational Linguistics.
  22. Valeria Pfeifer and Penny Pexman. 2023. When it pays to be insincere: On the benefits of verbal irony. Current Directions in Psychological Science, 33. ↗
  23. Antonio Reyes, Paolo Rosso, and Tony Veale. 2013. A multidimensional approach for detecting irony in twitter. Language Resources and Evaluation, 47:239– 268. ↗
  24. Iulian Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of the AAAI conference on artificial intelligence, volume 30. ↗
  25. Yufei Tian, Divyanshu Sheth, and Nanyun Peng. 2022. A unified framework for pun generation with humor principles. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3253– 3261, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  26. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models. Preprint, arXiv:2307.09288. ↗
  27. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. SemEval-2018 task 3: Irony detection in English tweets. In Proceedings of the 12th International Workshop on Semantic Evaluation, pages 39–50, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  28. Yuwei Wu, Xuezhe Ma, and Diyi Yang. 2021. Personalized response generation via generative split memory network. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1956–1970, Online. Association for Computational Linguistics. find
  29. Qingcheng Zeng and An-Ran Li. 2022. A survey in automatic irony processing: Linguistic, cognitive, and multi-X perspectives. In Proceedings of the 29th International Conference on Computational Linguistics, pages 824–836, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. ↗
  30. Mengdi Zhu, Zhiwei Yu, and Xiaojun Wan. 2019. A neural approach to irony generation. ArXiv, abs/1909.06200. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/balestrucci-2024-i-m-sure-you-re/