Elamur Research⌕
☰
home / papers / 2024 / Human and System Perspectives on the Expression of Irony: An Analysis of Likelihood Labels and Rationales

Human and System Perspectives on the Expression of Irony: An Analysis of Likelihood Labels and Rationales

Paper · 2024 · Aaron Maladry, Alessandra Teresa Cignarella, Els Lefever et al.

Summary

Irony detection usually treats a tweet as simply ironic or not, even though irony is ambiguous by nature and depends on shared background knowledge. The authors revisit the English SemEval-2018 task 3 tweet corpus. They found that the original labels had been assigned while irony hashtags were still visible, but systems were trained and tested on tweets with those hashtags removed. To fix this mismatch, they re-annotated the formerly ironic tweets without the hashtags. Annotators gave each tweet a 7-point irony likelihood score, which combines the yes/no decision with a confidence level. For tweets they judged ironic, they also marked the trigger words behind that judgement, following guidelines grounded in irony theory. Agreement was measured on 200 tweets annotated by four people. The authors then compared fine-tuned transformer classifiers against these labels and compared the models' token attributions, computed with Layer Integrated Gradients, against the human trigger spans. Models performed better on tweets that annotators labelled confidently and consistently. They located triggers reasonably well but still placed much importance on irrelevant tokens. The resource makes it possible to evaluate both what a model decides and why.

Key points

  • Data: the SemEval-2018 task 3 corpus of English tweets from Dec 2014 to Jan 2015, collected via #irony, #sarcasm and #not. Earlier work showed that 19% of hashtagged tweets were not actually ironic, and that in 52% of ironic ones the hashtag was needed to see the irony.
  • Only tweets previously marked ironic were re-labelled, with the irony hashtags stripped. Earlier "not ironic" labels were kept as they were.
  • Annotation team and scheme:
    • Three linguistics students plus the principal investigator annotated the data.
    • A 7-point Likert scale was used.
    • Trigger spans could be marked only when the likelihood score was 5 or higher.
    • Punctuation such as "?!" or "!!!" was allowed as a trigger.
  • Agreement on 200 tweets was measured with Fleiss' Kappa, Krippendorff's Alpha, pairwise Cohen's Kappa and AC2 (via irrCAC):
    • It was moderate to good at 7 points.
    • It dropped as labels were merged into coarser scales.
    • For a 3-way scale, grouping labels 3–5 as "doubt" worked best.
    • For a binary scale, counting "not sure" (4) as non-ironic was advised.
  • Model rationales were obtained with Layer Integrated Gradients and compared with the human trigger spans.

Interesting ideas

  • I1. Merge a binary decision and a confidence level into one ordinal scale, then collapse it to coarser granularities afterwards and check which merge keeps agreement highest.
  • I2. Strip artefacts that leaked label information (here, the hashtags) before annotating, so that humans and systems see exactly the same input.
  • I3. Ask annotators not just for a label but for the minimal span that justifies it, keeping phrases intact and dropping redundant parts.
  • I4. Use human rationale spans as a reference to score whether a model's importance attributions land on the right tokens, separately from accuracy.
  • I5. Break down model performance by annotator confidence and agreement, so that hard, ambiguous items are separated from genuine errors.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): the irony detector in prag could output a graded likelihood instead of a flag. Downstream implicature could then act only on high-confidence readings and keep doubtful ones as hedged alternatives.
  • I2 (learn): when building silver data or asking LLM teachers, give the witnesses precisely the text Mova will see. That prevents gold labels from relying on cues the system never gets.
  • I3 (frames): trigger spans can be mapped onto UD subtrees, giving explicit evidence nodes for an irony or contrast frame. This fits Mova's requirement that every step be explainable.
  • I4 (learn): evaluation could check that rule firings or parser decisions cite the same evidence a human would, acting as a control against right-for-wrong-reasons behaviour.
  • I5 (answer): test sets for the readers could be stratified by human certainty, so that failures on contested items are not weighted like failures on clear ones.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksSocial Psychology · 306 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksSentiment & emotion · 972 works

Cite

Cite as Aaron Maladry, Alessandra Teresa Cignarella, Els Lefever, Cynthia van Hee and Veronique Hoste. 2024. Human and System Perspectives on the Expression of Irony: An Analysis of Likelihood Labels and Rationales. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{maladry2024human,
  title = {{Human and System Perspectives on the Expression of Irony: An Analysis of Likelihood Labels and Rationales}},
  author = {Aaron Maladry, Alessandra Teresa Cignarella, Els Lefever, Cynthia {van Hee} and Veronique Hoste},
  booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
  year = {2024},
  doi = {10.63317/25ahfv7udkj8},
  url = {https://doi.org/10.63317/25ahfv7udkj8}
}

References

36 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, and Isabelle Augenstein. 2020. A diagnostic study of explainability techniques for text classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3256–3274, Online. Association for Computational Linguistics. ↗
  2. Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa Anke, and Leonardo Neves. 2020. TweetEval: Unified benchmark and comparative evaluation for tweet classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1644–1650, Online. Association for Computational Linguistics.
  3. Steven M Beitzel, Eric C Jensen, and Ophir Frieder. 2009. Map. Encyclopedia of Database Systems, pages 1691–1692. find
  4. Emily M Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604. ↗
  5. Penelope Brown and Steven C Levinson. 1987. Politeness: Some Universals in Language Usage. Politeness: Some Universals in Language Usage. Cambridge University Press.
  6. MY Cai, Y Lin, and Wen-Jun Zhang. 2016. Study of the optimal number of rating bars in the likert scale. In Proceedings of the 18th international conference on information integration and web-based applications and services, pages 193– 198. find
  7. Noam Chomsky. 1973. Conditions on transformations. In A festschrift for Morris Halle, New York. Holt, Rinehart & Winston. find
  8. Alessandra Teresa Cignarella, Cristina Bosco, Viviana Patti, and Mirko Lai. 2018. Application and analysis of a multi-layered scheme for irony on the Italian Twitter Corpus TWITTIRÒ. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
  9. Alessandra Teresa Cignarella, Cristina Bosco, Viviana Patti, et al. 2017. Twittiro: a social media corpus with a multi-layered annotation for irony. In CEUR Workshop Proceedings, volume 2006, pages 1–6. CEUR. find
  10. Alessandra Teresa Cignarella, Manuela Sanguinetti, Cristina Bosco, Rosso Paolo, et al. 2020. Marking Irony Activators in a Universal Dependencies Treebank: The Case of an Italian Twitter Corpus. In Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5098–5105. European Language Resources Association (ELRA). find
  11. Alessandra Teresa Cignarella, Manuela Sanguinetti, Cristina Bosco, Paolo Rosso, et al. 2019. Is This an Effective Way to Annotate Irony Activators? In Proceedings of the 6th Italian Conference on Computational Linguistics (CLiC-it 2019), volume 2481, pages 1–6. CEUR-WS. find
  12. Rebecca Clift. 1999. Irony in conversation. Language in society, 28(4):523–553. ↗
  13. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. ↗
  14. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. ↗
  15. Kilem L Gwet. 2011. On the krippendorff’s alpha coefficient. Manuscript submitted for publication. Retrieved October, 2(2011):2011. find
  16. Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear. find
  17. Hugo Hutt, Richard Everson, Murray Grant, John Love, and George Littlejohn. 2013. How clumpy is my image? Evaluating crowdsourced annotation tasks. In 2013 13th UK Workshop on Computational Intelligence (UKCI), pages 136–143. IEEE. ↗
  18. Gilles Jacobs and Véronique Hoste. 2022. Sentivent: enabling supervised information extraction of company-specific events in economic and financial news. Language Resources and Evaluation, 56(1):225–257. ↗
  19. Aditya Joshi, Pushpak Bhattacharyya, and Mark J Carman. 2017. Automatic sarcasm detection: A survey. ACM Computing Surveys (CSUR), 50(5):1–22. ↗
  20. Jihen Karoui, Farah Benamara, Véronique Moriceau, Viviana Patti, Cristina Bosco, and Nathalie Aussenac-Gilles. 2017. Exploring the impact of pragmatic phenomena on irony detection in tweets: A multilingual corpus study. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 262–272.
  21. Klaus Krippendorff. 2011. Computing krippendorff’s alpha-reliability. Computing, 1:25–2011. find
  22. Grace E Lee and Aixin Sun. 2019. A study on agreement in pico span annotations. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1149–1152. find
  23. Tyler Loakman, Aaron Maladry, and Chenghua Lin. 2023. The iron(ic) melting pot: Reviewing human evaluation in humour, irony and sarcasm generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6676–6689, Singapore. Association for Computational Linguistics.
  24. Aaron Maladry, Els Lefever, Cynthia Van Hee, and Veronique Hoste. 2023a. A fine line between irony and sincerity: Identifying bias in transformer models for irony detection. In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, pages 315–324. find
  25. Aaron Maladry, Els Lefever, Cynthia Van Hee, and Véronique Hoste. 2023b. The limitations of irony detection in dutch social media. Language Resources and Evaluation, pages 1–32. ↗
  26. Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. BERTweet: A pre-trained language model for English Tweets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 9–14.
  27. Pradeesh Parameswaran, Andrew Trotman, Veronica Liesaputra, and David Eyers. 2021. Detecting the target of sarcasm is hard: Really?? Information Processing & Management, 58(4):102599. ↗
  28. Ellen Riloff, Ashequl Qadir, Prafulla Surve, Lalindra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 704–714.
  29. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In International conference on machine learning, pages 3319–3328. PMLR. ↗
  30. Werner Vach and Oke Gerke. 2023. Gwet’s ac1 is not a substitute for cohen’s kappa – a comparison of basic properties. MethodsX, 10:102212. ↗
  31. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2016. Exploring the realization of irony in twitter data. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 1794–1799.
  32. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018a. SemEval-2018 task 3: Irony detection in English tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 39–50, New Orleans, Louisiana. Association for Computational Linguistics. Cynthia Van Hee, Els.
  33. Lefever, and Véronique Hoste. 2018b. We usually don’t like going to the dentist : using common sense to detect irony on twitter. Computational Linguistics, 44(4):793– 832.
  34. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018c. We usually don’t like going to the dentist: Using common sense to detect irony on twitter. Computational Linguistics, 44(4):793– 832. ↗
  35. Byron C Wallace, Laura Kertz, Eugene Charniak, et al. 2014. Humans require context to infer ironic intent (so computers probably do, too). In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 512–516.
  36. Wilson and Sperber. 2012. Explaining irony. Meaning and relevance, pages 123–145. Language Resource References find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/maladry-2024-human-system-perspectives-expression-irony/