Elamur Research⌕
☰
home / papers / 2022 / Tackling Irony Detection using Ensemble Classifiers

Tackling Irony Detection using Ensemble Classifiers

Paper · 2022 · Christoph Turban and Udo Kruschwitz

Summary

The paper tackles automatic irony detection in English tweets. This task stays hard because an ironic message often means the reverse of its literal wording, and the cue sits outside the text. The authors pursue two directions. First, they build ensemble classifiers from several fine-tuned transformer models. Second, they test whether automatically augmenting the scarce training data helps. All experiments use the SemEval 2018 Task 3 benchmark, which has both a binary ironic/not-ironic subtask and a fine-grained multi-class subtask. On the multi-class setting, the ensembles beat strong baselines with statistically significant gains. On the binary setting, they match the state of the art. The augmentation strategies changed results somewhat but did not decide performance. Code and detailed results are released on GitHub. The work matters because irony flips sentiment, appears in hateful and abusive content, and challenges chatbots. The multi-class case is closest to real applications, which need to tell harmless situational irony apart from insulting sarcasm.

Key points

  • Benchmark: the SemEval 2018 Task 3 shared task, which targets spotting irony in English-language tweets. It uses the annotation scheme of Van Hee (2017), which has four labels: irony by polarity clash, situational irony, other irony, and not ironic.
  • There are two research questions. One asks whether ensembles of fine-tuned transformers beat state-of-the-art baselines. The other asks whether data augmentation improves classification.
  • Multi-class results show statistically significant improvement over strong baselines. Binary results are on par with the best existing systems.
  • Data augmentation had a measurable but non-decisive effect.
  • Motivation cites Sanguinetti et al. (2018), who found that 11% of hateful Italian tweets express hate through irony. The annotation studies behind the dataset also showed that human irony judgements are quite subjective.

Interesting ideas

  • I1. Treat irony as a typed phenomenon (polarity clash vs. situational vs. other) rather than a yes/no flag, since applications care which kind it is.
  • I2. The "Great game" example shows that irony detection means reversing a word's literal polarity using knowledge of the situation (the team lost).
  • I3. Combine several independently fine-tuned models into one ensemble, because diverse models err differently and their joint decision is more robust.
  • I4. Pose data augmentation as its own research question and measure its effect separately, rather than assuming it helps; here the gains were small.
  • I5. Report statistical significance against strong baselines, especially on the harder fine-grained task where differences are subtle.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): the irony module in prag could emit a type label. A polarity clash would be flagged when the literal evaluation in a frame contradicts the speaker's evident stance.
  • I2 (world): check positive evaluative predicates against the world state of the text. A clash between "great" and a recorded loss event then becomes an explainable irony signal instead of a black-box score.
  • I3 (learn): this mirrors the witness-parser setup. Irony or speech-act silver labels could be accepted only where several differently built classifiers agree.
  • I4 (learn): run synthetic or LLM-generated training data as an explicit ablation with negative controls. Keep it only if it measurably beats the unaugmented baseline.
  • I5 (learn): add significance tests to evaluation runs, so that small gains on irony or pragmatics sets are not mistaken for real progress.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: University of Regensburg.

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.English · 1,373 worksSentiment & emotion · 608 works

Cite

Cite as Christoph Turban and Udo Kruschwitz. 2022. Tackling Irony Detection using Ensemble Classifiers. Proceedings of the Thirteenth Language Resources and Evaluation Conference.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{turban2022tackling,
  title = {{Tackling Irony Detection using Ensemble Classifiers}},
  author = {Christoph Turban and Udo Kruschwitz},
  booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation Conference},
  year = {2022},
  doi = {10.63317/3ka8ogee7ssv},
  url = {https://doi.org/10.63317/3ka8ogee7ssv}
}

References

54 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aggarwal, C. C. and Zhai, C. (2012). Mining text data. Springer Science & Business Media. Not ironic - 0.843 0.704 - 0.847 0.835 0.820 Irony by clash - 0.697 0.608 - 0.782 0.764 0.764 Situational irony - 0.376 0.433 - 0.591 0.599 0.617 Other - 0.114 0.233 - 0.141 0.244 0.190 ↗
  2. Attardo, S. (2007). Irony as relevant inappropriateness. Irony in language and thought, pages 135– 174. find
  3. Barbe, K. (1995). Irony in context, volume 34. John Benjamins Publishing. find
  4. Basile, V., Novielli, N., Croce, D., Barbieri, F., Nissim, M., and Patti, V. (2021). Sentiment Polarity Classification at EVALITA: Lessons Learned and Open Challenges. IEEE Trans. Affect. Comput., 12(2):466–478. find
  5. Baziotis, C., Nikolaos, A., Papalampidi, P., Kolovou, A., Paraskevopoulos, G., Ellinas, N., and Potamianos, A. (2018). NTUA-SLP at SemEval-2018 task 3: Tracking ironic tweets using ensembles of word and character level attentive RNNs. pages 613–621, June.
  6. Bowes, A. and Katz, A. (2011). When sarcasm stings. Discourse Processes, 48(4):215–236. ↗
  7. Bueno, R. O., Pardo, F. M. R., Farı́as, D. I. H., Rosso, P., Montes-y-Gómez, M., and Medina-Pagola, J. (2019). Overview of the Task on Irony Detection in Spanish Variants. In IberLEF@SEPLN, volume 2421 of CEUR Workshop Proceedings, pages 229– 256. CEUR-WS.org. find
  8. Carvalho, P., Sarmento, L., Silva, M. J., and De Oliveira, E. (2009). Clues for detecting irony in user-generated contents: oh...!! it’s” so easy”;-. In Proceedings of the 1st international CIKM workshop on Topic-sentiment analysis for mass opinion, pages 53–56. find
  9. Cervone, A., Stepanov, E. A., Celli, F., and Riccardi, G. (2017). Irony detection: from the twittersphere to the news space. In CLiC-it 2017-Italian Conference on Computational Linguistics, volume 2006. find
  10. Cignarella, A. T., Frenda, S., Basile, V., Bosco, C., Patti, V., Rosso, P., et al. (2018). Overview of the EVALITA 2018 task on Irony Detection in Italian tweets (IronITA). In Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian (EVALITA 2018), volume 2263, pages 1–6. CEUR-WS. ↗
  11. Clark, H. H. and Gerrig, R. J. (1984). On the pretense theory of irony. American Psychological Association. ↗
  12. Colebrook, C. (2002). Irony in the Work of Philosophy. U of Nebraska Press. find
  13. Corrêa, U. B., Coelho, L., Santos, L., and de Freitas, L. A. (2021). Overview of the IDPT task on irony detection in portuguese at iberlef 2021. Proces. del Leng. Natural, 67:269–276. ↗
  14. Croes, E. A. and Antheunis, M. L. (2021). Can we be friends with Mitsuku? A longitudinal study on the process of relationship formation between humans and a social chatbot. Journal of Social and Personal Relationships, 38(1):279–300. find
  15. Dekker, K. and van der Goot, R. (2020). Synthetic data for English lexical normalization: How close can we get to manually annotated data? In Proceedings of the 12th Language Resources and Evaluation Conference, pages 6300–6309, Marseille, France, May. European Language Resources Association.
  16. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171– 4186, Minneapolis, Minnesota, June. Association for Computational Linguistics. ↗
  17. Farı́as, D. I. H., Patti, V., and Rosso, P. (2016). Irony detection in Twitter: The role of affective content. ACM Transactions on Internet Technology (TOIT), 16(3):1–24. find
  18. Feldman, R. (2013). Techniques and applications for sentiment analysis. Communications of the ACM, 56(4):82–89. ↗
  19. Floridi, L. and Chiriatti, M. (2020). Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30(4):681–694. ↗
  20. Ghosh, A. and Veale, T. (2018). Ironymagnet at semeval-2018 task 3: A siamese network for irony detection in social media. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 570–575.
  21. Ghosh, D., Vajpayee, A., and Muresan, S. (2020). A report on the 2020 sarcasm detection shared task. arXiv preprint arXiv:2005.05814. ↗
  22. González, J.-Á., Hurtado, L.-F., and Pla, F. (2019). Elirf-upv at irosva: Transformer encoders for spanish irony detection. In IberLEF@ SEPLN, pages 278–284. find
  23. González-Ibánez, R., Muresan, S., and Wacholder, N. (2011). Identifying sarcasm in Twitter: a closer look. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 581–586.
  24. Hagen, M., Potthast, M., Büchner, M., and Stein, B. (2015). Webis: An ensemble for Twitter sentiment detection. In Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015), pages 582–589. find
  25. Juuti, M., Gröndahl, T., Flanagan, A., and Asokan, N. (2020). A little goes a long way: Improving toxic language classification despite data scarcity. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2991–3009, Online, November. Association for Computational Linguistics. ↗
  26. Karoui, J., Benamara, F., Moriceau, V., Patti, V., Bosco, C., and Aussenac-Gilles, N. (2017). Exploring the impact of pragmatic phenomena on irony detection in tweets: A multilingual corpus study. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers. Association for Computational Linguistics (ACL).
  27. Kreuz, R. and Caucci, G. (2007). Lexical influences on the perception of sarcasm. In Proceedings of the Workshop on computational approaches to Figurative Language, pages 1–4.
  28. Lee, H., Yu, Y., and Kim, G. (2020). Augmenting data for sarcasm detection with unlabeled conversation context. In Proceedings of the Second Workshop on Figurative Language Processing, pages 12–17, Online, July. Association for Computational Linguistics.
  29. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692. ↗
  30. Liu, B. (2020). Sentiment analysis: Mining opinions, sentiments, and emotions. Cambridge university press. find
  31. Maynard, D. and Bontcheva, K. (2015). Understanding climate change tweets: an open source toolkit for social media analysis. In EnviroInfo/ICT4S (1), pages 242–250. find
  32. Maynard, D. G. and Greenwood, M. A. (2014). Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In LREC 2014 proceedings. ELRA.
  33. Mladenović, M., Ošmjanski, V., and Stanković, S. V. (2022). Cyber-aggression, cyberbullying, and cybergrooming: A survey and research challenges. ACM Computing Surveys, 54(1). find
  34. Nguyen, D. Q., Vu, T., and Nguyen, A. T. (2020). BERTweet: A pre-trained language model for English Tweets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations.
  35. Pak, A. and Paroubek, P. (2010). Twitter as a corpus for sentiment analysis and opinion mining. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10), Valletta, Malta, May. European Language Resources Association (ELRA).
  36. Potamias, R. A., Siolas, G., and Stafylopatis, A.-G. (2020). A transformer-based approach to irony and sarcasm detection. Neural Computing and Applications, 32(23):17309–17320. ↗
  37. Reyes, A., Rosso, P., and Buscaldi, D. (2012). From humor recognition to irony detection: The figurative language of social media. Data & Knowledge Engineering, 74:1–12. ↗
  38. Rohanian, O., Taslimipoor, S., Evans, R., and Mitkov, R. (2018). Wlv at semeval-2018 task 3: Dissecting tweets in search of irony. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 553–559.
  39. Rosenthal, S., Ritter, A., Nakov, P., and Stoyanov, V. (2014). SemEval-2014 Task 9: Sentiment Analysis in Twitter. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 73–80, 01.
  40. Rosenthal, S., Nakov, P., Kiritchenko, S., Mohammad, S. M., Ritter, A., and Stoyanov, V. (2015). SemEval-2015 Task 10: Sentiment Analysis in Twitter. 451-463. In Proc. of the 9th International Workshop on Semantic Evaluation.
  41. Sanguinetti, M., Poletto, F., Bosco, C., Patti, V., and Stranisci, M. (2018). An Italian Twitter corpus of hate speech against immigrants. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
  42. Singh, A., Blanco, E., and Jin, W. (2019). Incorporating emoji descriptions improves tweet classification. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2096– 2101. find
  43. Sulis, E., Irazú Hernández Farı́as, D., Rosso, P., Patti, V., and Ruffo, G. (2016). Figurative messages and affect in twitter. Know.-Based Syst., 108(C):132–143, sep. find
  44. Tran, H. N. and Kruschwitz, U. (2021). ur-iw-hnt at GermEval 2021: An ensembling strategy with multiple BERT models. In Proceedings of the GermEval 2021 Shared Task on the Identification of Toxic, Engaging, and Fact-Claiming Comments, pages 83–87, Duesseldorf, Germany, September. Association for Computational Linguistics. ↗
  45. Utsumi, A. (1996). A unified theory of irony and its computational formalization. In COLING 1996 Volume 2: The 16th International Conference on Computational Linguistics.
  46. Van Hee, C., Lefever, E., and Hoste, V. (2016). Guidelines for annotating irony in social media text, version 2.0. LT3 Technical Report Series. find
  47. Van Hee, C., Lefever, E., and Hoste, V. (2018). Semeval-2018 task 3: Irony detection in english tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 39–50.
  48. Van Hee, C. (2017). Can machines sense irony?: exploring automatic irony detection on social media. Ph.D. thesis, Ghent University. find
  49. Wang, P.-Y. A. (2013). #irony or #sarcasm — a quantitative and qualitative study based on Twitter. In Proceedings of the 27th Pacific Asia Conference on Language, Information, and Computation (PACLIC 27), pages 349–356, Taipei, Taiwan, November. Department of English, National Chengchi University.
  50. Whitfield, D. (2021). Using GPT-2 to create synthetic data to improve the prediction performance of NLP machine learning classification models. CoRR, abs/2104.10658. ↗
  51. Wilson, D. and Sperber, D. (1992). On verbal irony. Lingua, 87(1):53–76. ↗
  52. Wu, C., Wu, F., Wu, S., Liu, J., Yuan, Z., and Huang, Y. (2018). THU NGN at SemEval-2018 task 3: Tweet irony detection with densely connected LSTM and multi-task learning. In Proceedings of The 12th International Workshop on Semantic Evaluation, New Orleans, Louisiana, June. Association for Computational Linguistics. find
  53. Wullach, T., Adler, A., and Minkov, E. (2021). Fight fire with fire: Fine-tuning hate detectors using large samples of generated hate speech. In Marie-Francine Moens, et al., editors, Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021, pages 4699–4705. Association for Computational Linguistics. find
  54. Zimmerman, S., Kruschwitz, U., and Fox, C. (2018). Improving hate speech detection with deep learning ensembles. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/turban-2022-tackling-irony-detection-ensemble-classifiers/