Elamur Research⌕
☰
home / papers / 2023 / EPIC: Multi-Perspective Annotation of a Corpus of Irony

EPIC: Multi-Perspective Annotation of a Corpus of Irony

Summary

Irony is subjective: whether a reply reads as ironic depends partly on the reader's cultural background. Most irony datasets still merge annotator judgments into a single label. This paper introduces EPIC (English Perspectivist Irony Corpus), described by the authors as the first irony resource built on data perspectivism, in which every individual annotation is kept rather than aggregated.

EPIC has 3,000 Post–Reply pairs from Reddit and Twitter. They cover five English varieties: Irish, British, American, Indian and Australian. Crowdworkers on Prolific from the same five countries labelled each reply as ironic or not, given its post. The authors examined how origin, age and gender relate to irony judgments and to conversation topics. They then trained perspective-aware classifiers on the labels of particular demographic groups.

Different groups produced noticeably different models. The perspectivist classifiers were generally more confident than a non-perspectivist baseline. On a perspective-based test set, they identified ironic texts with higher precision. The resource therefore supports studying how irony perception differs across groups, and it shows that majority voting can erase minority viewpoints.

Key points

  • Corpus composition: 3,000 pairs, split into 1,500 from Twitter and 1,500 from Reddit, with 600 per English variety (300 per platform).
  • Data sources:
    • Reddit comments came from Pushshift (January 2020 – June 2021) and subreddits such as r/AskReddit, r/CasualUK, r/ireland, r/australia and r/india. Only English pairs were kept, using LangID.
    • Twitter pairs were located by geolocation.
  • Annotation:
    • 76 native-speaker annotators took part (16 from the UK, 15 from each other country), each with a 99% Prolific completion rate.
    • Each instance received five labels, and each annotator labelled 200 items.
    • Average pay was 12.66 GBP per hour.
  • Demographic and topic findings:
    • Majority-vote aggregation would hide some groups' views, for example annotators identifying as Asian.
    • Disagreement depends on topic: it is stronger for Labour, Lifestyle and Politics, and weaker for Arts and Health.
  • Model variation: Disagreement between group-specific models ranged from 3% to 29%. Age showed the largest differences, peaking between boomers and genY. Among countries, the UK and Ireland models diverged most.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

31 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 29 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.acl-long.774

@inproceedings{frenda2023epic,
  title = {{EPIC: Multi-Perspective Annotation of a Corpus of Irony}},
  author = {Frenda, Simona and Pedrani, Alessandro and Basile, Valerio and Lo, Soda Marem and Cignarella, Alessandra Teresa and Panizzon, Raffaella and Marco, Cristina and Scarlini, Bianca and Patti, Viviana and Bosco, Cristina and Bernardi, Davide},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.774},
  url = {https://doi.org/10.18653/v1/2023.acl-long.774}
}

References

31 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gavin Abercrombie, Valerio Basile, Sara Tonelli, Verena Rieser, and Alexandra Uma, editors. 2022. Proceedings of the 1st Workshop on Perspectivist Approaches to NLP @LREC2022. European Language Resources Association, Marseille, France. find
  2. Sohail Akhtar, Valerio Basile, and Viviana Patti. 2019. A new measure of polarization in the annotation of hate speech. In AI*IA 2019 – Advances in Artificial Intelligence, pages 588–603, Cham. Springer International Publishing. find
  3. Sohail Akhtar, Valerio Basile, and Viviana Patti. 2020. Modeling annotator perspective and polarized opinions to improve hate speech detection. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 8(1):151–154. find
  4. Linah Alhaidari, Khaled Alyoubi, and Fahd Alotaibi. 2022. Detecting irony in arabic microblogs using deep convolutional neural networks. International Journal of Advanced Computer Science and Applications, 13(1). ↗
  5. Francesco Barbieri, Valerio Basile, Danilo Croce, Malvina Nissim, Nicole Novielli, and Viviana Patti. 2016. Overview of the Evalita 2016 SENTIment POLarity Classification Task. In Proceedings of 3rd Italian Conference on Computational Linguistics (CLiC-it 2016) & 5th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. CEUR-WS.org. ↗
  6. Valerio Basile, Federico Cabitza, Andrea Campagner, and Michael Fell. 2021a. Toward a perspectivist turn in ground truthing for predictive computing. CoRR, abs/2109.04270. ↗ our card
  7. Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, and Alexandra Uma. 2021b. We need to consider disagreement in evaluation. In Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future, pages 15–21, Online. Association for Computational Linguistics. find our card
  8. Alessandra Teresa Cignarella, Simona Frenda, Valerio Basile, Cristina Bosco, Viviana Patti, and Paolo Rosso. 2018. Overview of the EVALITA 2018 Task on Irony Detection in Italian Tweets (IronITA). In Proceedings of the 6th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian (EVALITA 2018). CEUR-WS.org. ↗ our card
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  10. Elena Filatova. 2012. Irony and Sarcasm: Corpus Generation and Analysis Using Crowdsourcing. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC 2012), pages 392–398. European Language Resources Association. find our card
  11. Simona Frenda, Alessandra Teresa Cignarella, Valerio Basile, Cristina Bosco, Viviana Patti, and Paolo Rosso. 2022. The unbearable hurtfulness of sarcasm. Expert Systems with Applications, 193:116398. ↗
  12. Aniruddha Ghosh, Guofu Li, Tony Veale, Paolo Rosso, Ekaterina Shutova, John Barnden, and Antonio Reyes. 2015. Semeval-2015 Task 11: Sentiment Analysis of Figurative Language in Twitter. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015). ACL. find
  13. Aditya Joshi, Pushpak Bhattacharyya, and Mark J Carman. 2018. Investigations in computational sarcasm. Springer Singapore. ↗
  14. Jihen Karoui, Farah Benamara, Véronique Moriceau, Viviana Patti, Cristina Bosco, and Nathalie Aussenac-Gilles. 2017. Exploring the impact of pragmatic phenomena on irony detection in tweets: A multilingual corpus study. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 262–272, Valencia, Spain. Association for Computational Linguistics. find our card
  15. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. CoRR, abs/1910.13461. ↗ our card
  16. Aaron Maladry, Els Lefever, Cynthia Van Hee, and Veronique Hoste. 2022. Irony Detection for Dutch: a Venture into the Implicit. In Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis, pages 172–181. find
  17. Reynier Ortega-Bueno, Berta Chulvi, Francisco Rangel, Paolo Rosso, and Elisabetta Fersini. 2022. Profiling irony and stereotype spreaders on twitter (irostereo). In CLEF 2022 Working Notes, volume 3180. CEUR-WS. find
  18. Reynier Ortega-Bueno, Francisco Rangel, D Hernández Farıas, Paolo Rosso, Manuel Montes-y Gómez, and José E Medina Pagola. 2019. Overview of the task on irony detection in spanish variants. In Proceedings of the Iberian languages evaluation forum (IberLEF 2019), co-located with 34th conference of the Spanish Society for natural language processing (SEPLN 2019). CEUR-WS. org, volume 2421, pages 229–256. find
  19. Barbara Plank. 2022. The ’problem’ of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP 2022. Association for Computational Linguistics. ↗ our card
  20. Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz. 2021. On releasing annotator-level labels and information in datasets. In Proceedings of the Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop, pages 133–138, Punta Cana, Dominican Republic. Association for Computational Linguistics. find our card
  21. Antonio Reyes, Paolo Rosso, and Davide Buscaldi. 2012. From Humor Recognition to Irony Detection: The Figurative Language of Social Media. Data & Knowledge Engineering, 74:1–12. ↗
  22. Edwin Simpson, Erik-Lân Do Dinh, Tristan Miller, and Iryna Gurevych. 2019. Predicting humorousness and metaphor novelty with Gaussian process preference learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5716–5728, Florence, Italy. Association for Computational Linguistics. find
  23. Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9275–9293, Online. Association for Computational Linguistics. find
  24. Abdel Aziz Taha, Leonhard Hennig, and Petr Knoth. 2022. Confidence estimation of classification based on the distribution of the neural network output layer. arXiv preprint arXiv:2210.07745. ↗
  25. Alexandra Uma, Tommaso Fornaciari, Anca Dumi-trache, Tristan Miller, Jon Chamberlain, Barbara Plank, Edwin Simpson, and Massimo Poesio. 2021. SemEval-2021 task 12: Learning with disagreements. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 338– 347, Online. Association for Computational Linguistics. find
  26. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2016. Exploring the realization of irony in Twitter data. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pages 1794–1799. European Language Resources Association. find our card
  27. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. SemEval-2018 Task 3: Irony Detection in English Tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation (SemEval 2018), pages 39–50. ACL. find our card
  28. Bernard Lewis Welch. 1947. The generalization of ‘student’s’ probem when several different population variances are involved. Biometrika, 34(1-2):28–35. find
  29. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics. find
  30. Rong Xiang, Xuefeng Gao, Yunfei Long, Anran Li, Emmanuele Chersoni, Qin Lu, and Chu-Ren Huang. 2020. Ciron: a New Benchmark Dataset for Chinese Irony Detection. In Proceedings of the 12th Language Resources and Evaluation Conference (LREC 2020). ELRA. find our card
  31. Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019. Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3914–3923, Hong Kong, China. Association for Computational Linguistics. ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/frenda-2023-epic-multi-perspective-annotation-corpus/