Elamur Research⌕
☰
home / papers / 2024 / MultiPICo: Multilingual Perspectivist Irony Corpus

MultiPICo: Multilingual Perspectivist Irony Corpus

Summary

Irony is subjective: whether a reply seems ironic depends on the reader. Most irony datasets still merge annotator judgments into one label, and no earlier resource was both multilingual and disaggregated. The authors fill this gap with MultiPICo, a perspectivist corpus of 18,778 Post-Reply exchanges from Reddit and Twitter, covering 9 languages and 25 regional varieties. Native-speaker crowdworkers on Prolific were balanced by gender and, where possible, by nationality. They decided whether each reply was ironic given its post. Every individual label is kept, together with annotator metadata such as age, ethnicity, and student and employment status. The analysis finds marked disagreement between demographic cohorts, with age and student status splitting judgments most strongly. In several languages, agreement rises when an annotator's nationality matches the variety of the text. The corpus also works as a benchmark: the authors test LLMs on irony detection, compare plain prompts with sociodemographic perspective-taking prompts, and probe model positionality. The resource supports NLP that accounts for cultural and demographic variation instead of assuming one ground truth.

Key points

  • Scale: 18,778 conversations (8,956 from Reddit, 9,822 from Twitter) were annotated by 506 annotators. They produced 94,342 labels, about 5.02 per instance.
  • Language coverage: 5 varieties each of English, Spanish and Arabic, 3 of German, 2 each of French and Portuguese, plus Italian, Dutch and Hindi.
  • Data and quality control:
    • Reddit data came via Pushshift (January 2020 to June 2021) from 26 subreddits. Twitter data came via the Stream API with geolocation.
    • LangID was used for language filtering.
    • Attention checks removed 13 annotators.
    • The Arabic subset could not be balanced because too few annotators were available.
  • Findings: In German, English and Spanish, annotators agree more on texts in the variety of their own country. ChatGPT's judgments align most closely with the youngest annotators. Prompting strategies compared were base, trait-based and data-driven perspectives (e.g., with PolyLM).
  • Availability: The corpus is released on Hugging Face (Multilingual-Perspectivist-NLU/MultiPICo) under CC-BY 4.0.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.acl-long.849

@inproceedings{casola2024multipico,
  title = {{MultiPICo: Multilingual Perspectivist Irony Corpus}},
  author = {Casola, Silvia and Frenda, Simona and Lo, Soda Marem and Sezerer, Erhan and Uva, Antonio and Basile, Valerio and Bosco, Cristina and Pedrani, Alessandro and Rubagotti, Chiara and Patti, Viviana and Bernardi, Davide},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.849},
  url = {https://doi.org/10.18653/v1/2024.acl-long.849}
}

References

37 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sohail Akhtar, Valerio Basile, and Viviana Patti. 2019. A new measure of polarization in the annotation of hate speech. In AI*IA 2019 – Advances in Artificial Intelligence, pages 588–603, Cham. Springer International Publishing. find
  2. Sohail Akhtar, Valerio Basile, and Viviana Patti. 2021. Whose opinions matter? Perspective-aware models to identify opinions of hate speech victims in abusive language detection. arXiv preprint arXiv:2106.15896. — [arXiv:2106.15896](https://arxiv.org/abs/2106.15896) find
  3. Dina Almanea and Massimo Poesio. 2022. ArMIS -the Arabic Misogyny and Sexism Corpus with Annotator Subjective Disagreements. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2282–2291, Marseille, France. European Language Resources Association. find
  4. Lora Aroyo and Chris Welty. 2015. Truth is a lie: Crowd truth and the seven myths of human annotation. AI Magazine, 36(1):15–24. find our card
  5. Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, Alexandra Uma, et al. 2021. We need to consider disagreement in evaluation. In Proceedings of the 1st workshop on benchmarking: past, present and future, pages 15–21. Association for Computationa… find our card
  6. Tilman Beck, Hendrik Schuff, Anne Lauscher, and Iryna Gurevych. 2023. How (not) to use sociodemographic information for subjective nlp tasks. CoRR. find
  7. Emily M Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604. find
  8. Federico Cabitza, Andrea Campagner, and Valerio Basile. 2023. Toward a perspectivist turn in ground truthing for predictive computing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 6860–6868. find our card
  9. Silvia Casola, Soda Lo, Valerio Basile, Simona Frenda, Alessandra Cignarella, Viviana Patti, and Cristina Bosco. 2023. Confidence-based ensembling of perspective-aware models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3496–3507, Singapore. Assoc… find
  10. Myra Cheng, Esin Durmus, and Dan Jurafsky. 2023. Marked personas: Using natural language prompts to measure stereotypes in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1504–1532, Toronto, Canada. Associatio… find our card
  11. Ameet Deshpande, Vishvak Murahari, Tanmay Rajpuro-hit, Ashwin Kalyan, and Karthik Narasimhan. 2023. Toxicity in chatgpt: Analyzing persona-assigned language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, page 1236–1270. find our card
  12. Virginia Dignum. 2023. Responsible Artificial Intelligence: Recommendations and Lessons Learned. In Responsible AI in Africa: Challenges and Opportunities, pages 195–214. Springer International Publishing Cham. find
  13. Maxwell Forbes, Jena D. Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. 2020. Social chemistry 101: Learning to reason about social and moral norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 653–670, Online. Association for Computatio… find
  14. Simona Frenda, Soda Marem Lo, Silvia Casola, Bianca Scarlini, Cristina Marco, Valerio Basile, and Davide Bernardi. 2023a. Does anyone see the irony here? Analysis of perspective-aware model predictions in irony detection. In ECAI 2023 Workshop on Perspectivist Approaches to NLP. find
  15. Simona Frenda, Alessandro Pedrani, Valerio Basile, Soda Marem Lo, Alessandra Teresa Cignarella, Raffaella Panizzon, Cristina Marco, Bianca Scarlini, Viviana Patti, Cristina Bosco, and Davide Bernardi. 2023b. EPIC: Multi-perspective annotation of a corpus of irony. In Proceedings of the 61st Annual M… find our card
  16. Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. 2023. Bias and fairness in large language models: A survey. find our card
  17. EunJeong Hwang, Bodhisattwa Majumder, and Niket Tandon. 2023. Aligning language models to user opinions. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5906– 5919, Singapore. Association for Computational Linguistics. find
  18. Hadas Kotek, Rikker Dockum, and David Sun. 2023. Gender bias and stereotypes in large language models. In Proceedings of The ACM Collective Intelligence Conference, CI ’23, page 12–24, New York, NY, USA. Association for Computing Machinery. find
  19. Elisa Leonardelli, Stefano Menini, Alessio Palmero Aprosio, Marco Guerini, and Sara Tonelli. 2021. Agreeing to disagree: Annotating offensive language datasets with annotators’ disagreement. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, page 10528–10539. find
  20. Elisa Leonardelli, Alexandra Uma, Gavin Abercrombie, Dina Almanea, Valerio Basile, Tommaso Fornaciari, Barbara Plank, Verena Rieser, and Massimo Poesio. 2023. Semeval-2023 task 11: Learning with disagreements (lewidi). In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval… find
  21. Yingji Li, Mengnan Du, Rui Song, Xin Wang, and Ying Wang. 2023. A survey on fairness in large language models. find
  22. Soda Marem Lo and Valerio Basile. 2023. Hierarchical clustering of label-based annotator representations for mining perspectives. In Proceedings of the 2nd Workshop on Perspectivist Approaches to NLP co-located with 26th European Conference on Artificial Intelligence (ECAI 2023), Kraków, Poland, Sep… find
  23. Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. 2022. Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Transactions of the Association for Computational Linguistics, 10:92–110. find our card
  24. Matthias Orlikowski, Paul Röttger, Philipp Cimiano, and Dirk Hovy. 2023. The ecological fallacy in annotation: Modeling human label variation goes beyond sociodemographics. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1017… find
  25. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Train… find
  26. Barbara Plank. 2022. The "problem" of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671–10682. find our card
  27. Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz. 2021. On releasing annotator-level labels and information in datasets. In Proceedings of the Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop, page 133–138. find our card
  28. Pratik Sachdeva, Renata Barreto, Geoff Bacon, Alexander Sahn, Claudia von Vacano, and Chris Kennedy. 2022. The measuring hate speech corpus: Leveraging rasch measurement theory for data perspectivism. In Proceedings of the 1st Workshop on Perspectivist Approaches to NLP @LREC2022, pages 83–94, Marse… find
  29. Sebastin Santy, Jenny Liang, Ronan Le Bras, Katharina Reinecke, and Maarten Sap. 2023. NLPositionality: Characterizing design biases of datasets and models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9080–9102, Toronto, C… find
  30. Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Ling… find our card
  31. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennigho… find
  32. Edwin Simpson, Erik-Lân Do Dinh, Tristan Miller, and Iryna Gurevych. 2019. Predicting humorousness and metaphor novelty with Gaussian process preference learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5716–5728, Florence, Italy. Association… find
  33. Alexandra Uma, Tommaso Fornaciari, Anca Dumi-trache, Tristan Miller, Jon Chamberlain, Barbara Plank, Edwin Simpson, and Massimo Poesio. 2021a. Semeval-2021 task 12: Learning with disagreements. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 338– 347. find
  34. Alexandra Uma, Tommaso Fornaciari, Anca Dumi-trache, Tristan Miller, Jon Chamberlain, Barbara Plank, Edwin Simpson, and Massimo Poesio. 2021b. SemEval-2021 task 12: Learning with disagreements. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 338– 347, O… find
  35. Alexandra N Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021c. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385–1470. find
  36. Ruyuan Wan, Jaehyung Kim, and Dongyeop Kang. 2023. Everyone’s voice matters: Quantifying annotation disagreement using demographic information. In Proceedings of the 37th AAAI Conference on An-rtificial Intelligence - AAAI Special Track on AI for Social Impact. find
  37. Example of task: Message This man is so completely focused towards engineering riots. Reply Gotta stick to your strengths Is the reply ironic? Ironic or Not ironic A.2 In Table 9, we report the characteristics of each sociodemographic cohort. In Tables 10 and 8, we report the details of the sets of… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/casola-2024-multipico-multilingual-perspectivist-irony-corpus/