Elamur Research⌕
☰
home / papers / 2026 / GSI:detect - A Perspectivist Approach to Gender Stereotypes Identification in Italian

GSI:detect - A Perspectivist Approach to Gender Stereotypes Identification in Italian

Paper · 2026 · Davide Testa, Sofia Brenna, Manuela Speranza et al.

Summary

Most NLP work on gender stereotypes treats them as part of sexism or hate speech detection. This leaves out subtle cases, such as stereotypes voiced unintentionally or framed as compliments. Annotation is also usually reduced to a single majority label, even though judging stereotypes is highly subjective. The authors address both gaps with GSI:detect, an Italian dataset of short, naturally occurring texts. The texts were gathered manually from social media comments and informative websites, cover stereotypes about both women and men, and include texts with no stereotypes at all. Four annotators labelled every text. Their individual binary decisions are kept and also averaged into a scalar score, which preserves disagreement and allows evaluation of closed LLMs without access to model probabilities. Baseline experiments with several LLMs show clear performance gaps between models and notable weaknesses overall. This suggests current LLMs may not be reliable for this task. The resource supports linguistic study of everyday stereotyping and serves as a demanding, subjectivity-aware benchmark.

Key points

  • GSI:detect contains 1,010 short Italian texts totalling 52,118 tokens. The texts mix formal and informal registers and show stereotypes at varying levels of prototypicality, including none.
  • Annotation follows a perspectivist approach with four annotators. Disagreement is retained and summarized as the mean of their binary judgments, rather than collapsed into one gold label.
  • The dataset is the reference resource for the GSI:detect shared task at the Evalita 2026 campaign. It is released on GitHub under a Creative Commons NonCommercial-ShareAlike license.
  • Unlike the template-based multilingual stereotype data of Mitchell et al. (2025), the texts were written spontaneously online. The focus is on non-hateful, implicit or ambiguous stereotypes, such as micro-aggressions.
  • LLM baselines vary noticeably across models and expose clear limitations. This raises doubts about using current LLMs for detecting and classifying gender stereotypes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TrentoTrento · Italy87 works · 15,761 citations🎓Sapienza University of RomeRome · Italy60 works · 10,105 citations🔬Fondazione Bruno KesslerTrento · Italy35 works · 43,945 citations🎓Free University of Bozen-BolzanoBolzano · Italy3 works · 344 citations

Also on the byline: Istituto Italiano di Studi Germanici.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksSocial Sciences · 1,954 worksGender Studies · 52 works

🧭 Topics 1Research subjects and languages the work is about.Bias, fairness & safety · 587 works

Cite

Cite as Davide Testa, Sofia Brenna, Manuela Speranza, Gloria Comandini, Stefania Cavagnoli and Bernardo Magnini. 2026. GSI:detect - A Perspectivist Approach to Gender Stereotypes Identification in Italian. Proceedings of the the fifth edition of NLPerspectives.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/373ps8r3346w

@inproceedings{testa2026gsi,
  title = {{GSI:detect - A Perspectivist Approach to Gender Stereotypes Identification in Italian}},
  author = {Davide Testa, Sofia Brenna, Manuela Speranza, Gloria Comandini, Stefania Cavagnoli and Bernardo Magnini},
  booktitle = {Proceedings of the the fifth edition of NLPerspectives},
  year = {2026},
  doi = {10.63317/373ps8r3346w},
  url = {https://doi.org/10.63317/373ps8r3346w}
}

References

26 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. C. Harrison and K. D. Tanner. 2018. Language matters: Considering microaggressions in science. CBE - Life Sciences Education, 17:1–8. find
  2. L. Arcuri and M.R. Cadinu. 1998. Gli Stereotipi. Dinamiche psicologiche e contesto delle relazioni sociali. Il Mulino, Bologna. find
  3. Lynne M. Jackson. 2011. The psychology of prejudice: From attitudes to social action. American Psychological Association. find
  4. V. Basile, F. Cabitza, and A. Campagner. 2023. Toward a perspectivist turn in ground truthing for predictive computing. In Toward a Perspectivist Turn in Ground Truthing for Predictive Computing, pages 6860–6868, Washington DC. Association for the Advancement of Artificial Intelligence.
  5. H. Kirk, W. Yin, B. Vidgen, and P. Röttger. 2023. Semeval-2023 task 10: Explainable detection of online sexism. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), page 2193–2210, Strouds-bourg. Association for Computational Linguistics.
  6. Valerio Basile. 2020. It’s the end of the gold standard as we know it. on the impact of pre-aggregation on the evaluation of highly subjective tasks. In Proceedings of the AIxIA 2020 Discussion Papers Workshop, pages 31–40. CEUR Workshop Proceedings. find
  7. Manfred Klenner, Anne G¨ohring, and Michael Amsler. 2020. Harmonization sometimes harms. In Proceedings of the 5th Swiss Text Analytics Conference (SwissText) & 16th Conference on Natural Language Processing (KONVENS). CEUR Workshop Proceedings. find
  8. Irene Biemmi. 2020. Educazione sessista. Stereotipi di genere nei libri delle elementari. Rosenberg & Sellier, Torino. find
  9. Marco Madeddu, Simona Frenda, Mirko Lai, Viviana Patti, and Valerio Basile. 2023. Dis-aggreghate it corpus: A disaggregated italian dataset of hate speech. In Proceedings of the 9th Italian Conference on Computational Linguistics (CLiC-it 2023), pages 243–250. CEUR Workshop Proceedings. find
  10. Yang Trista Cao, Anna Sotnikova, Hal Daumé III, Rachel Rudinger, and Linda Zou. 2022. Theo-rygrounded measurement of u.s. social stereotypes in english language models. In Proceed-M. find
  11. Mitchell, G. Attanasio, I. Baldini, M. Clinciu, J. Clive, P. Delobelle, M. Dey, S. Hamilton, T. Dill, J. Doughman, R. Dutt, A. Ghosh, J. Zosa Forde, C. Holtermann, L.A. Kaffee, T. Laud, A. Lauscher, R.L. Lopez-Davila, M. Masoud, N. Nangia, A. Ovalle, G. Pistilli, D. Radev, B. Savoldi, V. Raheja, J.… find
  12. G. Peruzzi, V. Bernardini, R. Lombardi, C. Rinaldi, M. Bacio, L. Bainotti, and G. Viggiani. 2019b. Il bias del gender. Durango, Andria. find
  13. L. Plaza, J. Carrillo-de Albornoz, R. Morante, E. Amigó, J. Gonzalo, D. Spina, and P. Rosso. 2023. Overview of exist 2023: sexism identification in social networks. In Advances in Information Retrieval: 45th European Conference on Information Retrieval, ECIR 2023, Dublin, Ireland, April 2–6, 2023, P… find
  14. Qwen-Team. 2025. Qwen3 technical report. find
  15. G. Rizos and B.J. Schuller. 2020. Average jane, where art thou? – recent avenues in efficient machine learning under subjectivity uncertainty. In M.J. Lesot, S. Vieira, M.Z. Reformat, J.P. Carvalho, A. Wilbik, B. Bouchon-Meunier, and R.R. Yager, editors, Information Processing and Management of Unce… find
  16. Benedetta Muscato, Chandana Sree Mala, Marta Marchiori Manerba, Gizem Gezici, and Fosca Giannotti. 2024. An overview of recent approaches to enable diversity in large language models through aligning with human perspectives. In Proceedings of the 3rd Workshop on Perspectivist Approaches to NLP (NLPe… find
  17. OpenAI. 2024a. Gpt-4 technical report.
  18. OpenAI. 2024b. Gpt-4o system card.
  19. OpenAI. 2025. Gpt-5 system card. https://cdn.openai.com/gpt-5-system-card.pdf. Accessed: October 2025. ↗
  20. Anaelia Ovalle, Palash Goyal, Jwala Dhamala, Zachary Jaggers, Kai-Wei Chang, Aram Galstyan, Richard Zemel, and Rahul Gupta. 2023. “i’m fully who i am”: Towards centering transgender and non-binary voices to measure biases in open language generation. In Proceedings of the 2023 ACM Conference on Fair… find
  21. M. Sanguinetti, F. Poletto, C. Bosco, V. Patti, and Stranisci M. 2018. An italian twitter corpus of hate speech against immigrants. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), pages 2798–2805, Paris. European Language Resources Associatio…
  22. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 29971–30004. PMLR.
  23. D.W. Sue. 2010. Microaggressions in everyday life. Race, gender and sexual orientation. John Wiley & Sons, Hoboken. find
  24. Maja Pavlovic and Massimo Poesio. 2024. The effectiveness of LLMs as annotators: A comparative overview and empirical analysis of direct representation. In Proceedings of the 3rd Workshop on Perspectivist Approaches to NLP (NLPerspectives) @ LREC-COLING 2024, pages 100– 110, Torino, Italia. ELRA and…
  25. Zeerak Talat, Hagen Blix, Josef Valvoda, Maya Indira Ganesh, Ryan Cotterell, and Adina Williams. 2022. On the machine learning of ethical judgments from natural language. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu… find
  26. G. Peruzzi, V. Bernardini, and R. Lombardi. 2019a. Le questioni di genere dei giovani. un’indagine M. Wojatzki, T. Horsmann, D. Gold, and T. Zesch. 2018. Do women perceive hate differently: Examining the relationship between hate speech, gender, and agreement judgement. In Proceedings of the 14th co… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/testa-2026-gsi-detect-perspectivist-approach-gender/