Elamur Research⌕
☰
home / papers / 2026 / Team Prisma at GSI:detect: Comparing PB&J Persona-Based and Few-Shot Approaches to Gender Stereotype Detection at EVALITA 2026

Team Prisma at GSI:detect: Comparing PB&J Persona-Based and Few-Shot Approaches to Gender Stereotype Detection at EVALITA 2026

Paper · 2026 · Claudia Zaghi

Summary

This paper reports Team Prisma's entry to GSI:detect at EVALITA 2026, a perspectivist shared task on finding gender stereotypes in short Italian texts drawn from social media and informative websites. The task has two parts. The first is a regression score in [0,1] derived from four annotators' binary labels. The second assigns each text to one of six stereotype categories.

Working in a closed setting with Claude 3.5 Sonnet, the author compares two prompting strategies:

  • Persona-based (zero-shot): the PB&J (Persona-based Judgment) framework builds four synthetic annotators that approximate the real annotators' demographics. Each persona is profiled through Big Five traits, Schwartz values and Primal World Beliefs, and is used alone or in a four-persona vote.
  • Few-shot: a plain prompt with the Italian guidelines and 35 stratified development examples, with no persona framing.

The few-shot system clearly won, placing 20th of 54 runs on regression and first among the 12 teams on classification. The personas behaved differently from one another, but every persona run labelled far fewer texts as stereotypical than the gold data does. The takeaway is that simulated personas can reproduce annotator variety but still need example-based calibration to match task conventions.

Key points

  • Data: about 1,000 Italian texts, split 20% development and 80% test (810 test samples). Each text has four annotators: three female and one male, aged 20–47.
  • Runs: six in total. Run 1 aggregates four personas (Balanced, Traditional, Sensitive, Academic). Runs 2–5 each use a single persona at temperatures from 0.3 to 0.8. Run 6 is the few-shot baseline (5 examples per class, temperature 0.3).
  • Regression results: few-shot reached NMSE 0.7041 (MSE 0.1257). Persona runs scored NMSE 2.04–2.24, which is worse than simply predicting the mean.
  • Classification results: few-shot reached F1 Macro 0.6115, ranking 1/47. Its per-category F1 ranged from 0.84 for physical down to 0.57 for relational. Persona runs scored F1 0.17–0.29, with the ensemble doing best among them.
  • Detection rates: personas flagged stereotypes in only 17.4–29.0% of texts, against 70.1% in the gold data, while few-shot flagged 82.6%. The author notes that the scale mismatch, varying temperatures, and an anomalously low rate for Persona D limit how far the persona differences can be interpreted.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

🧭 Topics 1Research subjects and languages the work is about.Bias, fairness & safety · 587 works

Cite

Cite as Claudia Zaghi. 2026. Team Prisma at GSI:detect: Comparing PB&J Persona-Based and Few-Shot Approaches to Gender Stereotype Detection at EVALITA 2026. Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zaghi2026team,
  title = {{Team Prisma at GSI:detect: Comparing PB\&J Persona-Based and Few-Shot Approaches to Gender Stereotype Detection at EVALITA 2026}},
  author = {Claudia Zaghi},
  booktitle = {Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026)},
  year = {2026},
  url = {https://aclanthology.org/2026.evalita-1.16/}
}

References

18 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. L. Arcuri, M. R. Cadinu, Gli Stereotipi. Dinamiche psicologiche e contesto delle relazioni sociali, Il Mulino, Bologna, 1998. find
  2. M. R. Costa-jussà, An analysis of gender bias studies in natural language processing, Nature Machine Intelligence 1 (2019) 495–496. find
  3. S. L. Blodgett, S. Barocas, H. Daumé III, H. Wallach, Language (technology) is power: A critical survey of “bias” in NLP, in: D. Jurafsky, J. Chai, N. Schluter, J. Tetreault (Eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational… — Language (Technology) is Power: A Critical Survey of “Bias” in NLP
  4. T. Bolukbasi, K. Chang, J. Y. Zou, V. Saligrama, A. Kalai, Man is to computer programmer as woman is to homemaker? debiasing word embeddings, CoRR abs/1607.06520 (2016). URL: http://arxiv.org/abs/1607.06520. arXiv:1607.06520 . ↗
  5. P. Chiril, V. Moriceau, F. Benamara, A. Mari, G. Origgi, M. Coulomb-Gully, An annotated corpus for sexism detection in French tweets, in: N. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, S. Piperidis (… find
  6. J. Zhao, T. Wang, M. Yatskar, V. Ordonez, K.-W. Chang, Gender bias in coreference resolution: Evaluation and debiasing methods, in: M. Walker, H. Ji, A. Stent (Eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tec…
  7. Y. Li, G. Zhang, H. Hong, Y. Wang, C. Lin, Overview of the nlpcc 2025 shared task: Gender bias mitigation challenge, in: X.-L. Mao, Z. Ren, M. Yang (Eds.), Natural Language Processing and Chinese Computing, Springer Nature Singapore, Singapore, 2026, pp. 453–464. find
  8. G. Comandini, M. Speranza, S. Brenna, D. Testa, S. Cavagnoli, B. Magnini, Gsi:detect at evalita 2026: Overview of the task on detecting gender stereotypes in italian, in: Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA… find
  9. F. Cutugno, A. Miaschi, A. P. Aprosio, G. Rambelli, L. Siciliani, M. A. Stranisci, Evalita 2026: Overview of the 9th evaluation campaign of natural language processing and speech tools for italian, in: Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for I… find
  10. L. Aroyo, C. Welty, Truth is a lie: Crowd truth and the seven myths of human annotation, AI Magazine 36 (2015) 15–24. URL: https://doi.org/10.1609/aimag.v36i1.2564. doi: 10.1609/aimag. v36i1.2564 . ↗
  11. A. Uma, T. Fornaciari, A. Dumitrache, T. Miller, J. Chamberlain, B. Plank, E. Simpson, M. Poesio, Learning from disagreement: A survey, Journal of Artificial Intelligence Research 72 (2021) 1385–1470. find
  12. Z. Waseem, Are you a racist or am I seeing things? Annotator influence on hate speech detection on Twitter, in: Proceedings of the First Workshop on NLP and Computational Social Science, Austin, Texas, 2016, pp. 138–142.
  13. M. Y. Guan, V. Gulshan, A. M. Dai, G. E. Hinton, Who said what: Modeling individual labelers improves classification, 2018. URL: https://arxiv.org/abs/1703.08774. arXiv:1703.08774 . — [arXiv:1703.08774](https://arxiv.org/abs/1703.08774) ↗
  14. B. Joshi, X. Ren, S. Swayamdipta, R. Koncel-Kedziorski, T. Paek, Improving language model personas via rationalization with psychological scaffolds, in: C. Christodoulopoulos, T. Chakraborty, C. Rose, V. Peng (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2025, Association… find
  15. F. Cabitza, A. Campagner, V. Basile, Toward a perspectivist turn in ground truthing for predictive computing, Proceedings of the AAAI Conference on Artificial Intelligence 37 (2023) 6860–6868. URL: http://dx.doi.org/10.1609/aaai.v37i6.25840. doi: 10.1609/aaai.v37i6.25840 . ↗
  16. L. R. Goldberg, The structure of phenotypic personality traits, American Psychologist 48 (1993) 26–34. find
  17. S. H. Schwartz, Universals in the content and structure of values: Theoretical advances and empirical tests in 20 countries, Advances in Experimental Social Psychology 25 (1992) 1–65. find
  18. J. D. Clifton, J. D. Baker, C. L. Park, et al., Primal world beliefs, Psychological Assessment 31 (2019) 82–99. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zaghi-2026-team-prisma-gsi-detect-comparing/