Elamur Research⌕
☰
home / papers / 2026 / Investigating How LLMs Propagate Female Stereotypes: Comparing What Models Say via Prompts with What They Represent in Their Embeddings

Investigating How LLMs Propagate Female Stereotypes: Comparing What Models Say via Prompts with What They Represent in Their Embeddings

Summary

The paper asks whether gender stereotypes stored inside language models show up in the same way in the text those models produce, and whether instruction tuning changes that link. The authors compare BERT-base, base LLaMA-2-7b and instruction-tuned LLaMA-2-7b-Chat. They merge the GEST dataset with the gender portion of StereoSet and focus on seven stereotype categories about women, such as emotional, caring, weak and beautiful. Male categories serve as controls.

Internal bias is measured with Directional Embedding Probing, which projects sentence embeddings onto a gender direction taken from a chosen layer. Output bias is measured with masked pronoun prediction for BERT and with prompts asking the LLaMA models to name a character. The two signals agree most for LLaMA-2-Chat, both in its outputs and in a significant correlation. BERT agrees less, and the base LLaMA-2 is mixed.

The authors conclude that instruction tuning goes along with closer agreement between what a model encodes and what it outputs. They also find that prompt wording strongly affects whether hidden bias becomes visible. Fairness audits should therefore check both levels.

Key points

  • Data: GEST (3,565 sentences, 16 stereotype categories) was merged with 242 StereoSet gender sentences, which were manually mapped to the GEST categories. The filtered corpus has 3,658 sentences.
  • Main results: LLaMA-2-Chat outputs were female-aligned in 60–100% of prompts, with embedding–output correlation r = 0.55 (p = 0.0008). For BERT the figures were 0–60% and r = 0.39 (p = 0.054), which is not significant.
  • Probing layers: layer 6 for BERT, chosen by the fewest unintended female scores on male controls. For base LLaMA-2 and LLaMA-2-Chat the authors chose layers 28 and 32, where male and female pronouns were furthest apart.
  • Small samples: each category has only 5 prompts, so the 95% Wilson confidence intervals are wide. Some base LLaMA-2 generations were incoherent.
  • Embedding structure: t-SNE plots show female and male stereotype sentences heavily overlapping. LLaMA-2-Chat shows only slightly more local structure.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/48i8c4bmbog3

@inproceedings{nunez2026investigating,
  title = {{Investigating How LLMs Propagate Female Stereotypes: Comparing What Models Say via Prompts with What They Represent in Their Embeddings}},
  author = {Nuñez, Andrea Valderrey and Bloem, Jelke},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/48i8c4bmbog3},
  url = {https://doi.org/10.63317/48i8c4bmbog3}
}

References

45 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marion Bartl, Malvina Nissim, and Albert Gatt. 2020. Unmasking contextual stereotypes: Measuring and mitigating BERT’s gender bias. In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, pages 1–16, Barcelona, Spain (Online). Association for Computational Linguistics. find
  2. Christine Basta, Marta R. Costa-jussà, and Noe Casas. 2019. Evaluating the underlying gender bias in contextualized word embeddings. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 33–39, Florence, Italy. Association for Computational Linguistics. find our card
  3. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT), pages 610–623. ACM. find
  4. Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021. Stereotyping Norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint… find
  5. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. Advances in neural information processing systems, 29. find our card
  6. Douglas G. Bonett. 2020. Point-biserial correlation: Interval estimation, hypothesis testing, meta-analysis, and sample size determination. British Journal of Mathematical and Statistical Psychology, 73(Suppl 1):113–144. find
  7. Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. find
  8. Yuen Chen, Vethavikashini Chithrra Raghuram, Justus Mattern, Rada Mihalcea, and Zhijing Jin. 2025. Causally testing gender bias in LLMs: A case study on occupational bias. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 4999–5019, Albuquerque, New Mexico. Association… find
  9. Demografix ApS. Genderize.io: Name-to-gender prediction api. https://genderize.io/. Accessed: 2025-06-27. ↗
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL-HLT. find
  11. Fangyu Dong, Pavan Ammanamanchi, Ying Zhang, Hwaran Kim, Sebastian Riedel, and Pontus Stenetorp. 2024. Disclosure and mitigation of gender bias in LLMs. find
  12. Keith Dunnigan. 2008. Confidence interval calculation for binomial proportions. In MWSUG Conference. find
  13. Constanza Fierro, Jiaang Li, and Anders Søgaard. 2024. Does instruction tuning make LLMs more consistent? arXiv preprint arXiv:2404.15206. — [arXiv:2404.15206](https://arxiv.org/abs/2404.15206) find
  14. Wei Guo and Aylin Caliskan. 2021. Detecting emergent intersectional biases: Contextualized word embeddings contain a distribution of human-like biases. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’21, page 122–133, New York, NY, USA. Association for Computing Mach… find
  15. Patrick Haller, Ansar Aynetdinov, and Alan Akbik. 2024. OpinionGPT: Modelling explicit biases in instruction-tuned LLMs. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: System Demonstrations)… find
  16. Aurélie Herbelot, Eva von Redecker, and Johanna Müller. 2012. Distributional techniques for philosophical enquiry. In Proceedings of the 6th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 45–54, Avignon, France. Association for Computational Linguistics… find
  17. Hugging Face. 2023. LLaMA 2 on Hugging Face: State-of-the-art open-access language models. https://huggingface.co/blog/llama2. Accessed: 2025-04-22. ↗
  18. Itay Itzhak, Gabriel Stanovsky, Nir Rosenfeld, and Yonatan Belinkov. 2024. Instructed to bias: Instruction-tuned language models exhibit emergent cognitive bias. Transactions of the Association for Computational Linguistics, 12:771–785. find
  19. Shirin R. Kapoor and Arvind Narayanan. 2023. Quantifying ChatGPT’s gender bias. AI Snake Oil Blog, accessed April 17, 2025. find
  20. Hadas Kotek, Rikker Dockum, and David Sun. 2023. Gender bias and stereotypes in large language models. In Proceedings of The ACM Collective Intelligence Conference (CI ’23), pages 12–24, Delft, Netherlands. Association for Computing Machinery. find
  21. Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. Measuring bias in contextualized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 166–172, Florence, Italy. Association for Computational Linguistics. find our card
  22. Hongyi Li, Jiawei Ye, Jie Wu, Tianjie Yan, Chu Wang, and Zhixin Li. 2025. JailPO: A novel black-box jailbreak framework via preference optimization against aligned LLMs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 27419–27427. find
  23. Jiarui Liu, Jivitesh Jain, Mona T. Diab, and Nishant Subramani. 2025. LLM microscope: What model internals reveal about answer correctness and context utilization. In The First Workshop on the Interplay of Model Behavior and Model Internals. find
  24. Xueguang Lu, Max Bartolo, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786. find our card
  25. Li Lucy and David Bamman. 2021. Gender and representation bias in GPT-3 generated stories. In Proceedings of the Third Workshop on Narrative Understanding, pages 48–55, Virtual. Association for Computational Linguistics. find our card
  26. Kristian Lum, Jacy Reese Anthis, Kevin Robinson, Chirag Nagpal, and Alexander Nicholas D’Amour. 2025. Bias in language models: Beyond trick tests and towards RUTEd evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13… find
  27. Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and… find our card
  28. Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereotypical bias in pre-trained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL), pages 5356–5371. find
  29. Hadas Orgad, Seraphina Goldfarb-Tarrant, and Yonatan Belinkov. 2022. How gender debiasing affects internal model representations, and why it matters. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pag… find
  30. Iñigo Parra. 2024. UnMASKed: Quantifying gender biases in masked language models through linguistically informed job market prompts. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 61–70, St. Julian’s, M… find
  31. Matúš Pikuliak, Stefan Oresko, Andrea Hrckova, and Marian Simko. 2024. Women are beautiful, men are leaders: Gender stereotypes in machine translation and language modeling. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3060–3083, Miami, Florida, USA. Association fo… find
  32. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. arXiv preprint arXiv:1908.10084. find
  33. Gerard Salton and Michael J. McGill. 1983. Introduction to Modern Information Retrieval. McGraw-Hill. find
  34. Aleix Sant, Carlos Escolano, Audrey Mash, Francesca De Luca Fornaciari, and Maite Melero. 2024. The power of prompts: Evaluating and mitigating gender bias in MT with LLMs. In Proceedings of the 5th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 94– 139, Bangkok, Thailand. As… find
  35. Carolin M. Schuster, Maria-Alexandra Roman, Shashwat Ghatiwala, and Georg Groh. 2025. Profiling bias in LLMs: Stereotype dimensions in contextual word embeddings. In Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies… find
  36. Preethi Seshadri, Pouya Pezeshkpour, and Sameer Singh. 2022. Quantifying social biases using templates is unreliable. In NeurIPS Workshop on Trustworthy and Socially Responsible Machine Learning (TSRML). find
  37. McNair Shah, Saleena Angeline Sartawita, Adhitya Rajendra Kumar, Naitik Chheda, Kevin Zhu, Vasu Sharma, Sean O’Brien, and Will Cai. 2025. Death by a thousand directions: Exploring the geometry of harmfulness in LLMs through sub-concept probing. In The First Workshop on the Interplay of Model Behavio… find
  38. Shweta Soundararajan and Sarah Jane Delany. 2024. Investigating gender bias in large language models through text generation. In Proceedings of the 7th International Conference on Natural Language and Speech Processing (IC-NLSP 2024), pages 410–424, Trento. Association for Computational Linguistics. find
  39. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593–4601, Florence, Italy. Association for Computational Linguistics. find our card
  40. Marion Thaler, Abdullatif Köksal, Alina Leidinger, Anna Korhonen, and Hinrich Schütze. 2024. How far can bias go? – Tracing bias from pretraining data to alignment. find
  41. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. find
  42. Yishan Wang, Pia Sommerauer, and Jelke Bloem. 2025. The negation bias in large language models: Investigating bias reflected in linguistic markers. In Second Conference on Language Modeling. find
  43. Zeping Yu and Sophia Ananiadou. 2025. Understanding and mitigating gender bias in LLMs via interpretable neuron editing. arXiv preprint arXiv:2501.14457. — [arXiv:2501.14457](https://arxiv.org/abs/2501.14457) find
  44. Yachao Zhao, Bo Wang, Yan Wang, Dongming Zhao, Xiaojia Jin, Jijun Zhang, Ruifang He, and Yuexian Hou. 2024. A comparative study of explicit and implicit gender biases in large language models via self-evaluation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics,… find
  45. Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2023. LIMA: Less is more for alignment. Advances in Neural Information Processing Systems, 36:55006–55021. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nunez-2026-investigating-how-llms-propagate-female/