Elamur Research⌕
☰
home / papers / 2022 / Theory-Grounded Measurement of U.S. Social Stereotypes in English Language Models

Theory-Grounded Measurement of U.S. Social Stereotypes in English Language Models

Paper · 2022 · Yang Trista Cao, Anna Sotnikova, Hal Daumé III et al.

Summary

Language models absorb stereotypes from their training text, and these associations can harm marginalized groups once systems are deployed widely. Earlier ways of probing for them tend to cover only a few groups or rest on traits chosen ad hoc. The authors instead build on the Agency-Belief-Communion (ABC) model from social psychology. ABC describes how people perceive a social group through 16 opposing trait pairs, grouped into agency/socioeconomic success, conservative–progressive beliefs, and communion/warmth. They probe BERT and RoBERTa with templated prompts that pair many U.S. social groups with these traits. They compare two existing association measures, ILPS and CEAT, with a new one called the sensitivity test (SeT), which draws on ideas from active learning. To check the measures against people, they ran a study collecting group-trait judgments from U.S. participants. SeT matched human judgments best, although overall agreement between models and humans was only moderate. They then applied the method to intersectional identities. The value of the work is a theory-backed, extensible way to audit models for stereotypes across many groups.

Key points

  • The ABC model supplies 16 trait dyads (32 poles) in three dimensions, for example powerless↔powerful, religious↔science-oriented and cold↔warm.
  • The group list spans gender/sexuality, race/ethnicity, religion, socioeconomic status, age, disability, politics and nationality, with many template variations (singular/plural, belief framing, social expectation, trait-first, comparative, and others).
  • For CEAT, each word gets about 1000 sampled Reddit sentences to approximate its contextual embedding.
  • On race/gender intersections, taking Ghavami and Peplau (2013) as ground truth, emergent trait detection reached precision 0.83 and recall 0.65, against 0.72/0.50 for random guessing.
  • In the models, age and political stance tend to dominate other domains, while race and nationality tend to be dominated. Code and data are released under MIT at github.com/TristaCao/U.S_Stereotypes.

Interesting ideas

  • I1. ILPS divides the probability of a trait given the group by the trait's probability when the group slot is masked, which turns raw likelihood into weight of evidence and removes frequency effects.
  • I2. The same group-trait question is asked through many template variants (belief, expectation, comparative, trait-first), so a score does not depend on one particular phrasing.
  • I3. Competing automatic measures are ranked by how well they agree with independently collected human judgments, rather than by internal plausibility.
  • I4. The authors warn that averaging annotators cannot tell a split vote from a consensus, nor a strong rare view from a weak common one.
  • I5. The defaulting effect means unmarked group words silently stand for a prototype (for example, "men" read as a specific default subgroup).

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (world): Affinities between verb classes and argument types in the compiled knowledge could be stored as lift over a masked-slot prior, so that common words do not look specially related to everything.
  • I2 (learn): Evaluation probes could be run over sets of paraphrased templates, with the spread across variants reported as a robustness signal next to the mean.
  • I3 (learn): Whenever several confidence or agreement scores compete for silver-data selection, a small human-checked set can decide which score to trust.
  • I4 (learn): When witness parsers or LLM teachers disagree, the full vote distribution should be kept rather than one averaged label, so contested cases stay visible.
  • I5 (world): The text world could record such default assumptions as explicit, retractable defaults, so later answers can explain when they rest on an unstated prototype.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States325 works · 142,931 citations🎓University of Maryland, College ParkCollege Park · United States132 works · 25,380 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSafety Research · 543 worksSociology and Political Science · 577 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEnglish · 2,263 worksBias, fairness & safety · 517 works

Cite

Cite as Yang Trista Cao, Anna Sotnikova, Hal Daumé III, Rachel Rudinger and Linda Zou. 2022. Theory-Grounded Measurement of U.S. Social Stereotypes in English Language Models. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.naacl-main.92

@inproceedings{cao2022theory,
  title = {{Theory-Grounded Measurement of U.S. Social Stereotypes in English Language Models}},
  author = {Yang Trista Cao, Anna Sotnikova, Hal Daumé III, Rachel Rudinger and Linda Zou},
  booktitle = {Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-main.92},
  url = {https://doi.org/10.18653/v1/2022.naacl-main.92}
}

References

48 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Andrea Abele, Naomi Ellemers, Susan Fiske, Alex Koch, and Vincent Yzerbyt. 2020. Navigating the social world: Toward an integrated framework for evaluating self, individuals, and groups. Psychological review, 128. ↗
  2. Andrea Abele and Bogdan Wojciszke. 2014. Communal and agentic content a dual perspective model. Adv. Exp. Soc. Psychol., 50:198–255. find
  3. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big?. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623. ↗
  4. Alina Beygelzimer, Sanjoy Dasgupta, and John Langford. 2008. Importance weighted active learning. CoRR, abs/0812.4952. ↗
  5. Victor Bittorf, Benjamin Recht, Christopher Ré, and Joel Tropp. 2012. Factoring nonnegative matrices with linear programs. Advances in Neural Information Processing Systems, 2. find
  6. Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021. Stereotyping Norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1004–1015, Online. Association for Computational Linguistics. ↗
  7. Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. NeurIPS, pages 4349–4357. ↗
  8. Eduardo Bonilla-Silva. 1997. Rethinking racism: Toward a structural interpretation. American sociological review, pages 465–480. ↗
  9. Robert E. Botsch. 2011. Significance and Measures of Association. find
  10. Irene Browne and Joya Misra. 2003. The intersection of gender and race in the labor market. Annual review of sociology, 29(1):487–513. ↗
  11. J.S. Bruner, Brunswik E, L. Festinger, F. Heider, K.F. Muenzinger, C.E. Osgood, and D. Rapaport. 1957. Going beyond the information given. Contemporary approaches to cognition, pages 41–67. find
  12. Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. ↗
  13. Yang Trista Cao, Yada Pruksachatkun, Kai-Wei Chang, Rahul Gupta, Varun Kumar, Jwala Dhamala, and Aram Galstyan. 2022. On the intrinsic and extrinsic fairness evaluation metrics for contextualized language representations. ↗
  14. Patricia Hill Collins. 2002. Black feminist thought: Knowledge, consciousness, and the politics of empowerment. routledge. find
  15. Combahee River Collective. 1977. A Black Feminist Statement. na. find
  16. Combahee River Collective. 1983. The combahee river collective statement. Home girls: A Black feminist anthology, 1:264–274. find
  17. Kimberlé Crenshaw. 1989. Demarginalizing the intersection of race and sex: A black feminist critique of antidiscrimination doctrine, feminist theory and antiracist politics. u. Chi. Legal f., page 139. find
  18. Hal Daumé, III and Abhishek Kumar. 2017. Column squishing for multiclass updates (blog post). find
  19. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (1).
  20. Naomi Ellemers. 2017. Morality and the Regulation of Social Behavior: Groups as Moral Anchors. find
  21. Susan T. Fiske, Amy J. C. Cuddy, Peter Glick, and Jun Xu. 2002. A model of (often mixed) stereotype content: competence and warmth respectively follow from perceived status and competition. Journal of personality and social psychology, 82 6:878–902. ↗
  22. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé, and Kate Crawford. 2018. Datasheets for datasets. find
  23. Negin Ghavami and Letitia Anne Peplau. 2013. An intersectional analysis of gender and ethnic stereotypes: Testing three hypotheses. Psychology of Women Quarterly, 37(1):113–127. ↗
  24. Wei Guo and Aylin Caliskan. 2021. Detecting emergent intersectional biases: Contextualized word embeddings contain a distribution of human-like biases. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’21, page 122–133, New York, NY, USA. Association for Computing Machinery. find
  25. Timothy J. Hazen, Alexandra Olteanu, Gabriella Kazai, Fernando Diaz, and Michael Golebiewski. 2020. On the social and technical challenges of web search autosuggestion moderation. arXiv:2007.05039 [cs]. ArXiv: 2007.05039. bell hooks. 1992. Yearning: Race, gender, and cultural politics. Hypatia, 7(2). ↗
  26. Dirk Hovy and Diyi Yang. 2021. The importance of modeling social factors of language: Theory and practice. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 588–602, Online. Association for Computational Linguistics. ↗
  27. Lynne M. Jackson. 2011. The psychology of prejudice: From attitudes to social action. American Psychological Association. ↗
  28. Deborah K King. 1988. Multiple jeopardy, multiple consciousness: The context of a black feminist ideology. Signs: Journal of women in culture and society, 14(1):42–72. ↗
  29. Alex Koch, Roland Imhoff, Ron Dotsch, Christian Unkelbach, and Hans Alves. 2016. The abc of stereotypes about groups: Agency/socioeconomic success, conservative-progressive beliefs, and communion. Journal of personality and social psychology, 110:675–709. ↗
  30. Alex Koch, Vincent Yzerbyt, Andrea Abele, Naomi Ellemers, and Susan T. Fiske. 2021. Social evaluation: Comparing models across interpersonal, intra-group, intergroup, several-group, and many-group contexts, volume 63, page 1–68. Elsevier. find
  31. Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W. Black, and Yulia Tsvetkov. 2019. Measuring bias in contextualized word representations. CoRR, abs/1906.07337. ↗
  32. Carl A. Latkin, Catie Edwards, Melissa A. Davey-Rothwell, and Karin E. Tobin. 2017. The relationship between social desirability bias and self-reports of health, substance use, and social network factors among urban substance users in baltimore, maryland. Addictive Behaviors, 73:133–136. ↗
  33. Walter Lippmann. 1965. Public Opinion. New York :Free Press. find
  34. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  35. Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 622–628, Minneapolis, Minnesota. Association for Computational Linguistics.
  36. Moin Nadeem, Anna Bethke, and Siva Reddy. 2020. Stereoset: Measuring stereotypical bias in pretrained language models. CoRR, abs/2004.09456. ↗
  37. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953–1967, Online. Association for Computational Linguistics. ↗
  38. Debora Nozza, Federico Bianchi, and Dirk Hovy. 2021. HONEST: Measuring hurtful sentence completion in language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2398–2406, Online. Association for Computational Linguistics.
  39. Silviu Paun, Bob Carpenter, Jon Chamberlain, Dirk Hovy, Udo Kruschwitz, and Massimo Poesio. 2018. Comparing Bayesian Models of Annotation. Transactions of the Association for Computational Linguistics, 6:571–585. ↗
  40. Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2019. Social bias frames: Reasoning about social and power implications of language. CoRR, abs/1911.03891. ↗
  41. Anna Sotnikova, Yang Trista Cao, Hal Daumé III, and Rachel Rudinger. 2021. Analyzing stereotypes in generative text inference tasks. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4052–4065, Online. Association for Computational Linguistics. Dr. find
  42. Charles Stangor. 2014. Principles of social psychology – 1st international edition. BCcampus. find
  43. Amy C Steinbugler, Julie E Press, and Janice Johnson Dias. 2006. Gender, race, and affirmative action: Operationalizing intersectionality in survey research. Gender & Society, 20(6):805–825. ↗
  44. Sheldon Stryker. 1980. Symbolic interactionism: a social structural version. Benjamin/Cummings Pub. Co. find
  45. S. Wheeler and Richard Petty. 2001. The effects of stereotype activation on behavior: A review of possible mechanisms. Psychological bulletin, 127:797– 826. ↗
  46. I.J. Wod. 1985. Weight of evidence: A brief survey. Bayesian statistics, 2:249–270. find
  47. Vincent Y. Yzerbyt. 2018. The dimensional compensation model. Agency and Communion in Social Psychology. ↗
  48. Maxine Baca Zinn and Bonnie Thornton Dill. 1996. Theorizing difference from multiracial feminism. Feminist studies, 22(2):321–331. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cao-2022-theory-grounded-measurement-u-s/