Elamur Research
☰
home / papers / 2023 / Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models

Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2023.acl-long.84.pdf

Summary

The paper presents Marked Personas, a way to measure stereotypes in large language models without building a lexicon or labeling data. Existing bias benchmarks use hand-made templates or curated sentences. They either cover a narrow, fixed set of stereotypes or miss subtle harms, such as othering through positive-sounding words. The method has two steps. First, the model is asked in a zero-shot setting to describe an imagined person from a given demographic group, including intersectional groups and unmarked defaults such as white or man. Second, it finds the words that statistically separate personas of a marked group from the matching unmarked ones. The approach draws on the sociolinguistic idea of markedness. Applied to GPT-3.5 and GPT-4, the method shows that generated personas contain more racial stereotypes than human-written answers to identical prompts. Marked groups tend to be exoticized, and minoritized women are shown through tropes like tropicalism and hypersexualization. The same patterns appear in generated stories. This matters because bias that sounds positive slips past current measures and can still spread into downstream uses.

Key points

  • The Marked Words step uses the Fightin' Words method (Monroe et al., 2008) with an informative Dirichlet prior. It compares weighted log-odds of word use in a marked group's personas against those of each relevant unmarked group.
  • Human baselines come from Kambhatla et al. (2022). That study collected self-descriptions from Prolific crowd-workers, both as their own race and as an imagined one, using the same prompts.
  • The study covers five race/ethnicity groups and three gender groups, giving 15 combinations. For the story-generation test, 30 stories were produced per combination.
  • Words that seem positive but still essentialize (e.g., resilience narratives, independent for women, martial arts in stories about Asians) are flagged as harmful. The authors argue that swapping negative language for positive language does not remove the harm.
  • Measured against the five desiderata from Cao et al. (2022), the framework meets four; it does not meet exhaustiveness. The generated personas and code are released at github.com/myracheng/markedpersonas. Limitations: only OpenAI API models were tested, only in English, and only against American stereotypes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

120 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 117 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.acl-long.84

@inproceedings{cheng2023marked,
  title = {{Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models}},
  author = {Cheng, Myra and Durmus, Esin and Jurafsky, Dan},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.84},
  url = {https://doi.org/10.18653/v1/2023.acl-long.84}
}

References

110 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kim Allen. 2022. Re-claiming resilience and reimagining welfare: A response to angela mcrobbie. European Journal of Cultural Studies, 25(1):310– 315. find
  2. Evelyn Alsultany. 2012. Arabs and muslims in the media. In Arabs and Muslims in the Media. New York University Press. find
  3. Haozhe An, Zongxia Li, Jieyu Zhao, and Rachel Rudinger. 2023. SODAPOP: Open-ended discovery of social biases in social commonsense reasoning models. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1573–1596, Dubrovnik, Croatia. As… find
  4. Leah Iman Aniefuna, M Amari Aniefuna, and Jason M Williams. 2020. Creating and undoing legacies of resilience: Black women as martyrs in the black community under oppressive social control. Women & Criminal Justice, 30(5):356–373. find
  5. Sameena Azhar, Antonia RG Alvarez, Anne SJ Farina, and Susan Klumpner. 2021. “You’re so exotic looking”: An intersectional analysis of asian american and pacific islander stereotypes. Affilia, 36(3):282– 301. find
  6. April H Bailey, Adina Williams, and Andrei Cimpian. 2022. Based on billions of words on the internet, people= men. Science Advances, 8(13):eabm2463. find
  7. David Bamman, Brendan O’Connor, and Noah A Smith. 2013. Learning latent personas of film characters. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 352–361. find
  8. Soumya Barikeri, Anne Lauscher, Ivan Vulić, and Goran Glavaš. 2021. RedditBias: A real-world resource for bias evaluation and debiasing of conversational language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conf… find
  9. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623. find
  10. Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021. Stereotyping norwegian salmon: an inventory of pitfalls in fairness benchmark datasets. In Proc. 59th Annual Meeting of the Association for Computational Linguistics. find
  11. Stefan Blomkvist. 2002. The user as a personality-using personas as a tool for design. KTH-Royal Institute of Technology, Stockholm Www. Nada. Kth. Se/tessy/Blomkvist. Pdf, 980. find
  12. Markie LC Blumer, Y Gavriel Ansara, and Courtney M Watson. 2013. Cisgenderism in family therapy: How everyday clinical practices can delegitimize people’s gender self-designations. Journal of Family Psychotherapy, 24(4):267–285. find
  13. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems, 29. find our card
  14. Eduardo Bonilla-Silva. 2006. Racism without racists: Color-blind racism and the persistence of racial inequality in the United States. Rowman & Littlefield Publishers. find
  15. Wayne Brekhus. 1998. A sociology of the unmarked: Redirecting our focus. Sociological Theory, 16(1):34–51. find
  16. Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. find
  17. Yang Cao, Anna Sotnikova, Hal Daumé III, Rachel Rudinger, and Linda Zou. 2022. Theory-grounded measurement of us social stereotypes in english language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technol… find
  18. Stephanie Castelin and Grace White. 2022. “I’m a strong independent black woman”: The strong black woman schema and mental health in college-aged black women. Psychology of Women Quarterly, 46(2):196–208. find
  19. Connie S Chan. 1988. Asian-american women: Psychological responses to sexual exploitation and cultural stereotypes. Women & Therapy, 6(4):33–38. find
  20. Szu-Hsien Chang and Brian H Kleiner. 2003. Common racial stereotypes. Equal Opportunities International. find
  21. Sapna Cheryan and Hazel Rose Markus. 2020. Masculine defaults: Identifying and mitigating hidden cultural biases. Psychological Review, 127(6):1022. find
  22. Combahee River Collective. 1983. The combahee river collective statement. Home girls: A Black feminist anthology, 1:264–274. find
  23. Patricia Hill Collins. 1990. Black feminist thought in the matrix of domination. Black feminist thought: Knowledge, consciousness, and the politics of empowerment, 138(1990):221–238. find
  24. Alan Cooper. 1999. The inmates are running the asylum. Springer. find
  25. Simone De Beauvoir. 1952. The second sex, trans. find
  26. HM Parshley (New York: Vintage, 1974), 38. find
  27. Kay Deaux and Mary Kite. 1993. Gender stereotypes. find
  28. Fred Dervin. 2012. Cultural identity, representation and othering. In The Routledge handbook of language and intercultural communication, pages 195– 208. Routledge. find
  29. Emily Dinan, Angela Fan, Ledell Wu, Jason Weston, Douwe Kiela, and Adina Williams. 2020. Multidimensional gender bias classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 314–331. find
  30. Alice H Eagly, Christa Nater, David I Miller, Michèle Kaufmann, and Sabine Sczesny. 2020. Gender stereotypes have changed: A cross-temporal meta-analysis of us public opinion polls from 1946 to 2018. American psychologist, 75(3):301. find
  31. Ruth Frankenburg. 1993. White women, race matters: The social construction of whiteness. Routledge. find
  32. Ryan J Gallagher, Morgan R Frank, Lewis Mitchell, Aaron J Schwartz, Andrew J Reagan, Christopher M Danforth, and Peter Sheridan Dodds. 2021. Generalized word shift graphs: a method for visualizing and explaining pairwise comparisons between texts. EPJ Data Science, 10(1):4. find
  33. Patricia Garcia, Tonia Sutherland, Marika Cifor, Anita Say Chan, Lauren Klein, Catherine D’Ignazio, and Niloufar Salehi. 2020. No: Critical refusal as feminist data practice. In conference companion publication of the 2020 on computer supported cooperative work and social computing, pages 199–202. find
  34. Caron E Gentry. 2022. Misogynistic terrorism: it has always been here. Critical Studies on Terrorism, 15(1):209–224. find
  35. Negin Ghavami and Letitia Anne Peplau. 2013. An intersectional analysis of gender and ethnic stereotypes: Testing three hypotheses. Psychology of Women Quarterly, 37(1):113–127. find
  36. Herbert P Grice. 1975. Logic and conversation. In Speech acts, pages 41–58. Brill. find
  37. Wei Guo and Aylin Caliskan. 2021. Detecting emergent intersectional biases: Contextualized word embeddings contain a distribution of human-like biases. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 122–133. find
  38. Kimberlé W Crenshaw. 2017. On intersectionality: Essential writings. The New Press. find
  39. Peter Hegarty, Y Gavriel Ansara, and Meg-John Barker. 2018. Nonbinary gender identities. Gender, sex, and sexualities: Psychological perspectives, pages 53–76. find
  40. Alexander M Czopp, Aaron C Kay, and Sapna Cheryan. 2015. Positive stereotypes are pervasive and powerful. Perspectives on Psychological Science, 10(4):451–463. find
  41. Madeline E Heilman. 2001. Description and prescription: How gender stereotypes prevent women’s ascent up the organizational ladder. Journal of social issues, 57(4):657–674. find
  42. Mar Hicks. 2017. Programmed inequality: How Britain discarded women technologists and lost its edge in computing. MIT Press. find
  43. Bruce Hoffman. 1995. “Holy terror”: The implications of terrorism motivated by a religious imperative. Studies in Conflict & Terrorism, 18(4):271–284. find
  44. Bell Hooks. 2000. Feminist theory: From margin to center. Pluto Press. find
  45. Minlie Huang, Xiaoyan Zhu, and Jianfeng Gao. 2020. Challenges in building intelligent open-domain dialog systems. ACM Transactions on Information Systems (TOIS), 38(3):1–32. find
  46. Clayton Hutto and Eric Gilbert. 2014. Vader: A parsimonious rule-based model for sentiment analysis of social media text. In Proceedings of the international AAAI conference on web and social media, volume 8, pages 216–225. find
  47. Sune Qvotrup Jensen. 2011. Othering, identity formation and agency. Qualitative studies, 2(2):63–78. find
  48. Eva Jettmar and Clifford Nass. 2002. Adaptive testing: effects on user performance. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 129–134. find
  49. Gauri Kambhatla, Ian Stewart, and Rada Mihalcea. 2022. Surfacing racial stereotypes through identity portrayal. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1604–1615. find
  50. Abram Kardiner and Lionel Ovesey. 1951. The mark of oppression; a psychosocial study of the american negro. find
  51. Hannah Rose Kirk, Yennie Jun, Filippo Volpin, Haider Iqbal, Elias Benussi, Frederic Dreyer, Aleksandar Shtedritski, and Yuki Asano. 2021. Bias out-of-the-box: An empirical analysis of intersectional occupational biases in popular generative language models. Advances in neural information processing… find
  52. Mina Lee, Percy Liang, and Qian Yang. 2022. Coauthor: Designing a human-ai collaborative writing dataset for exploring language model capabilities. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI ’22, New York, NY, USA. Association for Computing Machinery. find
  53. Michael Lepori. 2020. Unequal representations: Analyzing intersectional biases in word embeddings using representational similarity analysis. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1720–1728. find
  54. Claude Lévi-Strauss. 1963. Structural anthropology. Basic books. find
  55. Amanda E Lewis. 2004. What group?” studying whites and whiteness in the era of “color-blindness. Sociological theory, 22(4):623–646. find
  56. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110. find
  57. Kelly Yu-Hsin Liao, Meifen Wei, and Mengxi Yin. 2020. The misunderstood schema of the strong black woman: Exploring its mental health consequences and coping responses among african american women. Psychology of Women Quarterly, 44(1):84–104. find
  58. Max Liboiron. 2021. Pollution is colonialism. In Pollution Is Colonialism. Duke University Press. find
  59. Li Lucy and David Bamman. 2021. Gender and representation bias in gpt-3 generated stories. In Proceedings of the Third Workshop on Narrative Understanding, pages 48–55. find
  60. Sheng-mei Ma. 2000. The deathly embrace: Orientalism and Asian American identity. U of Minnesota Press. find
  61. Stephanie Madon, Max Guyll, Kathy Aboufadel, Eu-lices Montiel, Alison Smith, Polly Palumbo, and Lee Jussim. 2001. Ethnic and national stereotypes: The princeton trilogy revisited and revised. Personality and social psychology bulletin, 27(8):996–1010. find
  62. Małgorzata Martynuska. 2016. The exotic other: representations of latina tropicalism in us popular culture. Journal of Language and Cultural Education, 4(2):73–81. find
  63. Chandler May, Alex Wang, Shikha Bordia, Samuel R Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. In Proceedings of NAACL-HLT, pages 622–628. find our card
  64. Angela McRobbie. 2020. Feminism and the politics of resilience: Essays on gender, media and the end of welfare. John Wiley & Sons. find
  65. Jodi Melamed. 2006. The spirit of neoliberalism: From racial liberalism to neoliberal multiculturalism. Social text, 24(4):1–24. find
  66. Arnold Mindell. 2006. Leader as Martial Artist: Techniques and Strategies for Resolving Conflict and Creating Community. Lao Tse Press, Limited. find
  67. Isabel Molina-Guzmán. 2010. Dangerous curves: Latina bodies in the media, volume 5. NYU Press. find
  68. Burt L Monroe, Michael P Colaresi, and Kevin M Quinn. 2008. Fightin’words: Lexical feature selection and evaluation for identifying the content of political conflict. Political Analysis, 16(4):372–403. find
  69. Michael J Muller and Kenneth Carey. 2002. Design as a minority discipline in a software company: toward requirements for a community of practice. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 383–390. find
  70. Sina Ali Muscati. 2002. Arab/muslim’otherness’: The role of racial constructions in the gulf war and the continuing crisis with iraq. Journal of Muslim Minority Affairs, 22(1):131–148. find
  71. Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. Stereoset: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long… find our card
  72. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel Bowman. 2020. Crows-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953–1967. find our card
  73. Tamara Nelson, Esteban V Cardemil, and Camille T Adeoye. 2016. Rethinking strength: Black women’s perceptions of the “strong black woman” role. Psychology of women quarterly, 40(4):551–563. find
  74. OpenAI. 2022. Openai: Introducing chatgpt. https: //openai.com/blog/chatgpt. [Online; accessed 9-May-2023]. find
  75. OpenAI. 2023. Gpt-4 technical report. arXiv. find our card
  76. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744. find
  77. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022. Bbq: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2086–2105. find our card
  78. Jemima Pierre. 2004. Black immigrants in the united states and the" cultural narratives" of ethnicity. Identities: Global studies in culture and power, 11(2):141– 170. find
  79. Robert J Podesva, Jermay Reynolds, Patrick Callier, and Jessica Baptiste. 2015. Constraints on the social meaning of released/t: A production and perception study of us politicians. Language Variation and Change, 27(1):59–87. find
  80. Tyler Reny and Sylvia Manzano. 2016. The negative effects of mass media stereotypes of latinos and immigrants. Media and minorities, 4:195–212. find
  81. Karen E Rosenblum and Toni-Michelle C Travis. 1996. The Meaning of Difference: American Constructions of Race, Sex, volume 52. McGraw-Hill. find
  82. Catherine Rottenberg. 2014. The rise of neoliberal feminism. Cultural studies, 28(3):418–437. find
  83. Edward Said. 1978. Orientalism: Western concepts of the orient. New York: Pantheon. find
  84. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? arXiv preprint arXiv:2303.17548. find our card
  85. Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power implications of language. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5477–5490. find our card
  86. Orna Sasson-Levy. 2013. A different kind of whiteness: Marking and unmarking of social boundaries in the construction of hegemonic ethnicity. In Sociological Forum, volume 28, pages 27–50. Wiley Online Library. find
  87. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. find
  88. Joseph A Schafer, Christopher W Mullins, and Stephanie Box. 2014. Awakenings: The emergence of white supremacist ideologies. Deviant Behavior, 35(3):173–196. find
  89. Timo Schick, Sahana Udupa, and Hinrich Schütze. 2021. Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in nlp. Transactions of the Association for Computational Linguistics, 9:1408– 1424. find our card
  90. Jack G Shaheen. 2003. Reel bad arabs: How hollywood vilifies a people. The ANNALS of the American Academy of Political and Social science, 588(1):171– 193. find
  91. Eric Michael Smith, Melissa Hall, Melanie Kambadur, Eleonora Presani, and Adina Williams. 2022. “I’m sorry to hear that”: Finding new biases in language models with a holistic descriptor dataset. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9180–92… find
  92. Richard Jean So and Edwin Roland. 2020. Race and distant reading. PMLA, 135(1):59–73. find
  93. Ann Laura Stoler et al. 1995. Race and the education of desire: Foucault’s history of sexuality and the colonial order of things. Duke University Press. find
  94. Yi Chern Tan and L Elisa Celis. 2019. Assessing social and intersectional biases in contextualized word representations. Advances in Neural Information Processing Systems, 32. find
  95. Milo Trujillo, Sam Rosenblatt, Guillermo De Anda-Jáuregui, Emily Moog, Briane Paul V Samson, Laurent Hébert-Dufresne, and Allison M Roth. 2021. find
  96. Debiasing CrowS-Pairs Stereoset S. Bias Frames CEAT ABC Marked Personas Aki Uchida. 1998. The orientalization of asian women in america. In Women’s Studies International Forum, volume 21, pages 161–174. Elsevier. find
  97. Natalie N Watson and Carla D Hunter. 2016. “I had to be strong” tensions in the strong black woman schema. Journal of Black Psychology, 42(5):424– 452. find
  98. Linda R Waugh. 1982. Marked and unmarked: A choice between unequals in semiotic structure. find
  99. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359. find our card
  100. Robert Wolfe, Mahzarin R. Banaji, and Aylin Caliskan. 2022. Evidence for hypodescent in visual semantic ai. 2022 ACM Conference on Fairness, Accountability, and Transparency. find
  101. Robert Wolfe and Aylin Caliskan. 2022a. American== white in multimodal language-and-image ai. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pages 800–812. find
  102. Robert Wolfe and Aylin Caliskan. 2022b. Markedness in visual semantic ai. 2022 ACM Conference on Fairness, Accountability, and Transparency. find
  103. Cheryl L Woods-Giscombé. 2010. Superwoman schema: African american women’s views on stress, strength, and health. Qualitative health research, 20(5):668–683. find
  104. Kathryn Woodward. 1997. Identity and difference, volume 3. Sage. find
  105. Xinchao Xu, Zhibin Gou, Wenquan Wu, Zheng-Yu Niu, Hua Wu, Haifeng Wang, and Shihang Wang. 2022. Long time no see! open-domain conversation with long-term persona memory. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2639–2650. find
  106. Mari Yoshihara. 2002. Embracing the East: White women and American orientalism. Oxford University Press. find
  107. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068. find
  108. Robin Zheng. 2016. Why yellow fever isn’t flattering: A case against racial fetishes. Journal of the American Philosophical Association, 2(3):400–419. ! ! ! ! ! ! ! ! ! ! ! ! ! ! find
  109. Specificity Natural Exhaustive Grounded Generalizes When the echo chamber shatters: Examining the use of community-specific language post-subreddit ban. In Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021), pages 164–178. ! ! ! ! ! ! ! find
  110. Table A1: Marked Personas uniquely satisfies 4 of the criteria introduced by Cao et al. (2022). find