Elamur Research
☰
home / papers / 2021 / StereoSet: Measuring stereotypical bias in pretrained language models

StereoSet: Measuring stereotypical bias in pretrained language models

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2021.acl-long.416.pdf

Summary

Pretrained language models learn from huge amounts of real-world text, so they can absorb harmful stereotypes about social groups. Earlier bias probes had two weaknesses. They used short, artificial template sentences, and they ignored whether a model could actually model language, so a weak model could look deceptively fair. The authors respond with the Context Association Test (CAT). In this test, a context naming a target group is paired with three continuations: stereotypical, anti-stereotypical and nonsensical. The test has two forms. The intrasentence version fills a blank inside a sentence, and the intersentence version picks the most likely follow-up sentence. Using crowdsourcing, they built StereoSet, a natural English benchmark covering gender, profession, race and religion. They scored BERT, GPT2, RoBERTa and XLNet on language-modeling quality and on their preference for stereotypes. All of these models showed clear stereotypical bias, and stronger language modeling generally came with more bias. The resource lets bias and modeling competence be judged together, and it works for both masked and autoregressive models.

Key points

  • StereoSet has 16,995 test triplets built around 321 target terms in four domains (gender, profession, race, religion). Data and code are released at stereoset.mit.edu.
  • Three metrics are reported:
    • lms: the language model score, i.e. how often meaningful options beat meaningless ones.
    • ss: the stereotype score, where 50 is ideal.
    • icat: the idealized CAT score, which combines lms and ss.
  • Larger versions of a model (e.g. BERT-base at 110M parameters vs. BERT-large at 340M) score higher on lms but also higher on ss. Pretraining corpus size showed no clear link to either metric.
  • Pseudo-likelihood scoring beats likelihood scoring on intrasentence CATs (avg lms 79.4 vs 75.7) but falls behind on intersentence CATs (75.98 vs 78.82). This points to trouble with scoring longer sequences.
  • For the ENSEMBLE model, race was the least biased domain (ss 61.8). The most stereotyped terms included mother (ss 77.8), software developer (75.9) and African (74.5). Terms like producer and Crimean showed little bias, and Muslim was a surprising low-bias outlier.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

526 papers of our library cite it (of the 140673 whose bibliographies we read); 25 of them have a page here, 501 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.acl-long.416 · arXiv 2004.09456

@inproceedings{nadeem2021stereoset,
  title = {{StereoSet: Measuring stereotypical bias in pretrained language models}},
  author = {Nadeem, Moin and Bethke, Anna and Reddy, Siva},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.416},
  eprint = {2004.09456},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.acl-long.416}
}

References

35 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Vamsi Aribandi, Yi Tay, and Donald Metzler. 2021. How reliable are model diagnostics? In Proceedings of ACL Findings. find
  2. Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604. find
  3. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of FAccT. find
  4. Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam T. Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Proceedings of Neural Information Processing Systems (NeurIPS), pages 4349–4357. find our card
  5. Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. find
  6. Patricia Hill Collins. 2004. Black sexual politics: African Americans, gender, and the new racism. Routledge. find
  7. Alexander M Czopp, Aaron C Kay, and Sapna Cheryan. 2015. Positive stereotypes are pervasive and powerful. Perspectives on Psychological Science, 10(4):451–463. find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of North American Chapter of the Association for Computational Linguistics, pages 4171–4186, Minneapolis, Minnesota. Association fo… find
  9. Djellel Difallah, Elena Filatova, and Panos Ipeirotis. 2018. Demographics and dynamics of mechanical turk workers. In Proceedings of the ACM International Conference on Web Search and Data Mining, WSDM ’18, pages 135 – 143, New York, NY, USA. Association for Computing Machinery. find
  10. Hila Gonen and Yoav Goldberg. 2019. Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them. find our card
  11. Anthony G. Greenwald and Mahzarin R. Banaji. 1995. Implicit social cognition: attitudes, self-esteem, and stereotypes. Psychological review, 102(1):4. find
  12. Jeremy Howard and Sebastian Ruder. 2018. Universal Language Model Fine-tuning for Text Classification. In Proceedings of the Association for Computational Linguistics, pages 328–339, Melbourne, Australia. Association for Computational Linguistics. find
  13. Milos Jakubicek, Adam Kilgarriff, Vojtech Kovar, Pavel Rychly, and Vit Suchomel. 2013. The tenten corpus family. In Proceedings of the International Corpus Linguistics Conference CL. find
  14. Adam Kilgarriff. 2009. Simple maths for keywords. In Proceedings of the Corpus Linguistics Conference 2009 (CL2009), page 171. find
  15. Svetlana Kiritchenko and Saif Mohammad. 2018. Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems. In Proceedings of Joint Conference on Lexical and Computational Semantics, pages 43–53. find our card
  16. Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. Measuring bias in contextualized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 166–172, Florence, Italy. Association for Computational Linguistics. find our card
  17. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  18. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the Association for Computational Linguistics, pages 142–150, Portland, Oregon, USA. Association for Computational Linguistics. find our card
  19. Thomas Manzini, Lim Yao Chong, Alan W Black, and Yulia Tsvetkov. 2019. Black is to criminal as caucasian is to police: Detecting and removing multiclass bias in word embeddings. In Proceedings of the North American Chapter of the Association for Computational Linguistics, pages 615–621, Minneapolis,… find
  20. Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. In Proceedings of the North American Chapter of the Association for Computational Linguistics, pages 622–628, Minneapolis, Minnesota. Association for Computational Li… find our card
  21. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of Neural Information Processing Systems (NeurIPS), NIPS 13, pages 3111 – 3119, Red Hook, NY, USA. Curran Associates Inc. find
  22. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP), pages 1953–1967, Online. Association for Computational Linguist… find our card
  23. Methods. In Proceedings of North American Chapter of the Association for Computational Linguistics, pages 15–20. find
  24. Brian Nosek, Mahzarin Banaji, and Anthony Greenwald. 2002. Math = male, me = female, therefore math != me. Journal of personality and social psychology, 83:44–59. find
  25. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. find
  26. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations. In Proceedings of the North American Chapter of the Association for Computational Linguistics), pages 2227–2237. Association for Computational… find
  27. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), IC… find our card
  28. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8). find
  29. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of North American Chapter of the Association for Computational Linguistics (NAACL), pages 8–14. find our card
  30. Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020. Masked language model scoring. In Proceedings of the of the Association for Computational Linguistics, Online. Association for Computational Linguistics. find our card
  31. Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. The woman worked as a babysitter: On biases in language generation. In Proceedings of the Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pag… find our card
  32. Amos Tversky and Daniel Kahneman. 1974. Judgment under uncertainty: Heuristics and biases. science, 185(4157):1124–1131. find
  33. Denny Vrandečić and Markus Krötzsch. 2014. Wikidata: A free collaborative knowledgebase. Commun. ACM, 57(10):78–85. find
  34. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d’e Buc, E. Fox, and R. Garnett, editors, Proceedings of Neural Information Pro… find our card
  35. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender Bias in Coreference Resolution: Evaluation and Debiasing Ethics and Data Statement Following Bender and Friedman (2018), we provide the following ethics and data statement. speaker to stereotype and anti-stereoty… find