Elamur Research⌕
☰
home / papers / 2023 / “Fifty Shades of Bias”: Normative Ratings of Gender Bias in GPT Generated English Text

“Fifty Shades of Bias”: Normative Ratings of Gender Bias in GPT Generated English Text

Paper · 2023 · Rishav Hada, Agrima Seth, Harshita Diddee et al.

Summary

Most gender-bias resources label a sentence as simply biased or not. They are usually built from rigid templates or from lexicon-driven web mining, and both approaches tend to miss subtle, implicit bias. This paper treats bias as a matter of degree instead.

The authors prompted GPT-3.5-Turbo with curated, gender-neutral seeds and hand-written in-context examples. The goal was statements ranging from explicit to implicit bias to neutral. Human annotators then rated them with Best–Worst Scaling (BWS), a comparative scheme in which people pick the most and least biased item in a small set. The result is a dataset of 1000 English statements with real-valued scores between 0 and 1, and repeat annotations show the scores are highly reliable.

The analyses cover:

  • how seed types and prompting strategies affect the output,
  • which themes people judge as more severe,
  • how existing models for related concepts perform on the new data.

Identity attack aligns most closely with gender bias. GPT-4 produces ratings close to human ones, but its stated reasons are often wrong. Graded severity matters because it allows proportionate mitigation rather than one blunt filter.

Key points

  • The dataset holds 1000 GPT-3.5-Turbo generated English statements, each assigned a score on a scale where 0 indicates the weakest negative bias and 1 the strongest. It is available via aka.ms/FiftyShadesofBias.
  • BWS uses 4-tuples: one best/worst judgment yields 5 of the 6 pairwise orderings, which avoids annotating on the order of N² pairs. Repeat annotations correlate at r ≈ 0.86.
  • Early generation rounds had several problems:
    • About 40% of outputs injected bias at the cost of logical sense.
    • Gendered seeds pushed the model toward explicit bias.
    • Outputs copied the syntax of the in-context examples.
    • The authors responded by switching to implicit, gender-neutral seeds.
  • Statements aimed at communities were judged more biased than those aimed at individuals. Among existing classifiers' concepts, identity attack correlates most with gender bias.
  • Annotators came from a single institute in India. Each annotator was capped at 8% of the task to limit exposure to harmful text.

Interesting ideas

  • I1. Best–Worst Scaling over 4-tuples turns a fuzzy, subjective judgment into stable real-valued scores cheaply, because each choice encodes most pairwise comparisons in the tuple.
  • I2. The authors caught generations that bolted on a forced causal clause (packaging explained by someone's "delicate hands") and treated them as a separate failure class.
  • I3. Generated samples closely mimicked the syntax of the in-context examples, so the authors deliberately diversified their prompts and seeds.
  • I4. GPT-4 gave human-like scores but often with flawed reasoning, so the score alone did not reveal whether the model understood the statement.
  • I5. Bias is modelled as a continuous severity, which lets downstream systems respond in proportion rather than with one binary filter.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Annotators or LLM teachers could pick the best and worst of four candidate parses or frame analyses, giving graded silver-quality scores instead of brittle accept/reject labels.
  • I2 (world): The world builder could flag causal links between events that no compiled knowledge supports, marking them as suspicious instead of storing them as facts.
  • I3 (tree): Any LLM-generated sentences used as extra parser training data should be checked for construction diversity, for example the spread of dependency relations and clause types, before going into a treebank.
  • I4 (answer): Evaluation of readers and students should check the explanation chain separately from the final answer, so that right answers reached for wrong reasons are counted as failures.
  • I5 (frames): Implicature, irony or face-threatening speech acts in prag could carry a graded strength score, letting later stages react to how strong the effect is, not just whether it is present.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations🎓University of MichiganAnn Arbor · United States70 works · 13,693 citations🏢Microsoft ResearchBengaluru · India13 works · 3,602 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 worksGender Studies · 52 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEnglish · 2,263 worksBias, fairness & safety · 517 works

Cite

Cite as Rishav Hada, Agrima Seth, Harshita Diddee and Kalika Bali. 2023. “Fifty Shades of Bias”: Normative Ratings of Gender Bias in GPT Generated English Text. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.emnlp-main.115

@inproceedings{hada2023fifty,
  title = {{“Fifty Shades of Bias”: Normative Ratings of Gender Bias in GPT Generated English Text}},
  author = {Rishav Hada, Agrima Seth, Harshita Diddee and Kalika Bali},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.115},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.115}
}

References

48 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shima Asaadi, Saif Mohammad, and Svetlana Kiritchenko. 2019. Big BiRD: A large, fine-grained, bigram relatedness dataset for examining semantic composition. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 505–516, Minneapolis, Minnesota. Association for Computational Linguistics. find
  2. Fatemeh Torabi Asr, Mohammad Mazraeh, Alexandre Lopes, Vasundhara Gautam, Junette Gonzales, Prashanth Rao, and Maite Taboada. 2021. The gender gap tracker: Using natural language processing to measure gender bias in media. PloS one, 16(1):e0245533. ↗
  3. Hritik Bansal, John Dang, and Aditya Grover. 2023. Peering through preferences: Unraveling feedback acquisition for aligning large language models. arXiv preprint arXiv:2308.15812. ↗
  4. Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604. ↗
  5. Jayadev Bhaskaran and Isha Bhallamudi. 2019. Good secretaries, bad truck drivers? occupational gender stereotypes in sentiment analysis. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 62–68. find
  6. Laura Biester, Vanita Sharma, Ashkan Kazemi, Naihao Deng, Steven Wilson, and Rada Mihalcea. 2022. Analyzing the effects of annotator gender across nlp tasks. In Proceedings of the 1st Workshop on Perspectivist Approaches to NLP@ LREC2022, pages 10–19. find
  7. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of “bias” in nlp. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454–5476.
  8. Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021. Stereotyping norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1004–1015. find
  9. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems, 29. ↗
  10. Tommaso Caselli, Valerio Basile, Jelena Mitrović, and Michael Granitzer. 2021. HateBERT: Retraining BERT for abusive language detection in English. In Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021), pages 17–25, Online. Association for Computational Linguistics.
  11. Jonathan P Chang, Caleb Chiam, Liye Fu, Andrew Wang, Justine Zhang, and Cristian Danescu-Niculescu-Mizil. 2020. Convokit: A toolkit for the analysis of conversations. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 57–60.
  12. Won Ik Cho, Ji Won Kim, Seok Min Kim, and Nam Soo Kim. 2019. On measuring gender bias in translation of gender-neutral pronouns. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 173–181.
  13. Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. 2022. Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Transactions of the Association for Computational Linguistics, 10:92–110. ↗
  14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  15. Ronen Eldan and Yuanzhi Li. 2023. Tinystories: How small can language models be and still speak coherent english? ↗
  16. Ali Emami, Paul Trichelair, Adam Trischler, Kaheer Suleman, Hannes Schulz, and Jackie Chi Kit Cheung. 2019. The knowref coreference corpus: Removing gender and number cues for difficult pronominal anaphora resolution. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3952–3961.
  17. T.N. Flynn and A.A.J. Marley. 2014. Best-worst scaling: theory and methods. In Stephane Hess and Andrew Daly, editors, Handbook of Choice Modelling, Chapters, chapter 8, pages 178–201. Edward Elgar Publishing. ↗
  18. Rishav Hada, Sohi Sudhir, Pushkar Mishra, Helen Yannakoudakis, Saif M. Mohammad, and Ekaterina Shutova. 2021. Ruddit: Norms of offensiveness for English Reddit comments. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2700–2717, Online. Association for Computational Linguistics.
  19. Hossein Hosseini, Sreeram Kannan, Baosen Zhang, and Radha Poovendran. 2017. Deceiving google’s perspective api built for detecting toxic comments. arXiv preprint arXiv:1702.08138. ↗
  20. David Jurgens. 2013. Embracing ambiguity: A comparison of annotation methodologies for crowdsourcing word sense labels. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 556–562, Atlanta, Georgia. Association for Computational Linguistics. find
  21. David A. Jurgens, Peter D. Turney, Saif M. Mohammad, and Keith J. Holyoak. 2012. Semeval-2012 task 2: Measuring degrees of relational similarity. In Proceedings of the First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the Main Conference and the Shared Task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation, SemEval ’12, page 356–364, USA. Association for Computational Linguistics.
  22. Wang-Cheng Kang, Jianmo Ni, Nikhil Mehta, Maheswaran Sathiamoorthy, Lichan Hong, Ed Chi, and Derek Zhiyuan Cheng. 2023. Do llms understand user preferences? evaluating llms on user rating prediction. arXiv preprint arXiv:2305.06474. ↗
  23. Svetlana Kiritchenko and Saif Mohammad. 2017. Best-worst scaling more reliable than rating scales: A case study on sentiment intensity annotation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 465–470, Vancouver, Canada. Association for Computational Linguistics. Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5356–5371, Online. Association for Computational Linguistics.
  24. Svetlana Kiritchenko and Saif M. Mohammad. 2016. Capturing reliable fine-grained sentiment associations by crowdsourcing and best–worst scaling. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 811–817, San Diego, California. Association for Computational Linguistics.
  25. B. Orme. 2009. Maxdiff analysis: Simple counting,individual-level logit, and hb. sawtooth software, inc. find
  26. Svetlana Kiritchenko, Xiaodan Zhu, and Saif Mohammad. 2014. Sentiment analysis of short informal text. The Journal of Artificial Intelligence Research (JAIR), 50. find
  27. Shahar Levy, Koren Lazar, and Gabriel Stanovsky. 2021. Collecting a large-scale gender bias dataset for coreference resolution and machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2470–2480, Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  28. J. J. Louviere. 1991. Best-worst scaling: A model for the largest difference judgments. working paper. find
  29. Jordan J. Louviere, Terry N. Flynn, and A. A. J. Marley. 2015. Best-Worst Scaling: Theory, Methods and Applications. Cambridge University Press. find
  30. Jiaxin Pei and David Jurgens. 2020. Quantifying intimacy in language. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5307–5326, Online. Association for Computational Linguistics. find
  31. Marcelo OR Prates, Pedro H Avelar, and Luís C Lamb. 2020. Assessing gender bias in machine translation: a case study with google translate. Neural Computing and Applications, 32:6363–6381. ↗
  32. Stanley Presser and Howard Schuman. 1996. Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context. SAGE Publications, Inc. find
  33. Bernhard Rieder and Yarden Skop. 2021. The fabrics of machine moderation: Studying the technical, normative, and organizational structure of perspective api. Big Data & Society, 8(2):20539517211046181. ↗
  34. Sarah T Roberts. 2016. Commercial content moderation: Digital laborers’ dirty work. find
  35. Saif Mohammad and Felipe Bravo-Marquez. 2017. Emotion intensities in tweets. In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017), pages 65–77, Vancouver, Canada. Association for Computational Linguistics.
  36. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI spring symposium: logical formalizations of commonsense reasoning, pages 90–95. find
  37. Saif Mohammad and Svetlana Kiritchenko. 2018. Understanding emotions: A dataset of tweets to study interactions between affect categories. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  38. Paul Röttger, Bertie Vidgen, Dirk Hovy, and Janet Pierrehumbert. 2022. Two contrasting data annotation paradigms for subjective nlp tasks. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 175–190.
  39. Saif M Mohammad and Peter D Turney. 2013a. Crowdsourcing a word–emotion association lexicon. Computational intelligence, 29(3):436–465. ↗
  40. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? arXiv preprint arXiv:2303.17548. ↗
  41. Saif M. Mohammad and Peter D. Turney. 2013b. Crowdsourcing a word-emotion association lexicon. CoRR, abs/1308.6297. ↗
  42. Burt L Monroe, Michael P Colaresi, and Kevin M Quinn. 2008. Fightin’words: Lexical feature selection and evaluation for identifying the content of political conflict. Political Analysis, 16(4):372–403. find
  43. Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Karolina Stanczak and Isabelle Augenstein. 2021. A survey on gender bias in natural language processing. arXiv preprint arXiv:2112.14168. ↗
  44. Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating gender bias in natural language processing: Literature review. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1630–1640. ↗
  45. Rajeev Verma, Rajarshi Roychoudhury, and Tirthankar Ghosal. 2022. The lack of theory is painful: Modeling harshness in peer review comments. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 925–935, Online only. Association for Computational Linguistics. find
  46. Bertie Vidgen, Alex Harris, Dong Nguyen, Rebekah Tromble, Scott Hale, and Helen Margetts. 2019. Challenges and frontiers in abusive content detection. Association for Computational Linguistics.
  47. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. ↗
  48. Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018. Mind the gap: A balanced corpus of gendered ambiguous pronouns. Transactions of the Association for Computational Linguistics, 6:605–617. A.3 Prompt: On a fine-grained real-valued scale of 0 to 1, give the following sentence a rating of the degree of gender bias. 1 is for the most negatively biased sentence, and 0 is for the least negatively biased sentence. Only give the score and nothing else. A.4 GPT-4 Reasoning Prompt ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hada-2023-fifty-shades-bias-normative-ratings/