Elamur Research⌕
☰
home / papers / 2023 / "Im not Racist but...": Discovering Bias in the Internal Knowledge of Large Language Models

"Im not Racist but...": Discovering Bias in the Internal Knowledge of Large Language Models

Paper · 2023 · Abel Salinas, Louis Penafiel, Robert McCormack et al.

Summary

Popular LLM bias benchmarks such as Stereoset and CrowS-Pairs rely on fixed, hand-written templates. These templates can leak into future training data, give different results under small rewordings, and do not keep up as social stereotypes change. The authors propose a prompt-only method that needs few templates and surfaces the stereotypes an arbitrary LLM has absorbed. It adapts the knowledge-crawling approach of Cohen et al. (2023). The method starts from seed groups and has the model complete short frames to produce initial subject–predicate–object triples. It then grows a knowledge graph recursively, reusing earlier triples as in-context examples to diversify predicates and objects. Optional prompt tweaks steer generation toward more offensive content. The authors applied the method to GPT-3 with nationality groups and analysed the output using BERTopic along the warmth and competence dimensions of the Stereotype Content Model. The analysis shows that stereotypical associations are distributed very unevenly across groups. The work offers a flexible auditing tool for LLMs. The authors stress that a model knowing a stereotype does not prove it will act on it in downstream tasks.

Key points

  • Seed prompts: Initial triples come from completing "<subject> <predicate>" frames with four predicates: "love", "hate", "are" and "can't". These were chosen to elicit content along both Stereotype Content Model dimensions, warmth and competence.
  • Expansion strategies: The graph grows in two ways.
    • Sampling examples with unique predicates pushes the model toward new predicates.
    • Sampling predicates by inverse frequency fills in objects for rarer relations.
  • Toxicity augmentations: One option prepends extra text to the initial frames. Another adds the header "Offensive Generalizations" to expansion prompts, which shifted the output distribution the most.
  • Topic findings on GPT-3: Several nationalities had no triples in the warmth topic (Topic 4), while over 15% of the knowledge generated for "Canadian people" fell into it. "Russian people" were the most likely to receive competence-related content and among the least likely to receive warmth-related content.
  • Stated limitations:
    • The method recovers only a subset of the model's stereotype knowledge.
    • It does not measure whether bias propagates into downstream tasks.
    • The analysis tools carry their own biases.
    • Intersectional groups and broader sets of protected classes remain unexplored.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: Aptima.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksBias, fairness & safety · 517 works

Cite

Cite as Abel Salinas, Louis Penafiel, Robert McCormack and Fred Morstatter. 2023. "Im not Racist but...": Discovering Bias in the Internal Knowledge of Large Language Models. arXiv:2310.08780.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.08780 · arXiv 2310.08780

@misc{salinas2023im,
  title = {{"Im not Racist but...": Discovering Bias in the Internal Knowledge of Large Language Models}},
  author = {Abel Salinas, Louis Penafiel, Robert McCormack and Fred Morstatter},
  year = {2023},
  doi = {10.48550/arxiv.2310.08780},
  eprint = {2310.08780},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.08780}
}

References

26 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021a. Stereotyping norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In ACL-IJCNLP 2021. find
  2. Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021b. Stereotyping Norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Join… find
  3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  4. Tuhin Chakrabarty, Debanjan Ghosh, Smaranda Muresan, and Nanyun Peng. 2020. Rˆ3: Reverse, retrieve, and rank for sarcasm generation with commonsense knowledge. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7976–7986, Online. Association for Computa…
  5. Roi Cohen, Mor Geva, Jonathan Berant, and Amir Globerson. 2023. Crawling the internal knowledge-base of language models. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1856–1869, Dubrovnik, Croatia. Association for Computational Linguistics.
  6. Kate Crawford. 2017. The trouble with bias. In Conference on Neural Information Processing Systems. Invited speaker. find
  7. Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 862–872… find
  8. Susan T Fiske, Amy JC Cuddy, Peter Glick, and Jun Xu. 2002. A model of (often mixed) stereotype content: competence and warmth respectively follow from perceived status and competition. Journal of personality and social psychology, 82(6):878. find
  9. Maarten Grootendorst. 2022. Bertopic: Neural topic modeling with a class-based tf-idf procedure. arXiv preprint arXiv:2203.05794.
  10. Dirk Hovy and Shrimai Prabhumoye. 2021. Five sources of bias in natural language processing. Language and Linguistics Compass, 15(8):e12432. find
  11. Hannah Rose Kirk, Yennie Jun, Filippo Volpin, Haider Iqbal, Elias Benussi, Frederic Dreyer, Aleksandar Shtedritski, and Yuki Asano. 2021. Bias out-of-the-box: An empirical analysis of intersectional occupational biases in popular generative language models. Advances in neural information processing… find
  12. Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. 2019. KagNet: Knowledge-aware graph networks for commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-… find
  13. Leland McInnes, John Healy, and Steve Astels. 2017. hdbscan: Hierarchical density based clustering. The Journal of Open Source Software, 2(11):205.
  14. Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2021a. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR), 54(6):1–35.
  15. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  16. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural…
  17. Simon Razniewski, Andrew Yates, Nora Kassner, and Gerhard Weikum. 2021. Language models as or for knowledge bases. find
  18. Timo Schick, Sahana Udupa, and Hinrich Schütze. 2021. Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in nlp. Transactions of the Association for Computational Linguistics, 9:1408– 1424.
  19. Preethi Seshadri, Pouya Pezeshkpour, and Sameer Singh. 2022. Quantifying social biases using templates is unreliable. arXiv preprint arXiv:2210.04337. — [arXiv:2210.04337](https://arxiv.org/abs/2210.04337) find
  20. Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. 2019a. The woman worked as a babysitter: On biases in language generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing…
  21. Ninareh Mehrabi, Pei Zhou, Fred Morstatter, Jay Pujara, Xiang Ren, and Aram Galstyan. 2021b. Lawyers are dishonest? quantifying representational harms in commonsense knowledge resources. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5016–5033, Onlin… find
  22. Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019b. The woman worked as a babysitter: On biases in language generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proces…
  23. Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long…
  24. Vered Shwartz, Rachel Rudinger, and Oyvind Tafjord. 2020. “you are grounded!”: Latent name artifacts in pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6850–6861, Online. Association for Computational Linguistics. find
  25. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953–1967, Online. Association for…
  26. Hao Zhou, Tom Young, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. 2018. Commonsense knowledge aware conversation generation with graph attention. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 4623–4629. International Join… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/salinas-2023-im-racist-but-discovering-bias/