Elamur Research⌕
☰
home / papers / 2023 / Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs

Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs

Paper · 2023 · Shashank Gupta, Vaishnavi Shrivastava, Ameet Deshpande et al.

Summary

The paper asks whether telling an LLM to adopt a socio-demographic identity changes its ability to solve reasoning problems that have nothing to do with that identity. The authors placed persona instructions in the system prompt, using three wordings, and evaluated the models zero-shot. Their 19 personas were drawn from five categories, spanning political affiliation, religion, disability, gender and race. The test set had 24 datasets: 22 subsets of MMLU, Sports Understanding from Big-Bench-Hard, and MBPP. Four models were tested: gpt-3.5-turbo-0613, gpt-3.5-turbo-1106, gpt-4-turbo-1106 and Llama-2-70b-chat.

When asked directly, the models deny stereotypes. Yet with a persona assigned, they often refuse tasks based on false assumptions about that group, or they quietly make more mistakes, and accuracy falls noticeably as a result. Asking the model in the prompt to avoid stereotypes barely helped. The work matters because persona prompting is spreading quickly, and it can expose hidden bias that is difficult to notice or fix.

Key points

  • With ChatGPT-3.5 (June 2023), 80% of personas performed worse on at least one dataset. Some datasets showed relative accuracy losses above 70%, and certain personas lost significantly on more than 80% of datasets.
  • The "physically-disabled person" persona scored on average 33% lower than the able-bodied persona and was affected on 96% of datasets. About 58% of its errors were explicit refusals.
  • The share of biased personas differed by model: 100% for the November 2023 ChatGPT-3.5, 80% for Llama-2-70b-chat, and 42% for GPT-4-Turbo, the lowest but still substantial.
  • Bias also appeared inside a single group. For example, the Atheist persona beat the Christian one on science subsets, and the Obama Supporter persona beat the Trump Supporter one on ethics.
  • The authors release their code and roughly 1.5 million model outputs. The paper was accepted at ICLR 2024.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States506 works · 330,052 citations🔬Allen Institute for Artificial IntelligenceSeattle · United States304 works · 106,138 citations🎓Princeton UniversityPrinceton · United States113 works · 48,612 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSafety Research · 626 worksSociology and Political Science · 626 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksReasoning · 1,685 works

Cite

Cite as Shashank Gupta, Vaishnavi Shrivastava, Ameet Deshpande, Ashwin Kalyan, Peter Clark, Ashish Sabharwal and Tushar Khot. 2023. Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs. arXiv:2311.04892.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2311.04892 · arXiv 2311.04892

@misc{gupta2023bias,
  title = {{Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs}},
  author = {Shashank Gupta, Vaishnavi Shrivastava, Ameet Deshpande, Ashwin Kalyan, Peter Clark, Ashish Sabharwal and Tushar Khot},
  year = {2023},
  doi = {10.48550/arxiv.2311.04892},
  eprint = {2311.04892},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2311.04892}
}

References

38 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gati Aher, Rosa I. Arriaga, and Adam T. Kalai. Using large language models to simulate multiple humans and replicate human subject studies. In ICML, 2023. find
  2. Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate. Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3):337–351, 2023.
  3. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. Program synthesis with large language models. ArXiv, abs/2108.07732, 2021.
  4. Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. The problem with bias: From allocative to representational harms in machine learning. In SIGCIS conference paper, 2017. find
  5. Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam T. Kalai. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In NeurIPS, 2016.
  6. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, John A. Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuan-Fang Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4. ArXiv, abs/2…
  7. Myra Cheng, Esin Durmus, and Dan Jurafsky. Marked personas: Using natural language prompts to measure stereotypes in language models. ArXiv, abs/2305.18189, 2023.
  8. David Danks and Alex John London. Algorithmic bias in autonomous systems. In IJCAI, 2017. find
  9. Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. Toxicity in chatgpt: Analyzing persona-assigned language models. ArXiv, abs/2304.05335, 2023.
  10. Sunipa Dev, Tao Li, J. M. Phillips, and Vivek Srikumar. On measuring and mitigating biased inferences of word embeddings. In AAAI, 2019. find
  11. Jonas Freiknecht and Wolfgang Effelsberg. Procedural generation of interactive stories using language models. In Proceedings of the 15th International Conference on the Foundations of Digital Games, pp. 1–8, 2020. find
  12. Perttu Hämäläinen, Mikke Tavast, and Anton Kunnari. Evaluating large language models in generating synthetic hci research data: a case study. In CHI, 2023. find
  13. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song, and Jacob Steinhardt. Measuring massive multitask language understanding. ArXiv, abs/2009.03300, 2020.
  14. Joseph Henrich, Steven J. Heine, and Ara Norenzayan. Most people are not weird. Nature, 466: 29–29, 2010. find
  15. John J. Horton. Large language models as simulated economic agents: What can we learn from homo silicus? Technical report, National Bureau of Economic Research, 2023. find
  16. Peter A. Jansen. From words to wires: Generating functioning electronic devices from natural language descriptions. ArXiv, abs/2305.14874, 2023. find
  17. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In NeurIPS, 2022.
  18. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
  19. Tao Li, Daniel Khashabi, Tushar Khot, Ashish Sabharwal, and Vivek Srikumar. Unqovering stereotypical biases via underspecified questions. In Findings@EMNLP, 2020. find
  20. Yingji Li, Mengnan Du, Rui Song, Xin Wang, and Y. Wang. A survey on fairness in large language models. ArXiv, abs/2308.10149, 2023. — [arXiv:2308.10149](https://arxiv.org/abs/2308.10149) find
  21. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for llm compression and acceleration, 2023. find
  22. OpenAI. Custom instructions for chatgpt, 2023a. https://openai.com/blog/custom-instructions-for-chatgpt. ↗
  23. OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023b.
  24. Jeongeon Park and Daeun Choi. Audilens: Configurable llm-generated audiences for public speech practice. Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, 2023. find
  25. Joon Sung Park, Lindsay Popowski, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Social simulacra: Creating populated prototypes for social computing systems. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, pp. 1–18, 2022.
  26. Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. In UIST, 2023. find
  27. Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development. ArXiv, abs/2307.07924, 2023.
  28. Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. Is chatgpt a general-purpose natural language processing task solver? ArXiv, abs/2302.06476, 2023.
  29. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. In NAACL, 2018.
  30. Leonard Salewski, Stephan Alaniz, Isabel Rio-Torto, Eric Schulz, and Zeynep Akata. In-context impersonation reveals large language models’ strengths and biases. ArXiv, abs/2305.14930, 2023. — [arXiv:2305.14930](https://arxiv.org/abs/2305.14930) find
  31. Emily Sheng, Josh Arnold, Zhou Yu, Kai-Wei Chang, and Nanyun Peng. Revealing persona biases in dialogue systems. ArXiv, abs/2104.08728, 2021. — [arXiv:2104.08728](https://arxiv.org/abs/2104.08728) find
  32. Mirac Suzgun, Nathan Scales, Nathanael Scharli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed Huai hsin Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. In Annual Meeting of the Association for Computational L…
  33. Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Bri… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  34. Yixin Wan, Jieyu Zhao, Nanyun Peng, Kai-Wei Chang, and Aman Chadha. Are personalized stochastic parrots more dangerous? Evaluating persona biases in dialogue systems. ArXiv, abs/2310.05280, 2023. find
  35. Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Unleashing cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration. ArXiv, abs/2307.05300, 2023.
  36. Edwin B Wilson. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 1927. find
  37. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. In NAACL, 2018.
  38. Jieyu Zhao, Daniel Khashabi, Tushar Khot, Ashish Sabharwal, and Kai-Wei Chang. Ethical-advice taker: Do language models understand natural language interventions? In Findings@ACL, 2021. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gupta-2023-bias-runs-deep-implicit-reasoning/