Elamur Research⌕
☰
home / papers / 2026 / Talent or Luck? Evaluating Attribution Bias in Large Language Models

Talent or Luck? Evaluating Attribution Bias in Large Language Models

Paper · 2026 · Chahat Raj, Mahika Banerjee, Jinhao Pan et al.

Summary

Most social-bias benchmarks for LLMs test surface associations, such as linking an identity to an occupation or trait. They rarely examine how a model explains why an outcome happened. This paper uses Attribution Theory from psychology to fill that gap. Heider's distinction between internal and external causes and Weiner's four causes (effort, ability, task difficulty, luck) become a bias-evaluation framework.

The authors built a benchmark of roughly 140k prompts from 400 templates. Each prompt describes a person succeeding or failing in one of ten societal scenarios. The model must choose among four explanations, each mapped to one attribution type. Identity is signalled through names. The framework has three settings:

  • Single-Actor: one person's outcome.
  • Actor-Actor: two people compared in the same situation.
  • Actor-Observer: a second identity or distracting context is present.

Bias is quantified with an internal–external differential, d, measuring how strongly a model prefers internal over external causes for a given group. In three open LLMs, attributions tend to favour dominant groups and disadvantage minorities. These asymmetries change when other identities appear. This suggests bias should also be audited in comparative, multi-person contexts.

Key points

  • The benchmark has about 140k prompts built from 400 templates. Scenario templates were generated with GPT-4o. Code and data are released at github.com/chahatraj/TalentorLuck.
  • It covers four social dimensions: binary gender (treated intersectionally), 15 nationalities, 6 racial groups and 6 religions. Each group is represented by five male and five female names.
  • Each of the ten scenarios comes in both a success and a failure version, and together they span Law & Policy, Sports, Economics, Healthcare, Media, Education, Environment, Workplace, Technology and Art & Leisure.
  • The models evaluated are Aya-Expanse-8B, Qwen-32B and LLaMA-3.3-70B. Effort and ability count as internal causes; difficulty and luck count as external causes. The analysis covers only the locus dimension of attribution, not stability or controllability.
  • Stated limitations: answers are closed-ended with a fixed four-cause taxonomy, and there is no ground truth for the "correct" attribution. Results therefore show asymmetry, not factual error. Open-ended generation is left for future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSafety Research · 543 worksSociology and Political Science · 577 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Chahat Raj, Mahika Banerjee, Jinhao Pan, Aylin Caliskan, Antonios Anastasopoulos and Ziwei Zhu. 2026. Talent or Luck? Evaluating Attribution Bias in Large Language Models. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.534

@inproceedings{raj2026talent,
  title = {{Talent or Luck? Evaluating Attribution Bias in Large Language Models}},
  author = {Chahat Raj, Mahika Banerjee, Jinhao Pan, Aylin Caliskan, Antonios Anastasopoulos and Ziwei Zhu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.534},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.534}
}

References

38 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Haozhe An, Christabel Acquaye, Colin Wang, Zongxia Li, and Rachel Rudinger. 2024. Do large language models discriminate in hiring decisions on the basis of race, ethnicity, and gender? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)… find
  2. Haozhe An and Rachel Rudinger. 2023. Nichelle and nancy: The influence of demographic attributes and tokenization length on first name biases. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 388–401, Toronto, Canada. Associat… find
  3. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623. find
  4. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems, 29.
  5. Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. find
  6. Sunipa Dev, Emily Sheng, Jieyu Zhao, Aubrie Amstutz, Jiao Sun, Yu Hou, Mattie Sanseverino, Jiin Kim, Akihiro Nishi, Nanyun Peng, et al. 2021. On measures of biases and harms in nlp. arXiv preprint arXiv:2108.03362. find
  7. Jessica Maria Echterhoff, Yao Liu, Abeer Alessa, Julian McAuley, and Zexue He. 2024. Cognitive bias in decision-making with LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 12640–12653, Miami, Florida, USA. Association for Computational Linguistics. find
  8. Sandra Graham and Valerie S Folkes. 2014. Attribution theory: Applications to achievement, mental health, and interpersonal conflict. Psychology Press. find
  9. Fritz Heider. 2013. The psychology of interpersonal relations. Psychology Press. find
  10. Edward E Jones and Richard E Nisbett. 1987. The actor and the observer: Divergent perceptions of the causes of behavior. In Preparation of this paper grew out of a workshop on attribution theory held at University of California, Los Angeles, Aug 1969. Lawrence Erlbaum Associates, Inc. find
  11. M Kamruzzaman and GL Kim. 2024. Prompting techniques for reducing social bias in llms through system 1 and system 2 cognitive processes. arXiv preprint arXiv:2404.17218. find
  12. Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. Quantifying social biases in contextual word representations. In 1st ACL Workshop on Gender Bias for Natural Language Processing. find
  13. Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2021. Towards understanding and mitigating social biases in language models. In International conference on machine learning, pages 6565–6576. PMLR.
  14. Bertram F Malle. 2006. The actor-observer asymmetry in attribution: a (surprising) meta-analysis. Psychological bulletin, 132(6):895. find
  15. Rowan Hall Maudslay, Hila Gonen, Ryan Cotterell, and Simone Teufel. 2019. It’s all in the name: mitigating gender bias with name-based counterfactual data substitution. arXiv preprint arXiv:1909.00871.
  16. Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2021. A survey on bias and fairness in machine learning. ACM computing surveys (CSUR), 54(6):1–35.
  17. Anjishnu Mukherjee, Aylin Caliskan, Ziwei Zhu, and Antonios Anastasopoulos. 2024. Global gallery: The fine art of painting culture portraits through multilingual instruction tuning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics:… find
  18. Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long…
  19. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953–1967, Online. Association for…
  20. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022. BBQ: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2086–2105, Dublin, Ireland. Asso…
  21. Chahat Raj, Anjishnu Mukherjee, Aylin Caliskan, Antonios Anastasopoulos, and Ziwei Zhu. 2024. Breaking bias, building bridges: Evaluation and mitigation of social biases in llms via contact hypothesis. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 7, pages 1180–1189. find
  22. Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null it out: Guarding protected attributes by iterative nullspace projection. arXiv preprint arXiv:2004.07667.
  23. Jessica A Robinson. 2017. Exploring attribution theory and bias. Communication Teacher, 31(4):209–213. find
  24. Lee Ross. 1977. The intuitive psychologist and his shortcomings: Distortions in the attribution process. In Advances in experimental social psychology, volume 10, pages 173–220. Elsevier. find
  25. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 8–1…
  26. Patrick Schramowski, Cigdem Turan, Nico Andersen, Constantin A Rothkopf, and Kristian Kersting. 2022. Large pre-trained language models contain human-like biases of what is right and wrong to do. Nature Machine Intelligence, 4(3):258–268. find
  27. Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. 2021. Societal biases in language generation: Progress and challenges. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (V…
  28. Eric Michael Smith, Melissa Hall, Melanie Kambadur, Eleonora Presani, and Adina Williams. 2022. " i’m sorry to hear that": Finding new biases in language models with a holistic descriptor dataset. arXiv preprint arXiv:2205.09209.
  29. Yasuaki Sumita, Koh Takeuchi, and Hisashi Kashima. 2024. Cognitive biases in large language models: A survey and mitigation experiments. arXiv preprint arXiv:2412.00323. — [arXiv:2412.00323](https://arxiv.org/abs/2412.00323) find
  30. Philip E Tetlock and Ariel Levi. 1982. Attribution bias: On the inconclusiveness of the cognition-motivation debate. Journal of Experimental Social Psychology, 18(1):68–88. find
  31. Yixin Wan, George Pu, Jiao Sun, Aparna Garimella, Kai-Wei Chang, and Nanyun Peng. 2023. “kelly is a warm person, joseph is a role model”: Gender biases in LLM-generated reference letters. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3730–3748, Singapore. Associatio…
  32. Bernard Weiner. 1985. An attributional theory of achievement motivation and emotion. Psychological review, 92(4):548. find
  33. Yuchen Wen, Keping Bi, Wei Chen, Jiafeng Guo, and Xueqi Cheng. 2024. Evaluating implicit bias in large language models by attacking from a psychometric perspective. arXiv preprint arXiv:2406.14023. find
  34. Kyra Wilson and Aylin Caliskan. 2024. Gender, race, and intersectional bias in resume screening via language model retrieval. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 7, pages 1578–1590. find
  35. Xi Ye and Greg Durrett. 2022. The unreliability of explanations in few-shot in-context learning. arXiv preprint arXiv:2205.03401, 67.
  36. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. arXiv preprint arXiv:1707.09457.
  37. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. arXiv preprint arXiv:1804.06876.
  38. Yachao Zhao, Bo Wang, and Yan Wang. 2025. Explicit vs. implicit: Investigating social bias in large language models through self-reflection. arXiv preprint arXiv:2501.02295. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/raj-2026-talent-or-luck-evaluating-attribution/