Elamur Research⌕
☰
home / papers / 2026 / FAIRGAMER: Evaluating Social Biases in LLM-Based Video Game NPCs

FAIRGAMER: Evaluating Social Biases in LLM-Based Video Game NPCs

Paper · 2026 · Bingkang Shi, Jen-tse Huang, Luo Long et al.

Summary

LLMs increasingly drive non-player characters (NPCs) in video games, and they can carry social stereotypes into gameplay. FAIRGAMER is presented as the first benchmark for measuring how such biases distort NPC decisions. The authors frame NPC behaviour through game theory and define three interaction modes: transaction (bargaining over discounts), cooperation (allocating resources) and competition (zero-sum contests). Each mode is crossed with four bias dimensions: class (in-game occupation), race, age and nationality, giving 12 tasks. Demographic attributes come from real-world sources (Wikipedia) and from fantasy and sci-fi games. Each prompt assigns identities to the model's own character and its counterpart. The model's JSON decisions are sampled repeatedly, and their consistency across identities is scored with a new metric, FairMCV, based on a multivariate coefficient of variation. Higher FairMCV means fairer decisions.

All seven tested frontier models showed biased behaviour, and larger models tended to be more biased. Chain-of-Thought prompting and fine-tuning reduce the bias only partly. The work matters for anyone deploying LLM agents in interactive or economic settings.

Key points

  • The dataset has 16,910 bilingual (English/Chinese) test cases built from 199 demographic attributes. The sources are ten Steam games (e.g., Baldur's Gate 3, Elden Ring, Civilization, Stellaris) and Wikipedia.
  • Seven LLMs were evaluated (3 closed-source, 4 open-source), with each prompt sampled ten times.
  • Grok-4-Fast was the most biased (average FairMCV 76.9%), while LLaMA-3.1-8B was the fairest (85.9%).
  • A Chain-of-Thought instruction raised average FairMCV to 83.6% for DeepSeek-V3.2 (+5.7) and to 88.0% for LLaMA-3.1-8B (+2.1). LoRA SFT on LLaMA-3.1-8B reached 89.7%.
  • The benchmark is released at github.com/BingkangShi/FairGamer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Johns Hopkins UniversityBaltimore · United States240 works · 96,487 citations🏛️Chinese Academy of SciencesBeijing · China194 works · 23,906 citations🎓University of Chinese Academy of SciencesBeijing · China147 works · 8,480 citations🔬Institute of Information EngineeringBeijing · China30 works · 1,411 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSafety Research · 543 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Bias, fairness & safety · 517 works

Cite

Cite as Bingkang Shi, Jen-tse Huang, Luo Long, Tianyu Zong, Hongzhu Yi, Yuanxiang Wang, Songlin Hu, Xiaodan Zhang and Zhongjiang Yao. 2026. FAIRGAMER: Evaluating Social Biases in LLM-Based Video Game NPCs. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.2015

@inproceedings{shi2026fairgamer,
  title = {{FAIRGAMER: Evaluating Social Biases in LLM-Based Video Game NPCs}},
  author = {Bingkang Shi, Jen-tse Huang, Luo Long, Tianyu Zong, Hongzhu Yi, Yuanxiang Wang, Songlin Hu, Xiaodan Zhang and Zhongjiang Yao},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2015},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2015}
}

References

56 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jinhao Duan, Renming Zhang, James Diffenderfer, Bhavya Kailkhura, Lichao Sun, Elias Stengel-Eskin, Mohit Bansal, Tianlong Chen, and Kaidi Xu. 2024. Gtbench: Uncovering the strategic reasoning limitations of llms via game-theoretic evaluations. arXiv preprint arXiv:2402.12348. find
  2. Sahar Abdelnabi, Amr Gomaa, Sarath Sivaprasad, Lea Schönherr, and Mario Fritz. 2024. Cooperation, competition, and maliciousness: Llm-stakeholders interactive negotiation. Advances in Neural Information Processing Systems, 37:83548–83599. find
  3. Seyed Hossein Alavi, Weijia Xu, Nebojsa Jojic, Daniel Kennett, Raymond T Ng, Sudha Rao, Haiyan Zhang, Bill Dolan, and Vered Shwartz. 2024. Game plot design with an llm-powered assistant: An empirical study with game designers. arXiv preprint arXiv:2411.02714. anuttacon. 2025. Whispers from the Star.… — [arXiv:2411.02714](https://arxiv.org/abs/2411.02714) find
  4. David Baidoo-Anu and Leticia Owusu Ansah. 2023. Education in the era of generative artificial intelligence (ai): Understanding the potential benefits of chatgpt in promoting teaching and learning. Journal of AI, 7(1):52–62. find
  5. Bauhinia AI. 2025. Aivilization. aivilization.ai/. Video Game. https:// find
  6. Angana Borah and Rada Mihalcea. 2024. Towards implicit bias detection and mitigation in multi-agent llm interactions. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9306– 9326. find
  7. Yongkang Du, Jen-tse Huang, Jieyu Zhao, and Lu Lin. 2025. Faircoder: Evaluating social bias of llms in code generation. arXiv preprint arXiv:2501.05396. — [arXiv:2501.05396](https://arxiv.org/abs/2501.05396) find
  8. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  9. Caoyun Fan, Jindou Chen, Yaohui Jin, and Hao He. 2024. Can large language models serve as rational players in game theory? a systematic analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17960–17967. find
  10. Virginia Felkner, Ho-Chun Herbert Chang, Eugene Jang, and Jonathan May. 2023. Winoqueer: A community-in-the-loop benchmark for anti-lgbtq+ bias in large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9126–914… find
  11. Neel Guha, Julian Nyarko, Daniel E Ho, Christopher Re, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, and 1 others. 2023. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. In Thirty-seventh Confere…
  12. Yue Guo, Yi Yang, and Ahmed Abbasi. 2022. Auto-debias: Debiasing masked language models with automated biased prompts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1012–1023.
  13. Jen-tse Huang, Eric John Li, Man Ho Lam, Tian Liang, Wenxuan Wang, Youliang Yuan, Wenxiang Jiao, Xing Wang, Zhaopeng Tu, and Michael R Lyu. 2024. How far are we on the decision-making of llms? evaluating llms’ gaming ability in multi-agent environments. arXiv preprint arXiv:2403.11807. — [arXiv:2403.11807](https://arxiv.org/abs/2403.11807) find
  14. Shachi H Kumar, Saurav Sahay, Sahisnu Mazumder, Eda Okur, Ramesh Manuvinakurike, Nicole Beckage, Hsuan Su, Hung-yi Lee, and Lama Nachman. 2024. Decoding biases: Automated methods and llm judges for gender bias detection in language models. arXiv preprint arXiv:2408.03907. — [arXiv:2408.03907](https://arxiv.org/abs/2408.03907) find
  15. Cheryl Lee, Chunqiu Steven Xia, Longji Yang, Jen-tse Huang, Zhouruixin Zhu, Lingming Zhang, and Michael R Lyu. 2024. A unified debugging approach via llm-based multi-agent synergy. arXiv preprint arXiv:2404.17153. — [arXiv:2404.17153](https://arxiv.org/abs/2404.17153) find
  16. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. find
  17. Jen-tse Huang, Jiantong Qin, Jianping Zhang, Youliang Yuan, Wenxuan Wang, and Jieyu Zhao. 2025a. Vis-bias: Measuring explicit and implicit social biases in vision language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 17981–18004. find
  18. Hanjun Luo, Haoyu Huang, Ziye Deng, Xuecheng Liu, Ruizhe Chen, and Zuozhu Liu. 2024. Bigbench: A unified benchmark for social bias in text-to-image generative models based on multi-modal llm. arXiv preprint arXiv:2407.15240. — [arXiv:2407.15240](https://arxiv.org/abs/2407.15240) find
  19. Jen-tse Huang, Yuhang Yan, Linqi Liu, Yixin Wan, Wenxuan Wang, Kai-Wei Chang, and Michael R Lyu. 2025b. Where fact ends and fairness begins: Redefining ai bias evaluation through cognitive biases. Findings of the Association for Computational Linguistics: EMNLP. find
  20. Alessandro Marincioni, Myriana Miltiadous, Katerina Zacharia, Rick Heemskerk, Georgios Doukeris, Mike Preuss, and Giulio Barbero. 2024. The effect of llm-based npc emotional states on player emotions: An analysis of interactive game play. In 2024 IEEE Conference on Games (CoG), pages 1–6. IEEE. find
  21. Jingyuan Huang, Jen-tse Huang, Ziyi Liu, Xiaoyuan Liu, Wenxuan Wang, and Jieyu Zhao. 2025c. Ai sees your location—but with a bias toward the wealthy world. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 18030–18050. find
  22. Chandler May, Alex Wang, Shikha Bordia, Samuel Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Sho…
  23. Inge Huijsmans, Ili Ma, Leticia Micheli, Claudia Civai, Mirre Stallen, and Alan G Sanfey. 2019. A scarcity mindset alters neural processing underlying consumer decision making. Proceedings of the National Academy of Sciences, 116(24):11699–11704. find
  24. Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is chatgpt a good translator? a preliminary study. arXiv preprint arXiv:2301.08745.
  25. Douglas Johnson, Rachel Goodman, J Patrinely, Cosby Stone, Eli Zimmerman, Rebecca Donald, Sam Chang, Sean Berkowitz, Avni Finn, Eiman Jahangir, and 1 others. 2023. Assessing the accuracy and reliability of ai-generated medical responses: an evaluation of the chat-gpt model. Research square. find
  26. Tarek Naous, Michael J Ryan, Alan Ritter, and Wei Xu. 2024. Having beer after prayer? measuring cultural bias in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16366–16393.
  27. John F Nash. 2024. Non-cooperative games. In The Foundations of Price Theory Vol 4, pages 329–340. Routledge. find
  28. John F Nash and 1 others. 1950. The bargaining problem. Econometrica, 18(2):155–162. find
  29. OpenAI. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  30. Daphne Koller and Avi Pfeffer. 1997. Representations and solutions for game-theoretic problems. Artificial intelligence, 94(1-2):167–215. find
  31. Ciprian Paduraru, Adelina Staicu, and Alin Stefanescu. 2024. Llm-based methods for the creation of unit tests in game development. Procedia Computer Science, 246:2459–2468. find
  32. James Konow. 2000. Fair shares: Accountability and cognitive dissonance in allocation decisions. American economic review, 90(4):1072–1092. find
  33. Xiangyu Peng, Jessica Quaye, Sudha Rao, Weijia Xu, Portia Botchway, Chris Brockett, Nebojsa Jojic, Gabriel DesGarennes, Ken Lobb, Michael Xu, and 1 others. 2024. Player-driven emergence in llm-driven game narrative. In 2024 IEEE Conference on Games (CoG), pages 1–8. IEEE. find
  34. Proxima. 2024. Suck Up! https://www.playsuckup.com/. Video Game. ↗
  35. Dan Qiao, Chenfei Wu, Yaobo Liang, Juntao Li, and Nan Duan. 2023. Gameeval: Evaluating llms on conversational games. arXiv preprint arXiv:2308.10032. — [arXiv:2308.10032](https://arxiv.org/abs/2308.10032) find
  36. Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt a general-purpose natural language processing task solver? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1339–1384.
  37. Jillian Ross, Yoon Kim, and Andrew W Lo. 2024. Llm economicus? mapping the behavioral biases of llms via utility theory. arXiv preprint arXiv:2408.02784. — [arXiv:2408.02784](https://arxiv.org/abs/2408.02784) find
  38. Caroline Roux, Kelly Goldsmith, and Andrea Bonezzi. 2015. On the psychology of scarcity: When reminders of resource scarcity promote selfish (and generous) behavior. Journal of consumer research, 42(4):615–631. find
  39. Lloyd S Shapley and 1 others. 1953. A value for n-person games. find
  40. Bingkang Shi, Xiaodan Zhang, Dehan Kong, Yulei Wu, Zongzhen Liu, Honglei Lyu, and Longtao Huang. 2024. General phrase debiaser: Debiasing masked language models at a multi-token level. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6345–6… find
  41. Zhengliang Shi, Ruotian Ma, Jen-tse Huang, Xinbei Ma, Xingyu Chen, Mengru Wang, Qu Yang, Yue Wang, Fanghua Ye, Ziyang Chen, and 1 others. 2025. Social welfare function leaderboard: When llm agents allocate social welfare. arXiv preprint arXiv:2510.01164. — [arXiv:2510.01164](https://arxiv.org/abs/2510.01164) find
  42. Rickard Stureborg, Dimitris Alikaniotis, and Yoshi Suhara. 2024. Large language models are inconsistent and biased evaluators. arXiv preprint arXiv:2405.01724.
  43. Amir Taubenfeld, Yaniv Dover, Roi Reichart, and Ariel Goldstein. 2024. Systematic biases in llm simulations of debates. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 251–267.
  44. John Von Neumann and Oskar Morgenstern. 2007. Theory of games and economic behavior: 60th anniversary commemorative edition. In Theory of games and economic behavior. Princeton university press. find
  45. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291. find
  46. Sibo Wang, Xiangkui Cao, Jie Zhang, Zheng Yuan, Shiguang Shan, Xilin Chen, and Wen Gao. 2024a. Vlbiasbench: A comprehensive benchmark for evaluating bias in large vision-language model. arXiv preprint arXiv:2406.14194. — [arXiv:2406.14194](https://arxiv.org/abs/2406.14194) find
  47. Yixu Wang, Yan Teng, Kexin Huang, Chengqi Lyu, Songyang Zhang, Wenwei Zhang, Xingjun Ma, Yu-Gang Jiang, Yu Qiao, and Yingchun Wang. 2024b. Fake alignment: Are llms really aligned well? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistic… find
  48. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837.
  49. Haoran Wu, Wenxuan Wang, Yuxuan Wan, Wenxiang Jiao, and Michael Lyu. 2023a. Chatgpt or grammarly? evaluating chatgpt on grammatical error correction benchmark. arXiv preprint arXiv:2303.13648. find
  50. Yue Wu, Xuan Tang, Tom M Mitchell, and Yuanzhi Li. 2023b. Smartplay: A benchmark for llms as intelligent agents. arXiv preprint arXiv:2310.01557. xAI. 2025. Grok 3 beta — the age of reasoning agents. find
  51. Lin Xu, Zhiyuan Hu, Daquan Zhou, Hongyu Ren, Zhen Dong, Kurt Keutzer, See Kiong Ng, and Jiashi Feng. 2024a. Magic: Investigation of large language model powered multi-agent in cognition, adaptability, rationality and collaboration. In Proceedings of the 2024 Conference on Empirical Methods in Natura… find
  52. Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li, and William Wang. 2024b. Pride and prejudice: Llm amplifies self-bias in self-refinement. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15474–15492.
  53. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, and 1 others. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671. find
  54. Jizhi Zhang, Keqin Bao, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. 2023. Is chatgpt fair for recommendation? evaluating fairness in large language model recommendation. In Proceedings of the 17th ACM Conference on Recommender Systems, pages 993–999. find
  55. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623.
  56. Fan Zhou, Yuzhou Mao, Liu Yu, Yi Yang, and Ting Zhong. 2023. Causal-debias: Unifying debiasing in pretrained language models and fine-tuning via causal invariant learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4227–… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shi-2026-fairgamer-evaluating-social-biases-llm/