Elamur Research⌕
☰
home / papers / 2026 / InsideOut: Measuring and Mitigating Insider–Outsider Bias in Interview Script Generation

InsideOut: Measuring and Mitigating Insider–Outsider Bias in Interview Script Generation

Paper · 2026 · Yixin Wan, Xingrun Chen and Kai-Wei Chang

Summary

The paper examines a subtle cultural bias in LLMs: when writing text "about" a culture, models tend to speak as members of dominant cultures, especially the US, while treating other cultures as foreign objects of curiosity. The authors call this insider–outsider bias and build the InsideOut benchmark to measure it. The benchmark has 4,000 prompts in which a model plays a journalist interviewing locals in 10 cultures. An LLM judge then decides whether each generated script sounds like an insider or an outsider. The authors also propose three metrics (CEP, CPD, CAG) that capture how this stance shifts across cultures.

They tested five LLMs. On average, more than 88% of US-set scripts took an insider stance, while non-Western contexts such as Papua New Guinea mostly received an outsider framing.

For mitigation, they compare a prompt-based baseline (FIP) with agent-based pipelines (MFA). The agent-based pipelines cut the bias substantially and consistently outperformed FIP. This suggests that inference-time agentic refinement is a practical way to reduce culturally hegemonic framing without retraining.

Key points

  • InsideOut benchmark: 4,000 interview-script prompts covering 10 cultures. The planned release is code at github.com/LyannaChen/InsideOut and data at huggingface.co/datasets/elaine1wan/insideout_data.
  • Metrics and models: Three metrics are used: CEP, which stands for Cultural Externality Percentage; CPD, short for Cultural Perspective Deviation; and CAG, the Cultural Alignment Gap. The models evaluated are ChatGPT, Llama, Mistral, Deepseek and Qwen.
  • Mitigation methods:
    • FIP injects fairness guidelines through the prompt.
    • MFA-SA has a single agent critique and refine its own output.
    • MFA-HA splits the work between a Critique Agent and a Refinement Agent.
    • MFA-Plan runs an iterative loop of critique, change plan, rewrite and validation.
  • Average reductions across the five models: MFA-SA lowered CPD by 53.81% and CAG by 60.84%. MFA-HA lowered CPD by 59.70% and CAG by 62.52%. MFA-Plan was reported only for ChatGPT, where it gave the best scores (CPD −77.00%, CAG −77.78%).
  • Best single-model CAG results: MFA-SA on Llama reduced CAG by 89.70%, and MFA-HA on Qwen by 82.54%. The mitigation experiments used a 500-item subset of InsideOut. The authors note limitations: a small set of cultures and English-only output.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California, Los AngelesLos Angeles · United States87 works · 54,961 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

Cite

Cite as Yixin Wan, Xingrun Chen and Kai-Wei Chang. 2026. InsideOut: Measuring and Mitigating Insider–Outsider Bias in Interview Script Generation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1094

@inproceedings{wan2026insideout,
  title = {{InsideOut: Measuring and Mitigating Insider–Outsider Bias in Interview Script Generation}},
  author = {Yixin Wan, Xingrun Chen and Kai-Wei Chang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1094},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1094}
}

References

53 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Badr AlKhamissi, Muhammad ElNokrashy, Mai Alkhamissi, and Mona Diab. 2024. Investigating cultural alignment of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12404–12422, Bangkok, Thailand. Association…
  2. Samir Amin. 1989. Eurocentrism. NYU Press. find
  3. Bushra Asseri, Estabrag Abdelaziz, and Areej Al-Wabil. 2025. Prompt engineering techniques for mitigating cultural bias against arabs and muslims in large language models: A systematic review. arXiv preprint arXiv:2506.18199. — [arXiv:2506.18199](https://arxiv.org/abs/2506.18199) find
  4. Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017. The problem with bias: From allocative to representational harms in machine learning. In Proceedings of the 9th Annual Conference of the Special Interest Group for Computing, Information and Society (SIGCIS), Philadelphia, PA. Ass… find
  5. Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. 2024. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954. find
  6. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. J. Mach. Learn. Res., 3(null):993–1022.
  7. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of “bias” in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454–5476, Online. Association for Computational Linguistics.
  8. Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. 2023. Assessing cross-cultural alignment between chatgpt and human societies: An empirical study. arXiv preprint arXiv:2303.17466.
  9. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. find
  10. Gerard Delanty. 2006. Modernity and the escape from eurocentrism. In Handbook of contemporary European social theory, pages 266–278. Routledge. find
  11. Enrique Dussel. 1993. Eurocentrism and modernity (introduction to the frankfurt lectures). boundary 2, 20(3):65–76. find
  12. Ruixiang Feng, Shen Gao, Xiuying Chen, Lisi Chen, and Shuo Shang. 2025. Culfit: A fine-grained cultural-aware llm training paradigm via multilingual critique data synthesis. arXiv preprint arXiv:2505.19484. find
  13. Joseph Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76:378–. find
  14. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2025. A survey on llm-as-a-judge. Preprint, arXiv:2411.15594. find
  15. Ilana Heintz, Ryan Gabbard, Mahesh Srivastava, Dave Barner, Donald Black, Majorie Friedman, and Ralph Weischedel. 2013. Automatic extraction of linguistic metaphors with LDA topic modeling. In Proceedings of the First Workshop on Metaphor in NLP, pages 58– 66, Atlanta, Georgia. Association for Compu… find
  16. William Held, Camille Harris, Michael Best, and Diyi Yang. 2023. A material lens on coloniality in nlp. arXiv preprint arXiv:2311.08391. find
  17. Ronald Inglehart and Wayne E. Baker. 2000. Modernization, cultural change, and the persistence of traditional values*. American Sociological Review, 65(1):19–51. find
  18. Hamed Jelodar, Yongli Wang, Chi Yuan, Xia Feng, Xiahui Jiang, Yanchao Li, and Liang Zhao. 2019. Latent dirichlet allocation (lda) and topic modeling: models, applications, a survey. Multimedia tools and applications, 78:15169–15211. find
  19. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  20. Rebecca L Johnson, Giada Pistilli, Natalia Menédez-González, Leslye Denisse Dias Duran, Enrico Panai, Julija Kalpokiene, and Donald Jay Bertulfo. 2022. The ghost in the machine has an american accent: value conflict in gpt-3. Preprint, arXiv:2203.07785. find
  21. Julia Kharchenko, Tanya Roosta, Aman Chadha, and Chirag Shah. 2024. How well do llms represent values across cultures? empirical analysis of llm responses based on hofstede cultural dimensions. Preprint, arXiv:2406.14805. — [arXiv:2406.14805](https://arxiv.org/abs/2406.14805) find
  22. Dayeon Ki, Rachel Rudinger, Tianyi Zhou, and Marine Carpuat. 2025. Multiple LLM agents debate for equitable cultural alignment. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 24841–24877, Vienna, Austria. Association for Comp…
  23. Haein Kong, Yongsu Ahn, Sangyub Lee, and Yunho Maeng. 2024. Gender bias in llm-generated interview responses. arXiv preprint arXiv:2410.20739. — [arXiv:2410.20739](https://arxiv.org/abs/2410.20739) find
  24. Cheng Li, Mengzhou Chen, Jindong Wang, Sunayana Sitaram, and Xing Xie. 2024a. Culturellm: Incorporating cultural differences into large language models. Preprint, arXiv:2402.10946.
  25. Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, and Huan Liu. 2025. From generation to judgment: Opportunities and challenges of llm-as-a-judge. Preprint, arXiv:2411.16594. find
  26. Huihan Li, Liwei Jiang, Jena D. Hwang, Hyunwoo Kim, Sebastin Santy, Taylor Sorensen, Bill Yuchen Lin, Nouha Dziri, Xiang Ren, and Yejin Choi. 2024b. Culture-gen: Revealing global cultural perception in language models through natural language prompting. Preprint, arXiv:2404.10199.
  27. Reem I. Masoud, Ziquan Liu, Martin Ferianc, Philip Treleaven, and Miguel Rodrigues. 2024. Cultural alignment in large language models: An explanatory analysis based on hofstede’s cultural dimensions. Preprint, arXiv:2309.12342.
  28. Meta. 2024. Llama 3.1 model card. find
  29. Burt Monroe, Michael Colaresi, and Kevin Quinn. 2009. Fightin’ words: Lexical feature selection and evaluation for identifying the content of political conflict. Political Analysis, 16. find
  30. Simon Münker. 2025. Cultural bias in large language models: Evaluating ai agents through moral questionnaires. arXiv preprint arXiv:2507.10073. — [arXiv:2507.10073](https://arxiv.org/abs/2507.10073) find
  31. Tarek Naous, Michael J. Ryan, Alan Ritter, and Wei Xu. 2024. Having beer after prayer? measuring cultural bias in large language models. Preprint, arXiv:2305.14456.
  32. OpenAI. 2024. Gpt-4o system card.
  33. OpenAI. 2025. Openai o4 mini systen card. find
  34. Bo Pang, Tingrui Qiao, Caroline Walker, Chris Cunningham, and Yun Sing Koh. 2025. Libra: Measuring bias of large language model from a local context. Preprint, arXiv:2502.01679. — [arXiv:2502.01679](https://arxiv.org/abs/2502.01679) find
  35. Philip M. Parker. 1997. National Cultures of the World: A Statistical Reference. Praeger, Westport, CT. find
  36. Siddhesh Pawar, Arnav Arora, Lucie-Aimée Kaffee, and Isabelle Augenstein. 2025. Presumed cultural identity: How names shape llm responses. arXiv preprint arXiv:2502.11995. find
  37. Richard Peet. 2005. From eurocentrism to americentrism. Antipode, 37(5). find
  38. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng… find
  39. Victor Roudmetof. 1994. Globalization or modernity? Comparative Civilizations Review, 31(31):3. find
  40. Jonathan Rystrøm, Hannah Rose Kirk, and Scott Hale. 2025. Multilingual != multicultural: Evaluating gaps between multilingual capabilities and cultural alignment in llms. Preprint, arXiv:2502.16534. find
  41. Shahnewaz Karim Sakib and Anindya Bijoy Das. 2024. Challenging fairness: A comprehensive exploration of bias in llm-based recommendations. In 2024 IEEE International Conference on Big Data (BigData), pages 1585–1592. find
  42. Hari Shankar, Vedanta S P, Tejas Cavale, Ponnurangam Kumaraguru, and Abhijnan Chakraborty. 2025. Sometimes the model doth preach: Quantifying religious bias in open llms through demographic analysis in asian nations. Preprint, arXiv:2503.07510. — [arXiv:2503.07510](https://arxiv.org/abs/2503.07510) find
  43. Lianghui Zhu, Xinggang Wang, and Xinlong Wang. 2023. Judgelm: Fine-tuned large language models are scalable judges. arXiv preprint arXiv:2310.17631.
  44. Shreya Shankar, J.D. Zamfirescu-Pereira, Bjoern Hartmann, Aditya Parameswaran, and Ian Arawjo. 2024. Who validates the validators? aligning llm-assisted evaluation of llm outputs with human preferences. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, UIST ’… find
  45. Ella Shohat and Robert Stam. 2014. Unthinking Eurocentrism: Multiculturalism and the media. Routledge. find
  46. Nicholas Sukiennik, Chen Gao, Fengli Xu, and Yong Li. 2025. An evaluation of cultural value alignment in llm. Preprint, arXiv:2504.08863. — [arXiv:2504.08863](https://arxiv.org/abs/2504.08863) find
  47. Yan Tao, Olga Viberg, Ryan S Baker, and René F Kizilcec. 2024. Cultural bias and cultural alignment of large language models. PNAS Nexus, 3(9):pgae346. find
  48. Manuel Tonneau, Diyi Liu, Samuel Fraiberger, Ralph Schroeder, Scott A. Hale, and Paul Röttger. 2024. From languages to geographies: Towards evaluating cultural bias in hate speech datasets. Preprint, arXiv:2404.17874. find
  49. Yixin Wan and Kai-Wei Chang. 2025. White men lead, black women help? benchmarking and mitigating language agency social biases in LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9082–9108, Vienna, Austria. Association fo… find
  50. Yixin Wan, George Pu, Jiao Sun, Aparna Garimella, Kai-Wei Chang, and Nanyun Peng. 2023. "kelly is a warm person, joseph is a role model": Gender biases in llm-generated reference letters. Preprint, arXiv:2310.09219.
  51. Hui Wei, Shenghua He, Tian Xia, Fei Liu, Andy Wong, Jingyang Lin, and Mei Han. 2025. Systematic evaluation of llm-as-a-judge in llm alignment tasks: Explainable metrics and diverse prompt templates. Preprint, arXiv:2408.13006. — [arXiv:2408.13006](https://arxiv.org/abs/2408.13006) find
  52. Weiqi Wu, Hongqiu Wu, Lai Jiang, Xingyuan Liu, Hai Zhao, and Min Zhang. 2024. From role-play to drama-interaction: An llm solution. In Findings of the Association for Computational Linguistics ACL 2024, pages 3271–3290. find
  53. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wan-2026-insideout-measuring-mitigating-insider-outsider/