Elamur Research⌕
☰
home / papers / 2025 / Aligned but Blind: Alignment Increases Implicit Bias by Reducing Awareness of Race

Aligned but Blind: Alignment Increases Implicit Bias by Reducing Awareness of Race

Paper · 2025 · Lihao Sun, Chengzhi Mao, Valentin Hofmann et al.

Summary

Value-aligned language models often look fair when asked directly about race, yet still show stereotypes in indirect word-association tests. This paper examines why that gap exists. The authors built 9,232 matched prompts adapted from psychological stereotype measures. The prompts differ only in how implicit they are, linking black/white to negative traits, guilt, or weapons. They tested Llama 3 base and aligned models on these prompts. Alignment almost eliminated explicit bias but raised implicit bias.

Interpretability tools explain the pattern. Using SelfIE and activation patching on Llama 3 8B, the authors found that aligned models tend to read the ambiguous words black and white as colors rather than as race. Because race is not represented, safety behavior is likely never triggered.

As a fix, they pushed racial meaning into early layers using activation steering and LoRA fine-tuning, which cut implicit bias substantially. The work suggests that making models more aware of race, rather than less, may be a better route to debiasing.

Key points

  • Prompt suite: 9,232 paired implicit/explicit prompts with four variants each to reduce artifacts. Control conditions use race-indicative names (e.g., DeShawn, Jake) and color prefixes. Code and data are at github.com/slhleosun/aligned-but-blind.
  • Explicit vs. implicit bias (Llama 3 70B): Explicit bias fell from 49.6% (base) to 8.13% (aligned). Implicit bias rose from 64.1% to 91.4%.
  • Effect of context: With racial names, the aligned model's implicit bias dropped to 38.5%. With color prefixes it stayed near the default level, at 93.6%.
  • Mechanism: Activation patching showed base models treat black/white as race and as color about equally often. Aligned models are 52.2% less likely to encode race in ambiguous contexts.
  • Mitigation: Injecting race-related activations reduced bias by 54.9% against baseline. Targeting early layers worked better than targeting later or all layers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Social Sciences · 1,772 worksSociology and Political Science · 577 works

Cite

Cite as Lihao Sun, Chengzhi Mao, Valentin Hofmann and Xuechunzi Bai. 2025. Aligned but Blind: Alignment Increases Implicit Bias by Reducing Awareness of Race. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.1078

@inproceedings{sun2025aligned,
  title = {{Aligned but Blind: Alignment Increases Implicit Bias by Reducing Awareness of Race}},
  author = {Lihao Sun, Chengzhi Mao, Valentin Hofmann and Xuechunzi Bai},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1078},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1078}
}

References

76 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Zeyuan Allen-Zhu and Yuanzhi Li. 2024. Physics of language models: Part 3.1, knowledge storage and extraction. Preprint, arXiv:2309.14316.
  2. Evan P Apfelbaum, Michael I Norton, and Samuel R Sommers. 2012. Racial color blindness: Emergence, practice, and implications. Current directions in psychological science, 21(3):205–209. find
  3. Xiaoyang Bai, Amanda Wang, Ilia Sucholutsky, and Thomas L. Griffiths. 2025. Explicitly unbiased large language models still form biased associations. Proceedings of the National Academy of Sciences, 122(8):e2416228122. find
  4. Xuechunzi Bai, Angelina Wang, Ilia Sucholutsky, and Thomas L. Griffiths. 2024. Measuring implicit bias in explicitly unbiased large language models. Preprint, arXiv:2402.04105. — [arXiv:2402.04105](https://arxiv.org/abs/2402.04105) find
  5. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared… — Constitutional AI: Harmlessness from AI Feedback
  6. Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. 2023. Eliciting latent predictions from transformers with the tuned lens. Preprint, arXiv:2303.08112.
  7. Leonard Bereska and Efstratios Gavves. 2024. Mechanistic interpretability for ai safety – a review. Preprint, arXiv:2404.14082. find
  8. Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. 2023. Language models can explain neurons in language models. OpenAI. find
  9. Eduardo Bonilla-Silva. 2021. Racism without racists: Color-blind racism and the persistence of racial inequality in America. Rowman & Littlefield. find
  10. Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. find
  11. Haozhe Chen, Carl Vondrick, and Chengzhi Mao. 2024. Selfie: Self-interpretation of large language model embeddings. Preprint, arXiv:2403.10949. — [arXiv:2403.10949](https://arxiv.org/abs/2403.10949) find
  12. Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21. AC… find
  13. Omkar Dige, Diljot Singh, Tsz Fung Yau, Qixuan Zhang, Borna Bolandraftar, Xiaodan Zhu, and Faiza Khan Khattak. 2024. Mitigating social biases in language models through unlearning. Preprint, arXiv:2406.13551. — [arXiv:2406.13551](https://arxiv.org/abs/2406.13551) find
  14. Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of interpretable machine learning. Preprint, arXiv:1702.08608.
  15. J. L. Eberhardt, P. A. Goff, V. J. Purdie, and P. G. Davies. 2004. Seeing black: Race, crime, and visual processing. Journal of Personality and Social Psychology, 87(6):876–893. find
  16. Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. 2021. Causal abstractions of neural networks. Preprint, arXiv:2106.02997. find
  17. Aryo Pradipta Gema, Pasquale Minervini, Luke Daines, Tom Hope, and Beatrice Alex. 2024. Parameter-efficient fine-tuning of llama for the clinical domain. Preprint, arXiv:2307.03042. find
  18. Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. 2023. Dissecting recall of factual associations in auto-regressive language models. Preprint, arXiv:2304.14767. find
  19. Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. 2024. Patchscopes: A unifying framework for inspecting hidden representations of language models. Preprint, arXiv:2401.06102. find
  20. A. G. Greenwald and M. R. Banaji. 2017. The implicit revolution: Reconceiving the relation between conscious and unconscious. American Psychologist, 72(9):861–871. find
  21. A. G. Greenwald, D. E. McGhee, and J. L. K. Schwartz. 1998. Measuring individual differences in implicit cognition: The implicit association test. Journal of Personality and Social Psychology, 74(6):1464–1480. find
  22. Jia-Chen Gu, Hao-Xiang Xu, Jun-Yu Ma, Pan Lu, Zhen-Hua Ling, Kai-Wei Chang, and Nanyun Peng. 2024. Model editing harms general abilities of large language models: Regularization to the rescue. Preprint, arXiv:2401.04700.
  23. Wes Gurnee and Max Tegmark. 2024. Language models represent space and time. Preprint, arXiv:2310.02207. find
  24. Michael Hanna, Ollie Liu, and Alexandre Variengien. 2023. How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. Preprint, arXiv:2305.00586.
  25. Stefan Heimersheim and Neel Nanda. 2024. How to use and interpret activation patching. Preprint, arXiv:2404.15255. — [arXiv:2404.15255](https://arxiv.org/abs/2404.15255) find
  26. V. Hofmann, P. R. Kalluri, D. Jurafsky, et al. 2024. Ai generates covertly racist decisions about people based on their dialect. Nature, 633:147–154. find
  27. Ari Holtzman, Peter West, and Luke Zettlemoyer. 2023. Generative models as a complex systems science: How can we make sense of large language model behavior? Preprint, arXiv:2308.00189. find
  28. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685.
  29. Jennifer Hu and Michael C. Frank. 2024. Auxiliary task demands mask the capabilities of smaller language models. Preprint, arXiv:2404.02418.
  30. Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. 2018. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). Preprint, arXiv:1711.11279. — [arXiv:1711.11279](https://arxiv.org/abs/1711.11279) find
  31. Divyanshu Kumar, Umang Jain, Sahil Agarwal, and Prashanth Harshangi. 2024. Investigating implicit bias in large language models: A large-scale study of over 50 llms. Preprint, arXiv:2410.12864. — [arXiv:2410.12864](https://arxiv.org/abs/2410.12864) find
  32. Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K. Kummerfeld, and Rada Mihalcea. 2024. A mechanistic understanding of alignment algorithms: A case study on dpo and toxicity. Preprint, arXiv:2401.01967. find
  33. Justin D Levinson, Huajian Cai, and Danielle Young. 2010. Guilty by implicit racial bias: The guilty/not guilty implicit association test. Ohio St. J. Crim. L., 8:187. find
  34. Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2024a. Inference-time intervention: Eliciting truthful answers from a language model. Preprint, arXiv:2306.03341.
  35. Tianlong Li, Zhenghua Wang, Wenhao Liu, Muling Wu, Shihan Dou, Changze Lv, Xiaohua Wang, Xiaoqing Zheng, and Xuanjing Huang. 2024b. Revisiting jailbreaking for large language models: A representation engineering perspective. Preprint, arXiv:2401.06824.
  36. Zongxi Li, Xianming Li, Yuzhang Liu, Haoran Xie, Jing Li, Fu lee Wang, Qing Li, and Xiaoqin Zhong. 2023. Label supervised llama finetuning. Preprint, arXiv:2310.01208. find
  37. Tom Lieberum, Matthew Rahtz, János Kramár, Neel Nanda, Geoffrey Irving, Rohin Shah, and Vladimir Maikulik. 2023. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla. Preprint, arXiv:2307.09458.
  38. Fangru Lin, Shaoguang Mao, Emanuele La Malfa, Valentin Hofmann, Adrian de Wynter, Xun Wang, Si-Qing Chen, Michael Wooldridge, Janet B. Pierrehumbert, and Furu Wei. 2025. One language, many gaps: Evaluating dialect fairness and robustness of large language models in reasoning tasks. Preprint, arXiv:2… — [arXiv:2410.11005](https://arxiv.org/abs/2410.11005) find
  39. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. Preprint, arXiv:2107.13586.
  40. AI @ Meta Llama Team. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  41. Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. 2024. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. Preprint, arXiv:2403.19647. find
  42. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2023. Locating and editing factual associations in gpt. Preprint, arXiv:2202.05262. find
  43. Kanishka Misra and Kyle Mahowald. 2024. Language models learn rare phenomena from less rare phenomena: The case of the missing aanns. Preprint, arXiv:2403.19827.
  44. Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. 2023. Progress measures for grokking via mechanistic interpretability. Preprint, arXiv:2301.05217.
  45. Leland Gerson Neuberg. 2003. Causality: Models, reasoning, and inference. Econometric Theory, 19(4):675– 685. find
  46. Michael I Norton, Samuel R Sommers, Evan P Apfelbaum, Natassia Pura, and Dan Ariely. 2006. Color blindness and interracial interaction: Playing the political correctness game. Psychological Science, 17(11):949–953. find
  47. B. A. Nosek, A. G. Greenwald, and M. R. Banaji. 2007. The implicit association test at age 7: A methodological and conceptual review. In J. A. Bargh, editor, Social psychology and the unconscious: The automaticity of higher mental processes, pages 265–292. Psychology Press. find
  48. Chris Olah. 2022. Mechanistic interpretability, variables, and the importance of interpretable bases. find
  49. Nina Panickassery. 2023. Reducing sycophancy and improving honesty via activation steering. LessWrong. find
  50. Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. 2024. Steering llama 2 via contrastive activation addition. Preprint, arXiv:2312.06681. find
  51. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R. Bowman. 2022. Bbq: A hand-built bias benchmark for question answering. Preprint, arXiv:2110.08193.
  52. Mica Pollock. 2004. Race wrestling: Struggling strategically with race in educational practice and research. American journal of education, 111(1):25–67. find
  53. Nirmalendu Prakash and Lee Ka Wei Roy. 2024. Interpreting bias in large language models: A feature-based approach. Preprint, arXiv:2406.12347. — [arXiv:2406.12347](https://arxiv.org/abs/2406.12347) find
  54. Paul Röttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Kirk, Hinrich Schuetze, and Dirk Hovy. 2024. Political compass or spinning arrow? towards more meaningful evaluations for values and opinions in large language models. In Proceedings of the 62nd Annual Meeting of the Associatio…
  55. Paul Röttger, Musashi Hinck, Valentin Hofmann, Kobi Hackenburg, Valentina Pyatkin, Faeze Brahman, and Dirk Hovy. 2025. Issuebench: Millions of realistic prompts for measuring issue bias in llm writing assistance. Preprint, arXiv:2502.08395.
  56. Michael Santacroce, Yadong Lu, Han Yu, Yuanzhi Li, and Yelong Shen. 2023. Efficient rlhf: Reducing the memory usage of ppo. Preprint, arXiv:2309.00754. find
  57. Hakim Sidahmed, Samrat Phatale, Alex Hutcheson, Zhuonan Lin, Zhang Chen, Zac Yu, Jarvis Jin, Sim-ral Chaudhary, Roman Komarytsia, Christiane Ahlheim, Yonghao Zhu, Bowen Li, Saravanan Ganesh, Bill Byrne, Jessica Hoffmann, Hassan Mansoor, Wei Li, Abhinav Rastogi, and Lucas Dixon. 2024. Parameter effic… — [arXiv:2403.10704](https://arxiv.org/abs/2403.10704) find
  58. Flannery G Stevens, Victoria C Plaut, and Jeffrey Sanchez-Burks. 2008. Unlocking the benefits of diversity: All-inclusive multiculturalism and positive organizational change. The journal of applied behavioral science, 44(1):116–133. find
  59. Alessandro Stolfo, Vidhisha Balachandran, Safoora Yousefi, Eric Horvitz, and Besmira Nushi. 2024. Improving instruction-following in language models through activation steering. Preprint, arXiv:2410.12877. — [arXiv:2410.12877](https://arxiv.org/abs/2410.12877) find
  60. Simeng Sun, Dhawal Gupta, and Mohit Iyyer. 2023. Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of rlhf. Preprint, arXiv:2309.09055.
  61. Kunal Suri, Prakhar Mishra, Saumajit Saha, and Atul Singh. 2023. Suryakiran at mediqa-sum 2023: Leveraging lora for clinical dialogue summarization. Preprint, arXiv:2307.05162. — [arXiv:2307.05162](https://arxiv.org/abs/2307.05162) find
  62. Alex Tamkin, Amanda Askell, Liane Lovitt, Esin Durmus, Nicholas Joseph, Shauna Kravec, Karina Nguyen, Jared Kaplan, and Deep Ganguli. 2023. Evaluating and mitigating discrimination in language model decisions. Preprint, arXiv:2312.03689. find
  63. Augustin Toma, Patrick R. Lawler, Jimmy Ba, Rahul G. Krishnan, Barry B. Rubin, and Bo Wang. 2023. Clinical camel: An open expert-level medical language model with dialogue-based knowledge encoding. Preprint, arXiv:2305.12031. find
  64. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. 2024. Steering language models with activation engineering. Preprint, arXiv:2308.10248.
  65. Angelina Wang, Xuechunzi Bai, Solon Barocas, and Su Lin Blodgett. 2023. Measuring stereotype harm from machine learning errors requires understanding who is being harmed by which errors in what ways. In ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO). find
  66. Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang T. Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, and Bo Li. 2024. Decodingtrust: A comprehensive assessment o…
  67. Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2022. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. Preprint, arXiv:2211.00593. find
  68. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How does llm safety training fail? Preprint, arXiv:2307.02483.
  69. Yotam Wolf, Noam Wies, Oshri Avnery, Yoav Levine, and Amnon Shashua. 2023. Fundamental limitations of alignment in large language models. arXiv preprint arXiv:2304.11082.
  70. Zhengxuan Wu, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky, Christopher D. Manning, and Christopher Potts. 2025. Axbench: Steering llms? even simple baselines outperform sparse autoencoders. Preprint, arXiv:2501.17148. find
  71. Zhihao Xu, Ruixuan Huang, Changyu Chen, and Xiting Wang. 2024. Uncovering safety risks of large language models through concept activation vector. Preprint, arXiv:2404.12038. — [arXiv:2404.12038](https://arxiv.org/abs/2404.12038) find
  72. Lilong Xue, Dan Zhang, Yuxiao Dong, and Jie Tang. 2024. Autore: Document-level relation extraction with large language models. Preprint, arXiv:2403.14888.
  73. Zeping Yu and Sophia Ananiadou. 2025. Understanding and mitigating gender bias in llms via interpretable neuron editing. Preprint, arXiv:2501.14457. — [arXiv:2501.14457](https://arxiv.org/abs/2501.14457) find
  74. Fred Zhang and Neel Nanda. 2024. Towards best practices of activation patching in language models: Metrics and methods. Preprint, arXiv:2309.16042. find
  75. Yazhou Zhang, Mengyao Wang, Youxi Wu, Prayag Tiwari, Qiuchi Li, Benyou Wang, and Jing Qin. 2024. Dialoguellm: Context and emotion knowledge-tuned large language models for emotion recognition in conversations. Preprint, arXiv:2310.11374. find
  76. Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. 2023. The clock and the pizza: Two stories in mechanistic explanation of neural networks. Preprint, arXiv:2306.17844. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2025-aligned-but-blind-alignment-increases/