Elamur Research⌕
☰
home / papers / 2026 / Measuring Mechanistic Independence: Can Bias Be Removed Without Erasing Demographics?

Measuring Mechanistic Independence: Can Bias Be Removed Without Erasing Demographics?

Paper · 2026 · Zhengyang Shan and Aaron Mueller

Summary

The paper asks whether stereotype-driven behavior in a language model can be removed while the model still correctly recognizes demographic information when that information is legitimately relevant. The authors build an automated pipeline around Gemma-2-9B with Gemma-Scope sparse autoencoders (SAEs). It covers five word-pairing tasks. Name–race and name–gender pairings act as a control, because there the association is real. Profession–race and profession–gender pairings measure stereotypes. Profession–education pairings fall in between. Prompts come in two directions: Demo-R places the label after the item, and Demo-L places it before. The authors select SAE features with integrated-gradient attribution, Pearson correlation, the intersection of the two, or their difference, and then ablate the selected features at inference time. Attribution-based ablation lowered race and gender profession stereotypes without hurting name recognition. For education, correlation-based features worked better, while attribution ablation caused the model to collapse onto a single default answer. The authors conclude that bias runs through task-specific circuits rather than generic demographic markers. Precise, dimension-aware interventions can therefore debias a model without retraining it or erasing demographic knowledge.

Key points

  • Setup: The model is Gemma-2-9B. Residual-stream activations were traced with NNsight and decomposed with pretrained Gemma-Scope SAEs. The authors generated 523 prompts, each containing eight items, across five tasks.
  • Feature selection: Four strategies were compared:
    • attribution, an indirect-effect estimate obtained via integrated gradients adapted for SAEs;
    • correlation with demographic labels;
    • only those features picked by both the attribution and the correlation methods;
    • the set difference between them.
  • Prior collapse: Ablating attribution-selected features on education tasks made the model predict a Bachelor's degree for most professions, which raised overall bias. Correlation-selected features avoided this failure.
  • Gender vs. race: Gender bias appears to rely on broad markers that affect professions uniformly, so it was easier to isolate. Race bias is tangled with recognition. Its ablation effects differed by profession, with clear gains for technical roles but sometimes worse bias for manual-labor jobs. Bias was measured as KL divergence from a uniform distribution.
  • Limitations:
    • Interventions do not transfer across dimensions: an education-targeted ablation that reduced gender bias increased race bias.
    • The study is restricted to English data, the Gemma-2 family, and three demographic axes, using simplified categories.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Arts and Humanities · 16,039 worksSocial Sciences · 1,772 worksSafety Research · 543 worksMathematics · 177 worksStatistics and Probability · 16 worksHistory and Philosophy of Science · 53 works

🧭 Topics 2Research subjects and languages the work is about.Bias, fairness & safety · 517 worksInterpretability · 130 works

Cite

Cite as Zhengyang Shan and Aaron Mueller. 2026. Measuring Mechanistic Independence: Can Bias Be Removed Without Erasing Demographics?. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.199

@inproceedings{shan2026measuring,
  title = {{Measuring Mechanistic Independence: Can Bias Be Removed Without Erasing Demographics?}},
  author = {Zhengyang Shan and Aaron Mueller},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.199},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.199}
}

References

37 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Haozhe An, Christabel Acquaye, Colin Wang, Zongxia Li, and Rachel Rudinger. 2024. Do large language models discriminate in hiring decisions on the basis of race, ethnicity, and gender? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)… find
  2. Dana Arad, Aaron Mueller, and Yonatan Belinkov. 2025. Saes are good for steering – if you select the right features. Preprint, arXiv:2505.20063.
  3. Xuechunzi Bai, Angelina Wang, Ilia Sucholutsky, and Thomas L. Griffiths. 2025. Explicitly unbiased large language models still form biased associations. Proceedings of the National Academy of Sciences, 122(8):e2416228122. find
  4. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of “bias” in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454– 5476, Online. Association for Computational Linguistics.
  5. Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, and 6 others. 2023. To… find
  6. Bhavik Chandna, Zubair Bashir, and Procheta Sen. 2025. Dissecting bias in llms: A mechanistic interpretability perspective. ArXiv, abs/2506.05166. — [arXiv:2506.05166](https://arxiv.org/abs/2506.05166) find
  7. Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. 2023. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600. find
  8. Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, pa… find
  9. Jaden Fiotto-Kaufman, Alexander R. Loftus, Eric Todd, Jannik Brinkmann, Koyena Pal, Dmitrii Troitskii, Michael Ripa, Adam Belfki, Can Rager, Caden Juang, Aaron Mueller, Samuel Marks, Arnab Sen Sharma, Francesca Lucchetti, Nikhil Prakash, Carla Brodley, Arjun Guha, Jonathan Bell, Byron C. Wallace, an… find
  10. Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. 2022. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 30–45, Abu Dhabi, United Arab Emirates. Asso…
  11. Hila Gonen and Yoav Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol…
  12. Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. 2024. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms. In ICML 2024 Workshop on Mechanistic Interpretability. find
  13. Jiho Jin, Woosung Kang, Junho Myung, and Alice Oh. 2025. Social bias benchmark for generation: A comparison of generation and QA-based evaluations. In Findings of the Association for Computational Linguistics: ACL 2025, pages 11215–11228, Vienna, Austria. Association for Computational Linguistics. find
  14. Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023. Inference-time intervention: eliciting truthful answers from a language model. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Ass…
  15. Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, Janos Kramar, Anca Dragan, Rohin Shah, and Neel Nanda. 2024. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Inte… find
  16. Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. 2025. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. Preprint, arXiv:2403.19647. find
  17. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. Curran Associates Inc. find
  18. Aaron Mueller, Atticus Geiger, Sarah Wiegreffe, Dana Arad, Iván Arcuschin, Adam Belfki, Yik Siu Chan, Jaden Fiotto-Kaufman, Tal Haklay, Michael Hanna, Jing Huang, Rohan Gupta, Yaniv Nikankin, Hadas Orgad, Nikhil Prakash, Anja Reusch, Aruna Sankaranarayanan, Shun Shao, Alessandro Stolfo, and 4 others… — [arXiv:2504.13151](https://arxiv.org/abs/2504.13151) find
  19. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022. BBQ: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2086–2105, Dublin, Ireland. Asso…
  20. Judea Pearl. 2001. Direct and indirect effects. In Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, UAI’01, page 411–420, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. find
  21. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  22. Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. 2024. Steering llama 2 via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15504–15522, Bangkok, Thaila… find
  23. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 8–1…
  24. Timo Schick, Sahana Udupa, and Hinrich Schütze. 2021. Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in NLP. Transactions of the Association for Computational Linguistics, 9:1408– 1424.
  25. Zhengyang Shan, Emily Diana, and Jiawei Zhou. 2025. Gender inclusivity fairness index (GIFI): A multilevel framework for evaluating gender diversity in large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 254… find
  26. Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, Stella Biderman, Adria Garriga-Alonso, Arthur Conmy, Neel Nanda, Jessica Rumbelow, Martin Wattenberg, Nandi Schoots, Joseph Miller, Eric J.… — [arXiv:2501.16496](https://arxiv.org/abs/2501.16496) find
  27. Lihao Sun, Chengzhi Mao, Valentin Hofmann, and Xuechunzi Bai. 2025. Aligned but blind: Alignment increases implicit bias by reducing awareness of race. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 22167–22184, Vienna, Austr…
  28. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, page 3319–3328. JMLR.org. find
  29. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Race–Profession, Gender–Profession, and Education–Profession (A.3). Pouya Tafti, Abe Friesen, Michelle Casbo… find
  30. Section B: Fluency Preservation and Output Quality Validation evaluates whether feature ablations maintain model generation capabilities through dual metrics: perplexity for formatting quality and valid label rate for semantic correctness. find
  31. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. 2024. Steering language models with activation engineering. Preprint, arXiv:2308.10248.
  32. Section C: Demo-R Ablation Results presents complete ablation outcomes across all demographic dimensions, with systematic error analysis (C.1). We provide detailed characterization of the highest-attribution features driving demographic bias (C.2). Section C.3 analyzes the computational mechanisms u… find
  33. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. In Advances in Neural Information Processing Systems, volume 33, pages 12388–12401. Curran Associates, Inc. find
  34. Yang Zhang, Fuli Feng, Xiangnan He, Tianxin Wei, Chonggang Song, Guohui Ling, and Yongdong Zhang. 2021. Causal intervention for leveraging popularity bias in recommendation. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’21,… find
  35. Section D: Demo-L Ablation Results provides parallel analysis for left-hand-side prompt formats, revealing fundamentally inverted patterns from Demo-R. find
  36. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  37. Section E: Validation on Winogender evaluates whether the same feature-level interventions reduce stereotype reliance in a naturalistic pronoun-resolution task, including analysis of the stereotype-challenging gotcha subset. • Section F: Top-k Feature Selection Validation provides empirical justific… — Representation Engineering: A Top-Down Approach to AI Transparency

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shan-2026-measuring-mechanistic-independence-can-bias/