Elamur Research⌕
☰
home / papers / 2026 / Towards Reliable AI Fairness: Challenges in Steering Features within Bias-Implicated Neurons

Towards Reliable AI Fairness: Challenges in Steering Features within Bias-Implicated Neurons

Summary

The paper tests whether steering internal features is a dependable way to reduce gender bias in LLMs, focusing on profession stereotypes such as doctor as male and nurse as female. The authors use gemma-2-2b and the transcoder released with GemmaScope, accessed through the Neuronpedia API. They build attribution graphs for many male- and female-context prompts and average each feature's attribution per gender. Four alternative scoring strategies then rank features to form a "global bias circuit." At inference time, a corrective vector is added to these features, with no retraining. The method is tested in four setups of increasing realism: naturalistic profession sentences, stereotypical adjectives, templated doctor/nurse prompts, and real-world doctor/nurse sentences. Steering works only within narrow parameter ranges. Stronger pushes can backfire, small parameter changes cause abrupt failure, and realistic inputs yield chaotic or incoherent text. The choice of scoring strategy barely matters, so the brittleness seems built into the steering mechanism itself. This argues for more robust, adaptive control methods before such techniques are used for fairness in practice.

Key points

  • Model and tools: gemma-2-2b (English only), the GemmaScope transcoder and public Sparse Autoencoders, and the Neuronpedia API. The analysis depends on the model-specific transcoder, which is not yet available for models like gemma-9b.
  • Features are scored from mean attributions in male vs. female contexts (ā male, ā female). Four scoring strategies were compared, and they differed significantly only under extreme, unrealistic steering intensity.
  • The paper describes three failure modes. Excessive strength made bias more persistent (Adjective Bias). In the templated setting, a small parameter change flipped a well-tuned intervention from correction to mere suppression, which the authors call "non-monotonic collapse" (Focused Context). Realistic sentences produced incoherent outputs (Realistic Context).
  • Proposed future work includes other models, morphologically gendered languages such as Spanish, intervening directly on monosemantic SAE features, and steering whose strength adapts to context.
  • Code, generated data, and the sentence dataset are released on GitHub, along with a live viewer for the results.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2iexsnkqn3j6

@inproceedings{garridomunoz2026reliable,
  title = {{Towards Reliable AI Fairness: Challenges in Steering Features within Bias-Implicated Neurons}},
  author = {Garrido-Munoz, Ismael and Montejo-Raez, Arturo and Martínez-Santiago, Fernando},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/2iexsnkqn3j6},
  url = {https://doi.org/10.63317/2iexsnkqn3j6}
}

References

39 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adam Zewe Adam Zewe \| MIT News. Unpacking the bias of large language models. find
  2. Alfred V. Aho and Jeffrey D. Ullman. 1972. The Theory of Parsing, Translation and Compiling, volume 1. Prentice-Hall, Englewood Cliffs, NJ. find
  3. Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persi… find
  4. American Psychological Association. 1983. Publications Manual. American Psychological Association, Washington, DC. find
  5. Rie Kubota Ando and Tong Zhang. 2005. A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6:1817–1853. find
  6. Galen Andrew and Jianfeng Gao. 2007. Scalable training of L1-regularized log-linear models. In Proceedings of the 24th International Conference on Machine Learning, pages 33–40. find
  7. Jacy Reese Anthis, Kristian Lum, Michael Ekstrand, Avi Feller, Alexander D’Amour, and Chenhao Tan. 2024. The impossibility of fair llms. ArXiv, abs/2406.03198. find
  8. Reza Bayat, Ali Rahimi-Kalahroudi, Mohammad Pezeshki, Sarath Chandar, and Pascal Vincent. 2025. Steering large language model activations in sparse spaces. ArXiv, abs/2503.00177. — [arXiv:2503.00177](https://arxiv.org/abs/2503.00177) find
  9. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 610–623, New York, NY, USA. Associat… find
  10. A. Bhattacharyya. 1946. On a measure of divergence between two multinomial populations. Sankhyā: The Indian Journal of Statistics (1933-1960), 7(4):401–406. find
  11. Ahsan Bilal, David Ebert, and Beiyu Lin. 2025. Llms for explainable ai: A comprehensive survey. find
  12. Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. CoRR, abs/1607.06520. find our card
  13. Sviatoslav Chalnev, Matthew Siu, and Arthur Conmy. 2024. Improving steering vectors by targeting sparse autoencoder features. ArXiv, abs/2411.02193. — [arXiv:2411.02193](https://arxiv.org/abs/2411.02193) find
  14. Bhavik Chandna, Zubair Bashir, and Procheta Sen. 2025. Dissecting bias in llms: A mechanistic interpretability perspective. ArXiv, abs/2506.05166. — [arXiv:2506.05166](https://arxiv.org/abs/2506.05166) find
  15. Ashok K. Chandra, Dexter C. Kozen, and Larry J. Stockmeyer. 1981. Alternation. Journal of the Association for Computing Machinery, 28(1):114– 133. find
  16. Zhoujie Ding, Ken Ziyu Liu, Pura Peetathawatchai, Berivan Isik, and Sanmi Koyejo. 2024. On fairness of low-rank adaptation of large models. ArXiv, abs/2405.17512. — [arXiv:2405.17512](https://arxiv.org/abs/2405.17512) find
  17. Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. 2024. Bias and fairness in large language models: A survey. Computational Linguistics, 50(3):1097–1179. find our card
  18. Ismael Garrido-Muñoz, Arturo Montejo-Ráez, Fernando Martínez-Santiago, and L. Alfonso Ureña-López. 2021. A survey on bias in deep nlp. Applied Sciences, 11(7). find
  19. Hila Gonen and Yoav Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. ArXiv, abs/1903.03862. find our card
  20. Dan Gusfield. 1997. Algorithms on Strings, Trees and Sequences. Cambridge University Press, Cambridge, UK. find
  21. Amr Hegazy, Mostafa Elhoushi, and Amr Alanwar. 2025. Guiding giants: Lightweight controllers for weighted activation steering in llms. ArXiv, abs/2505.20309. find
  22. Aliyah R Hsu, Georgia Zhou, Yeshwanth Cherapanamjeri, Yaxuan Huang, Anobel Y Odisho, Peter R Carroll, and Bin Yu. 2024. Efficient automated circuit discovery in transformers using contextual decomposition. arXiv preprint arXiv:2407.00886. — [arXiv:2407.00886](https://arxiv.org/abs/2407.00886) find
  23. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations. find
  24. Adam Karvonen and Samuel Marks. 2025. Robustly improving llm fairness in realistic settings via interpretability. ArXiv, abs/2506.10922. — [arXiv:2506.10922](https://arxiv.org/abs/2506.10922) find
  25. Hadas Kotek, Rikker Dockum, and David Q. Sun. 2023. Gender bias and stereotypes in large language models. Proceedings of The ACM Collective Intelligence Conference. find
  26. Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. 2024. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. find
  27. Johnny Lin. 2023. Neuronpedia: Interactive reference and tooling for analyzing neural networks. Software available from neuronpedia.org. find
  28. Kyle O’Brien, David Majercak, Xavier Fernandes, Richard Edgar, Jingya Chen, Harsha Nori, Dean Carignan, Eric Horvitz, and Forough Poursabzi-Sangde. 2024. Steering language model refusal with sparse autoencoders. ArXiv, abs/2411.11296. — [arXiv:2411.11296](https://arxiv.org/abs/2411.11296) find
  29. Avash Palikhe, Zhenyu Yu, Zichong Wang, and Wenbin Zhang. 2025. Towards Transparent AI: A Survey on Explainable Large Language Models. arXiv e-prints, page arXiv:2506.21812. — [arXiv:2506.21812](https://arxiv.org/abs/2506.21812) find
  30. Mohammad Sadegh Rasooli and Joel R. Tetreault. 2015. Yara parser: A fast and accurate dependency parser. Computing Research Repository, arXiv:1503.06733. Version 2. — [arXiv:1503.06733](https://arxiv.org/abs/1503.06733) find
  31. Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. 2023. Steering llama 2 via contrastive activation addition. ArXiv, abs/2312.06681. find
  32. Marco Scialanga, Thibault Laugel, Vincent Grari, and Marcin Detyniecki. 2025. SAKE: Steering Activations for Knowledge Editing. arXiv e-prints, page arXiv:2503.01751. find
  33. Emily Sheng, Kai-Wei Chang, P. Natarajan, and Nanyun Peng. 2021. Societal biases in language generation: Progress and challenges. ArXiv, abs/2105.04054. find our card
  34. Manan Suri, Nishit Anand, and Amisha Bhaskar. 2025. Mitigating memorization in llms using activation steering. ArXiv, abs/2503.06040. — [arXiv:2503.06040](https://arxiv.org/abs/2503.06040) find
  35. Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, Alex Tamkin, Esin Durmus, Tristan Hume, Francesco Mosconi, C Daniel Freeman, Theod… find
  36. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David S. Udell, Juan J. Vazquez, Ulisse Mini, and Monte Stuart MacDiarmid. 2023. Steering language models with activation engineering. find our card
  37. Jan Wehner, Sahar Abdelnabi, Daniel Tan, David Krueger, and Mario Fritz. 2025. Taxonomy, opportunities, and challenges of representation engineering for large language models. ArXiv, abs/2502.19649. — [arXiv:2502.19649](https://arxiv.org/abs/2502.19649) find
  38. Xin Xu, Wei Xu, Ningyu Zhang, and Julian McAuley. 2025. Biasedit: Debiasing stereotyped language models via model editing. ArXiv, abs/2503.08588. find
  39. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Troy Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, Zico Kolter, and Dan Hend… find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/munoz-2026-reliable-ai-fairness-challenges-steering/