Elamur Research⌕
☰
home / papers / 2023 / Identifying and Adapting Transformer-Components Responsible for Gender Bias in an English Language Model

Identifying and Adapting Transformer-Components Responsible for Gender Bias in an English Language Model

Paper · 2023 · Abhijith Chintam, Rahel Beloch, Willem Zuidema et al.

Summary

The paper asks whether gender bias in a language model can be traced to specific internal parts and then reduced by retraining only those parts. The study uses GPT-2 small and limits the search to its 144 attention heads. Three causal localization techniques are compared: causal mediation analysis (CMA), automated circuit discovery (ACDC), and a new, cheaper method called DiffMask+, which combines CMA with differentiable masking. The heads each method selects are then fine-tuned while the rest of the model stays frozen. Bias is measured on several benchmarks, including the revised CrowS-Pairs and WinoBias. General ability is checked with WikiText-103 perplexity and BLiMP.

The three methods largely agree, even though their compute costs differ enormously. Most of the relevant heads sit in the final four layers. Fine-tuning these few heads lowers bias on two of three bias measures and costs far less language-modelling quality than full fine-tuning. The work matters because it links interpretability findings to cheap, targeted model repair. It also shows how much the results depend on the choice of dataset.

Key points

  • The model studied is GPT-2 small, with 12 layers × 12 heads (144 attention heads). The search targets heads that carry gender information into the final position, where the next token is predicted.
  • Three localization methods are compared: CMA, ACDC circuits, and the new DiffMask+. ACDC is by far the most expensive. DiffMask+ is presented as an efficient way to find a minimal set of heads.
  • After full-model fine-tuning, WikiText-103 perplexity rises from 23.69 to 34.16. Fine-tuning only the selected heads raises it much less.
  • On BLiMP, every fine-tuned variant loses some overall accuracy. However, subject-verb agreement (SVA) improves slightly with the top-10 heads, and anaphor gender agreement (AGA) stays near the baseline.
  • The bias results are mixed. Models that improve most on CrowS-Pairs do not always improve on WinoBias. The authors suggest that coreference-related bias may live in other components, and that implicit and explicit bias may be encoded differently.

Interesting ideas

  • I1. Run several localization methods of very different cost and trust the components they agree on; agreement among independent probes is stronger evidence than any single probe.
  • I2. Retrain only the components shown to cause an unwanted behaviour, since a narrow update leaves the rest of the model largely intact.
  • I3. Measure side effects with both a broad score (perplexity) and narrow grammatical probes (BLiMP agreement subtasks), so collateral damage to specific skills does not hide in an average.
  • I4. The components found, and how well a fix generalizes, depend heavily on the dataset used; narrow template datasets may yield fixes that only partly transfer.
  • I5. Bias in coreference may be handled by different components than bias in plain next-token prediction, so one measure cannot certify another.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Treat cheap and expensive diagnostic tools as separate witnesses, and act on a suspected error source only when several of them point to it.
  • I2 (tree): When gparse makes a recurring error, first locate the responsible heads or layers, then fine-tune just those and leave the rest of the parser frozen.
  • I3 (learn): After any repair or retraining, rerun fixed negative-control suites per phenomenon, such as agreement and attachment types, alongside the global LAS.
  • I4 (learn): Draw diagnosis and repair data from structurally varied sources rather than one template family, so repair rules are not fitted to a single pattern.
  • I5 (frames): Test pronoun resolution separately with minimal pairs that swap gendered roles, checking that coreference follows syntax and discourse rather than occupational stereotypes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AmsterdamAmsterdam · Netherlands144 works · 173,906 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEnglish · 2,263 worksBias, fairness & safety · 517 works

Cite

Cite as Abhijith Chintam, Rahel Beloch, Willem Zuidema, Michael Hanna and Oskar van der Wal. 2023. Identifying and Adapting Transformer-Components Responsible for Gender Bias in an English Language Model. Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.blackboxnlp-1.29

@inproceedings{chintam2023identifying,
  title = {{Identifying and Adapting Transformer-Components Responsible for Gender Bias in an English Language Model}},
  author = {Abhijith Chintam, Rahel Beloch, Willem Zuidema, Michael Hanna and Oskar {van der Wal}},
  booktitle = {Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP},
  year = {2023},
  doi = {10.18653/v1/2023.blackboxnlp-1.29},
  url = {https://doi.org/10.18653/v1/2023.blackboxnlp-1.29}
}

References

59 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Maria Barrett, Yova Kementchedjhieva, Yanai Elazar, Desmond Elliott, and Anders Søgaard. 2019. Adversarial removal of demographic attributes revisited. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6330– 6335, Hong Kong, China. Association for Computational Linguistics. find
  2. Christine Basta, Marta R. Costa-jussà, and Noe Casas. 2019. Evaluating the underlying gender bias in contextualized word embeddings. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, page 33–39, Florence, Italy. Association for Computational Linguistics.
  3. Jasmijn Bastings, Wilker Aziz, and Ivan Titov. 2019. Interpretable neural predictions with differentiable binary variables. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2963–2977, Florence, Italy. Association for Computational Linguistics.
  4. Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James R. Glass. 2018. Identifying and controlling important neurons in neural machine translation. ArXiv, abs/1811.01157. ↗
  5. Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. 2023. Leace: Perfect linear concept erasure in closed form. arXiv preprint arXiv:2306.03819. ↗
  6. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623. ↗
  7. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397–2430. PMLR. find
  8. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of “bias” in nlp. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454–5476.
  9. Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021. Stereotyping Norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1004–1015, Online. Association for Computational Linguistics. ↗
  10. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems, 29. ↗
  11. Shikha Bordia and Samuel Bowman. 2019. Identifying and reducing gender bias in word-level language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 7–15.
  12. Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. ArXiv:1608.07187 [cs]. ↗
  13. Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. 2023. Towards automated circuit discovery for mechanistic interpretability. (arXiv:2304.14997). ArXiv:2304.14997 [cs]. ↗
  14. Nicola De Cao, Michael Schlichtkrull, Wilker Aziz, and Ivan Titov. 2021a. How do decisions emerge across layers in neural models? interpretation with differentiable masking. (arXiv:2004.14992). ArXiv:2004.14992 [cs, stat]. ↗
  15. Nicola De Cao, Leon Schmid, Dieuwke Hupkes, and Ivan Titov. 2021b. Sparse interventions in language models with differentiable masking. (arXiv:2112.06837). ArXiv:2112.06837 [cs]. ↗
  16. Pieter Delobelle, Ewoenam Tokpo, Toon Calders, and Bettina Berendt. 2022. Measuring fairness with biased rulers: A comparative study on bias metrics for pre-trained language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, page 1693–1706, Seattle, United States. Association for Computational Linguistics. find
  17. Sunipa Dev, Emily Sheng, Jieyu Zhao, Aubrie Amstutz, Jiao Sun, Yu Hou, Mattie Sanseverino, Jiin Kim, Akihiro Nishi, Nanyun Peng, et al. 2022. On measures of biases and harms in nlp. In Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022, pages 246–267. ↗
  18. Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. 2021. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9:160– 175. ↗
  19. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. 2021. A mathematical framework for transformer circuits. Transformer Circuits Thread. Https://transformer-circuits.pub/2021/framework/index.html. find
  20. Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. 2021. Causal abstractions of neural networks. (arXiv:2106.02997). ArXiv:2106.02997 [cs]. ↗
  21. Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. 2023. Dissecting recall of factual associations in auto-regressive language models. find
  22. Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. 2022. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 30–45, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  23. Yair Lakretz, German Kruszewski, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, and Marco Baroni. 2019. The emergence of number and syntax units in LSTM language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 11–20, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  24. Anne Lauscher, Tobias Lueken, and Goran Glavaš. 2021. Sustainable modular debiasing of language models. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4782–4797, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  25. Shahar Levy, Koren Lazar, and Gabriel Stanovsky. 2021. Collecting a large-scale gender bias dataset for coreference resolution and machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2470–2480, Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  26. Michael Gira, Ruisu Zhang, and Kangwook Lee. 2022. Debiasing pre-trained language models via efficient fine-tuning. In Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion, pages 59–69, Dublin, Ireland. Association for Computational Linguistics. find
  27. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), abs/2101.00190. ↗
  28. Michael Hanna, Ollie Liu, and Alexandre Variengien. 2023. How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model.
  29. Tom Lieberum, Matthew Rahtz, János Kramár, Neel Nanda, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. 2023. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla. ↗
  30. Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. 2023. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. find
  31. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning. ↗
  32. Dirk Hovy and Shannon L. Spruit. 2016. The social impact of natural language processing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 591–598, Berlin, Germany. Association for Computational Linguistics. ↗
  33. Diederik P. Kingma and Jimmy Ba. 2014. A method for stochastic optimization. abs/1412.6980. Adam: CoRR, Neeraja Kirtane and Tanvi Anand. 2022. Mitigating gender stereotypes in Hindi and Marathi. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 145–150, Seattle, Washington. Association for Computational Linguistics. ↗
  34. Gili Lior and Gabriel Stanovsky. 2023. Comparing humans and models on a similar scale: Towards cognitive gender bias evaluation in coreference resolution. ↗
  35. Ilya Loshchilov and Frank Hutter. 2017. weight decay regularization in adam. abs/1711.05101. ↗
  36. Fixing ArXiv, Christos Louizos, Max Welling, and Diederik P. Kingma. 2018. Learning sparse neural networks through L 0 regularization. In International Conference on Learning Representations. ↗
  37. Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. 2020. Gender bias in neural natural language processing. Logic, Language, and Security: Essays Dedicated to Andre Scedrov on the Occasion of His 65th Birthday, pages 189–202.
  38. Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), page 622–628, Minneapolis, Minnesota. Association for Computational Linguistics.
  39. Nicholas Meade, Elinor Poole-Dayan, and Siva Reddy. 2022. An empirical survey of the effectiveness of debiasing techniques for pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1878–1898, Dublin, Ireland. Association for Computational Linguistics. ↗
  40. Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null it out: Guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7237–7256, Online. Association for Computational Linguistics.
  41. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2023. Locating and editing factual associations in gpt. (arXiv:2202.05262). ArXiv:2202.05262 [cs]. ↗
  42. Abhilasha Ravichander, Yonatan Belinkov, and Eduard Hovy. 2021. Probing the probing paradigm: Does probing accuracy entail task relevance? In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3363–3377, Online. Association for Computational Linguistics.
  43. Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2022. Mass-editing memory in a transformer. arXiv preprint arXiv:2210.07229. ↗
  44. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. ArXiv, abs/1609.07843. ↗
  45. Hosein Mohebbi, Willem Zuidema, Grzegorz Chrupała, and Afra Alishahi. 2023. Quantifying context mixing in transformers. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3378–3400, Dubrovnik, Croatia. Association for Computational Linguistics. find
  46. Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. Stereoset: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), page 5356–5371, Online. Association for Computational Linguistics.
  47. Neel Nanda and Joseph Bloom. 2022. Transformerlens. find
  48. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953–1967, Online. Association for Computational Linguistics. ↗
  49. Aurélie Névéol, Yoann Dupont, Julien Bezançon, and Karën Fort. 2022. French crows-pairs: Extending a challenge dataset for measuring social bias in masked language models to a language other than english. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8521–8531.
  50. Judea Pearl. 2014. Interpretation and identification of causal mediation. Psychological methods, 19. ↗
  51. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  52. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 8–14, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  53. Timo Schick, Sahana Udupa, and Hinrich Schütze. 2021. Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in NLP. Transactions of the Association for Computational Linguistics, 9:1408– 1424. ↗
  54. Michael Sejr Schlichtkrull, Nicola De Cao, and Ivan Titov. 2021. Interpreting graph neural networks for {nlp} with differentiable edge masking. In International Conference on Learning Representations. ↗
  55. Thibault Sellam, Steve Yadlowsky, Ian Tenney, Jason Wei, Naomi Saphra, Alexander D’Amour, Tal Linzen, Jasmijn Bastings, Iulia Raluca Turc, Jacob Eisenstein, Dipanjan Das, and Ellie Pavlick. 2022. The multiberts: BERT reproductions for robustness analysis. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. ↗
  56. Irene Solaiman and Christy Dennison. 2021. Process for adapting language models to society (palms) with values-targeted datasets. In Advances in Neural Information Processing Systems, volume 34, pages 5861– 5873. Curran Associates, Inc. ↗
  57. Zeerak Talat, Aurélie Névéol, Stella Biderman, Miruna Clinciu, Manan Dey, Shayne Longpre, Sasha Luccioni, Maraim Masoud, Margaret Mitchell, Dragomir Radev, et al. 2022. You reap what you sow: On the challenges of bias evaluation under multilingual settings. In Proceedings of BigScience Episode# 5– Workshop on Challenges & Perspectives in Creating Large Language Models, pages 26–41.
  58. Oskar van der Wal, Dominik Bachmann, Alina Leidinger, Leendert van Maanen, Willem Zuidema, and Katrin Schulz. 2023. Undesirable biases in nlp: Averting a crisis of measurement. arXiv preprint arXiv:2211.13709. ↗
  59. Oskar Van Der Wal, Jaap Jumelet, Katrin Schulz, and Willem Zuidema. 2022. The birth of bias: A case study on the evolution of gender bias in an English language model. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 75–75, Seattle, Washington. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chintam-2023-identifying-adapting-transformer-components-responsible/