Elamur Research⌕
☰
home / papers / 2023 / Debiasing Algorithm through Model Adaptation

Debiasing Algorithm through Model Adaptation

Paper · 2023 · Tomasz Limisiewicz, David Mareček and Tomaš Musil

Summary

Large language models such as LLaMA absorb gender stereotypes from their training data, for example linking occupations like mechanic or nurse with one gender. The authors first measure this in LLaMA 7B–65B using WinoBias, StereoSet and a new interpretable generation metric. For that metric, prompts are built from the Bolukbasi et al. (2016) profession list and scored by the gap between "he" and "she" probabilities, which is then regressed on annotated factual and stereotypical gender scores. Causal tracing restores clean MLP activations inside a noise-corrupted run, and it identifies mid-upper feed-forward layers as the main carriers of bias. Their method, DAMA, multiplies the output weights of those layers by an orthogonal projection that removes a learned gender subspace, so the architecture and parameter count stay unchanged. The edited models are clearly less biased on all three bias measures, while perplexity and scores on OpenBookQA, ARC and MMLU stay close to the original. This offers a lightweight, interpretable post-hoc fix for open models, and the code and models are publicly released.

Key points

  • Bias metric: a linear fit y = a_s·x_s + a_f·x_f + b_0 relates the he/she probability gap to stereotypical and factual gender scores.
    • It fits all LLaMA sizes moderately well (R² > 0.35).
    • Factual cues outweigh stereotypical ones (a_f > a_s).
    • The intercept is positive, meaning the models generally favour male pronouns.
  • Causal tracing: following Meng et al., profession tokens are corrupted with Gaussian noise at about 3× the embedding standard deviation. The results single out mid-upper MLP layers as the "gender bias mediator."
  • DAMA configuration:
    • LLaMA 7B: layers 21–29 edited, 256 of 4096 dimensions projected out.
    • LLaMA 13B: layers 26–36, 512 dimensions.
    • LLaMA 30B: layers 39–55, 1024 dimensions.
    • LLaMA 65B: layers 52–71, 2048 dimensions.
  • Evaluation suite:
    • Bias: WinoBias (accuracy plus ∆S and ∆G gaps) and the StereoSet gender intrasentence subset (765 items; lms, ss, icat).
    • Language modeling: perplexity on Wikipedia 103.
    • Downstream tasks: OBQA, ARC Easy/Challenge and 5-shot MMLU.
  • Baselines and release: DAMA is compared with MEMIT and LoRA (rank 8) applied to the same 9 layers of LLaMA 7B. The code is at github.com/tomlimi/DAMA.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Charles UniversityPrague · Czechia133 works · 20,686 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 1Research subjects and languages the work is about.Bias, fairness & safety · 517 works

Cite

Cite as Tomasz Limisiewicz, David Mareček and Tomaš Musil. 2023. Debiasing Algorithm through Model Adaptation. arXiv:2310.18913.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.18913 · arXiv 2310.18913

@misc{limisiewicz2023debiasing,
  title = {{Debiasing Algorithm through Model Adaptation}},
  author = {Tomasz Limisiewicz, David Marevcek and Tomavs Musil},
  year = {2023},
  doi = {10.48550/arxiv.2310.18913},
  eprint = {2310.18913},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.18913}
}

References

45 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. LEACE: Perfect linear concept erasure in closed form. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/ forum?id=awIpKpwTwF. ↗
  2. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna M. Wallach. Language (technology) is Power: A Critical Survey of ”bias” in NLP. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguisti… — Language (Technology) is Power: A Critical Survey of “Bias” in NLP
  3. Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. Stereotyping Norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confe… find
  4. Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam Tauman Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, Decem…
  5. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018.
  6. Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6491–6506, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi… find
  7. Pieter Delobelle, Ewoenam Tokpo, Toon Calders, and Bettina Berendt. Measuring fairness with biased rulers: A comparative study on bias metrics for pre-trained language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human… find
  8. Chin-Lun Fu, Zih-Ching Chen, Yun-Ru Lee, and Hung-yi Lee. AdapterBias: Parameter-efficient Token-dependent Representation Shift for Adapters in NLP Tasks. In Marine Carpuat, Marie-Catherine de Marneffe, and Iván Vladimir Meza Ruı́z (eds.), Findings of the Association for Computational Linguistics:… find
  9. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer Feed-Forward Layers Are Key-Value Memories. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Vi… find
  10. Michael Gira, Ruisu Zhang, and Kangwook Lee. Debiasing Pre-trained Language Models via Efficient Fine-tuning. In Bharathi Raja Chakravarthi, B. Bharathi, John P. McCrae, Manel Zarrouk, Kalika Bali, and Paul Buitelaar (eds.), Proceedings of the Second Workshop on Language Technology for Equality, Div… find
  11. A.S. Goldberger, W.A. Shenhart, and S.S. Wilks. Econometric Theory. WILEY SERIES in PROB-ABILITY and STATISTICS: APPLIED PROBABILITY and STATIST ICS SECTION Series. J. Wiley, 1964. ISBN 978-0-471-31101-0. URL https://books.google.com/books?id= KZq5AAAAIAAJ. ↗
  12. Yue Guo, Yi Yang, and Ahmed Abbasi. Auto-Debias: Debiasing Masked Language Models with Automated Biased Prompts. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 20…
  13. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021.
  14. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum? id=nZeVKeeFYf9. ↗
  15. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1412.6980. ↗
  16. Anne Lauscher, Tobias Lueken, and Goran Glavaš. Sustainable modular debiasing of language models. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 4782–4797, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.…
  17. Hector Levesque, Ernest Davis, and Leora Morgenstern. The Winograd Schema Challenge. 2011. find
  18. Tomasz Limisiewicz and David Mareček. Don’t forget about pronouns: Removing gender bias in language models without losing factual gender information. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pp. 17–29, Seattle, Washington, July 2022. Association for… find
  19. Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. Gender Bias in Neural Natural Language Processing, pp. 189–202. Springer International Publishing, Cham, 2020a. ISBN 978-3-030-62077-6. doi: 10.1007/978-3-030-62077-6 14. URL https://doi.org/10.1007/978-3-030-62077-6_14. — [doi:10.1007/978-3-030-62077-6](https://doi.org/10.1007/978-3-030-62077-6) ↗
  20. Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. Gender Bias in Neural Natural Language Processing. In Vivek Nigam, Tajana Ban Kirigin, Carolyn L. Talcott, Joshua D. Guttman, Stepan L. Kuznetsov, Boon Thau Loo, and Mitsuhiro Okada (eds.), Logic, Language, and Security - E…
  21. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 36, 2022a. find
  22. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and Editing Factual Associations in GPT. In NeurIPS, 2022b. URL http://papers.nips.cc/paper_files/paper/2022/hash/ 6f1d43d5a82a37e89b0665b33bf3a182-Abstract-Conference.html. ↗
  23. Kevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov, and David Bau. Mass-Editing Memory in a Transformer. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. find
  24. OpenReview.net, 2023. URL https://openreview.net/pdf?id=MkbcAHIYgyS. ↗
  25. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
  26. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381–2391, Brussels, Belgium, October-November 2018.… find
  27. Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. In International Conference on Learning Representations, 2022. URL https://openreview.net/pdf?id=0DcZxeWfOPt. ↗
  28. Moin Nadeem, Anna Bethke, and Siva Reddy. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Paper…
  29. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process…
  30. Roberto Navigli, Simone Conia, and Björn Ross. Biases in large language models: Origins, inventory, and discussion. J. Data and Information Quality, 15(2), jun 2023. ISSN 1936-1955. doi: 10.1145/3597307. URL https://doi.org/10.1145/3597307. — [doi:10.1145/3597307](https://doi.org/10.1145/3597307) ↗
  31. Judea Pearl. Direct and indirect effects. In Proceedings of the Seventeenth conference on Uncertainty in artificial intelligence, UAI’01, pp. 411–420, San Francisco, CA, USA, August 2001. Morgan Kaufmann Publishers Inc. ISBN 978-1-55860-800-9. find
  32. Leonardo Ranaldi, Elena Sofia Ruzzetti, Davide Venditti, Dario Onorati, and Fabio Massimo Zanzotto. A Trip Towards Fairness: Bias and De-biasing in Large Language Models. CoRR, abs/2305.13862, 2023. doi: 10.48550/arXiv.2305.13862. URL https://doi.org/10. 48550/arXiv.2305.13862. find
  33. Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan Cotterell. Linear Adversarial Concept Erasure. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Marylan… find
  34. PMLR, 2022. URL https://proceedings.mlr.press/ v162/ravfogel22a.html. ↗
  35. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender Bias in Coreference Resolution. In Marilyn A. Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language T…
  36. Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, Matteo Negri, and Marco Turchi. Gender Bias in Machine Translation. Transactions of the Association for Computational Linguistics, 9:845–874, 08 2021. ISSN 2307-387X. doi: 10.1162/tacl a 00401. URL https://doi.org/10.1162/ tacl_a_00401.
  37. Karolina Stanczak and Isabelle Augenstein. A Survey on Gender Bias in Natural Language Processing. CoRR, abs/2112.14168, 2021. URL https://arxiv.org/abs/2112.14168. — [arXiv:2112.14168](https://arxiv.org/abs/2112.14168) ↗
  38. Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1679–1684, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P1…
  39. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Language Models. CoRR, abs/2…
  40. Oskar van der Wal, Dominik Bachmann, Alina Leidinger, Leendert van Maanen, Willem Zuidema, and Katrin Schulz. Undesirable biases in nlp: Averting a crisis of measurement, 2023. find
  41. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper… ↗
  42. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart M. Shieber. Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias. CoRR, abs/2004.12265, 2020. URL https://arxiv.org/abs/2004.12265. — [arXiv:2004.12265](https://arxiv.org/abs/2004.12265) ↗
  43. Zhongbin Xie and Thomas Lukasiewicz. An Empirical Analysis of Parameter-efficient Methods for Debiasing Pre-trained Language Models. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Lo… find
  44. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods. In Marilyn A. Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computat…
  45. Ran Zmigrod, S. J. Mielke, Hanna M. Wallach, and Ryan Cotterell. Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology. In Anna Korhonen, David R. Traum, and Lluı́s Màrquez (eds.), Proceedings of the 57th Conference of the Association for Computational…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/limisiewicz-2023-debiasing-algorithm-through-model-adaptation/