Elamur Research⌕
☰
home / papers / 2026 / When Debiasing Backfires: Counterintuitive Side Effects of Preprocessing-Based Stereotype Mitigation

When Debiasing Backfires: Counterintuitive Side Effects of Preprocessing-Based Stereotype Mitigation

Paper · 2026 · Yahan Zheng, John J. Guerrerio, Soroush Vosoughi et al.

Summary

The paper asks whether data-level debiasing actually removes stereotypes from pre-trained language models or simply moves them to other groups. The authors built debiased versions of an English Wikipedia snapshot aimed at six demographic groups (female/male, Black/Caucasian, Muslim/Christian). They used three strategies: dropping stereotypical sentences, dropping every mention of a group, and swapping group references. They then pre- or post-trained TinyBERT and GPT-2 on these corpora at two data scales and measured stereotyping toward all groups with StereoSet and CrowS-Pairs.

Stereotype scores typically fell for the targeted group. However, scores for non-targeted groups often shifted toward more stereotyping or counter-stereotyping, sometimes in unrelated categories. These "side effects" were asymmetric and hard to predict. Aggregate benchmark scores tended to hide them, and attention-rollout analysis revealed little change in attention flow. The authors conclude that preprocessing should not be relied on alone. They recommend reporting per-group, side-effect-aware diagnostics.

Key points

  • Models: TinyBERT (4 layers, ~14M parameters, encoder-only, MLM) and GPT-2 (~124M, decoder-only, CLM). Training data was the June 1, 2023 English Wikipedia snapshot. Evaluation used the intra-sentence part of StereoSet and CrowS-Pairs.
  • Interventions and grid: The three strategies were Debias-A-Group (DG), Remove-A-Group (RG) and Swap-References (SR). Each was run in pre- and post-training at 100% and 5% data, giving 12 settings.
  • Data quality: The DG stereotype detector scored F1 = 0.98 on CrowS-Pairs. LLM-generated SR swaps for race and religion were judged satisfactory in 96 of 100 annotated samples.
  • Larger model check (LLaMa-2-7B, RG-female):
    • Female SS dropped from 70.72 to 66.30, and overall iCAT rose from 64.19 to 66.44.
    • Black SS rose from 64.14 to 65.43, and Christian SS rose from 69.75 to 71.03.
  • Limitations: Results use a single seed (42), so claims rest on directional trends rather than significance tests. Code is released on GitHub (InDaCS-Lab).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Dartmouth CollegeHanover · United States41 works · 42,719 citations🎓Oakland UniversityRochester · United States7 works · 180 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSafety Research · 626 works

🧭 Topics 1Research subjects and languages the work is about.Bias, fairness & safety · 587 works

Cite

Cite as Yahan Zheng, John J. Guerrerio, Soroush Vosoughi and Weicheng Ma. 2026. When Debiasing Backfires: Counterintuitive Side Effects of Preprocessing-Based Stereotype Mitigation. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.486

@inproceedings{zheng2026when,
  title = {{When Debiasing Backfires: Counterintuitive Side Effects of Preprocessing-Based Stereotype Mitigation}},
  author = {Yahan Zheng, John J. Guerrerio, Soroush Vosoughi and Weicheng Ma},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.486},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.486}
}

References

36 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samira Abnar and Willem Zuidema. 2020. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4190–4197.
  2. Afra Feyza Akyürek, Sejin Paik, Muhammed Kocyigit, Seda Akbiyik, Serife Leman Runyun, and Derry Wijaya. 2022. On measuring social biases in prompt-based multi-task learning. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 551–564, Seattle, United States. Association f… find
  3. Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam T. Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in Neural Information Processing Systems, volume 29.
  4. Conrad Borchers, Dalia Sara Gala, Benjamin Gilburt, Eduard Oravkin, Wilfried Bounsi, Yuki M. Asano, and Hannah Rose Kirk. 2022. Looking for a handsome carpenter! debiasing gpt-3 job advertisements. arXiv preprint arXiv:2205.11374. find
  5. Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. find
  6. Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 862–872… find
  7. Harnoor Dhingra, Preetiha Jayashanker, Sayali Moghe, and Emma Strubell. 2023. Queer people are people first: Deconstructing sexual identity stereotypes in large language models. arXiv preprint arXiv:2307.00101. — [arXiv:2307.00101](https://arxiv.org/abs/2307.00101) find
  8. David Esiobu, Xiaoqing Tan, Saghar Hosseini, Megan Ung, Yuchen Zhang, Jude Fernandes, Jane Dwivedi-Yu, Eleonora Presani, Adina Williams, and Eric Michael Smith. 2023. Robbie: Robust bias evaluation of large generative language models. arXiv preprint arXiv:2311.18140. find
  9. Yacine Gaci, Boualem Benatallah, Fabio Casati, and Khalid Benabdeslem. 2022. Debiasing pretrained text encoders by paying attention to paying attention. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9582–9602. Association for Computational Linguisti… find
  10. Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. 2024. Bias and fairness in large language models: A survey. Computational Linguistics, 50(3):1097– 1179.
  11. Aparna Garimella, Rada Mihalcea, and Akhash Amarnath. 2022. Demographic-aware language model fine-tuning as a bias mitigation technique. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Na… find
  12. Somayeh Ghanbarzadeh, Yan Huang, Hamid Palangi, Radames Cruz Moreno, and Hamed Khanpour. 2023. Gender-tuning: Empowering fine-tuning for debiasing pre-trained language models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5448–5458, Toronto, Canada. Association for Co… find
  13. Michael Gira, Ruisu Zhang, and Kangwook Lee. 2022. Debiasing pre-trained language models via efficient fine-tuning. In Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion, pages 59–69, Dublin, Ireland. Association for Computational Linguistics. find
  14. Yue Guo, Yi Yang, and Ahmed Abbasi. 2022. Auto-debias: Debiasing masked language models with automated biased prompts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1012–1023, Dublin, Ireland. Association for Computational L…
  15. Lijie Hu, Yixin Liu, Ninghao Liu, Mengdi Huai, Lichao Sun, and Di Wang. 2023. Seat: stable and explainable attention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 12907–12915. find
  16. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. TinyBERT: Distilling BERT for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163– 4174, Online. Association for Computational Lingui…
  17. Xin Liu, Muhammad Khalifa, and Lu Wang. 2023. Bolt: Fast energy-based controlled text generation with tunable biases. arXiv preprint arXiv:2305.12018. find
  18. Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. 2020. Gender bias in neural natural language processing. In Logic, language, and security: essays dedicated to Andre Scedrov on the occasion of his 65th birthday, pages 189–202.
  19. Yu Lu, Jiali Zeng, Jiajun Zhang, Shuangzhi Wu, and Mu Li. 2021. Attention calibration for transformer in neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (… find
  20. Weicheng Ma, Henry Scheible, Brian Wang, Goutham Veeramachaneni, Pratim Chowdhary, Alan Sun, Andrew Koulogeorge, Lili Wang, Diyi Yang, and Soroush Vosoughi. 2023. Deciphering stereotypes in pre-trained language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pr… find
  21. Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long…
  22. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel Bowman. 2020. Crows-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953–1967.
  23. Helen Ngo, Cooper Raterink, João GM Araújo, Ivan Zhang, Carol Chen, Adrien Morisot, and Nicholas Frosst. 2021. Mitigating harm in language models with conditional-likelihood filtration. arXiv preprint arXiv:2108.07790. — [arXiv:2108.07790](https://arxiv.org/abs/2108.07790) find
  24. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, and 1 others. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730–27744.
  25. Swetasudha Panda, Ari Kobren, Michael Wick, and Qinlan Shen. 2022. Don‘t just clean it, proxy clean it: Mitigating bias by proxy in pre-trained models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5073–5085, Abu Dhabi, United Arab Emirates. Association for Computat… find
  26. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  27. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.
  28. Prasanna Sattigeri, Soumya Ghosh, Inkit Padhi, Pierre Dognin, and Kush R. Varshney. 2022. Fair infinitesimal jackknife: Mitigating the influence of biased training data points without refitting. In Advances in Neural Information Processing Systems, volume 35, pages 35894–35906. find
  29. Danielle Saunders, Rosie Sallis, and Bill Byrne. 2022. First the worst: Finding better gender translations during beam search. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3814– 3823, Dublin, Ireland. Association for Computational Linguistics. find
  30. Ewoenam Kwaku Tokpo and Toon Calders. 2022. Text style transfer for bias mitigation using masked language modeling. arXiv preprint arXiv:2201.08643. find
  31. Schrasing Tong, Eliott Zemour, Rawisara Lohanimit, and Lalana Kagal. 2024. Towards resource efficient and interpretable bias mitigation in natural language generation. In NeurIPS Safe Generative AI Workshop. find
  32. Ke Yang, Charles Yu, Yi R. Fung, Manling Li, and Heng Ji. 2023. Adept: A debiasing prompt framework. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 10780–10788. find
  33. Abdelrahman Zayed, Prasanna Parthasarathi, Gonçalo Mordido, Hamid Palangi, Samira Shabanian, and Sarath Chandar. 2023. Deep learning on a healthy data diet: Finding important examples for fairness. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 14593–14601. find
  34. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. 2019. Gender bias in contextualized word embeddings. arXiv preprint arXiv:1904.03310.
  35. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  36. Hanzhang Zhou, Zijian Feng, Zixiao Zhu, Junlang Qian, and Kezhi Mao. 2024. Unibias: Unveiling and mitigating llm bias through internal attention and ffn manipulation. arXiv preprint arXiv:2405.20612. — [arXiv:2405.20612](https://arxiv.org/abs/2405.20612) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zheng-2026-when-debiasing-backfires-counterintuitive-side/