How Far Can Bias Go? Tracing Bias from Pre-Training Data to Alignment
Summary
This paper asks where gender-occupation stereotypes in LLMs come from, in particular whether imbalances in pre-training data carry over into what a model generates. Such tracing is usually blocked because commercial training corpora are closed. The authors avoid this problem by using the fully open OLMo 7B and its pre-training corpus, Dolma. For 220 occupations, they count how often gendered terms appear in the same sentence as occupation terms in Dolma. Next, they give zero-shot templates with neutral, positive and negative framings to the base model and to a pair of versions refined through instruction tuning, OLMo 7B SFT and OLMo 7B Instruct, and record how the generated outputs divide by gender. Both distributions are compared with U.S. Bureau of Labor Statistics figures.
Dolma underrepresents women relative to real labour statistics. The base model mirrors this skew and slightly strengthens it, and changing hyperparameters or prompt type has little effect. Instruction tuning raises women's representation, but stereotypical occupational associations remain. The authors conclude that mitigation at the data level deserves more attention, since fixes applied after training are costly and incomplete.
Key points
- Data pipeline: The authors queried Dolma through the WIMBD platform (ElasticSearch), sampling 100,000 documents per occupation. They split the documents into sentences with nltk and counted gender–occupation co-occurrences within each sentence.
- Models:
- OLMo 7B, pre-trained on 2.46 trillion Dolma tokens.
- OLMo 7B SFT, tuned on the Tulu 2 SFT Mix.
- OLMo 7B Instruct, further aligned with DPO on Ultrafeedback Cleaned.
- Prompts: Each model received 13 neutral, 5 positive and 5 negative templates. The base model got sentence-completion prompts, while the tuned models got profile-writing instructions.
- Change in women's representation: Averaged across sectors, the shift was −8.52 for OLMo 7B (amplified underrepresentation), +10.53 for SFT and +27.12 for Instruct.
- Correlation: The authors computed Pearson correlations between data-level and output-level gender proportions. Code is released at github.com/marionthaler/tracing_bias.
Cite
@inproceedings{thaler2026how,
title = {{How Far Can Bias Go? Tracing Bias from Pre-Training Data to Alignment}},
author = {Thaler, Marion and Köksal, Abdullatif and Leidinger, Alina and Korhonen, Anna Anna and Schütze, Hinrich},
booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
year = {2026},
doi = {10.63317/4zeoky6waeng},
url = {https://doi.org/10.63317/4zeoky6waeng}
}References
- David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski. 1985. A learning algorithm for boltzmann machines. Cognitive science, 9(1):147– 169. find
- Sarah Alnegheimish, Alicia Guo, and Yi Sun. 2022. Using natural sentences for understanding biases in language models. arXiv preprint arXiv:2205.06303. — [arXiv:2205.06303](https://arxiv.org/abs/2205.06303) find
- Haozhe An, Christabel Acquaye, Colin Wang, Zongxia Li, and Rachel Rudinger. 2024. Do large language models discriminate in hiring decisions on the basis of race, ethnicity, and gender? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)… find
- Lena Armstrong, Abbey Liu, Stephen MacNeil, and Danaë Metaxa. 2024. The silicone ceiling: Auditing gpt’s race and gender biases in hiring. arXiv preprint arXiv:2405.04412. — [arXiv:2405.04412](https://arxiv.org/abs/2405.04412) find
- Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623. find
- Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. " O’Reilly Media, Inc.". find
- Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. CoRR, abs/1607.06520. find our card
- Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Demsz… find our card
- Rishi Bommasani, Percy Liang, and Tony Lee. 2023. Holistic evaluation of language models. Annals of the New York Academy of Sciences, 1525(1):140–146. find
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… find
- Laura Cabello, Anna Katrine Jørgensen, and Anders Søgaard. 2023. On the independence of association bias and empirical fairness in language models. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, pages 370–378. find
- Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. find
- Shan Chen, Jack Gallifant, Mingye Gao, Pedro Moreira, Nikolaj Munch, Ajay Muthukkumar, Arvind Rajan, Jaya Kolluri, Amelia Fiske, Janna Hastings, Hugo Aerts, Brian Anthony, Leo Anthony Celi, William G. La Cava, and Danielle S. Bitterman. 2024. Cross-care: Assessing the healthcare implications of pre-… — [arXiv:2405.05506](https://arxiv.org/abs/2405.05506) find
- Zhisheng Chen. 2023. Ethics and discrimination in artificial intelligence-enabled recruitment practices. Humanities and Social Sciences Communications, 10(1):1–12. find
- Jeffrey Dastin. 2022. Amazon scraps secret ai recruiting tool that showed bias against women. In Ethics of data and analytics, pages 296–299. Auerbach Publications. find
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. arxiv. arXiv preprint arXiv:1810.04805. find
- Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 862–872… find
- Xiangjue Dong, Yibo Wang, Philip S Yu, and James Caverlee. 2024. Disclosure and mitigation of gender bias in llms. arXiv preprint arXiv:2402.11190. — [arXiv:2402.11190](https://arxiv.org/abs/2402.11190) find
- Yanai Elazar, Akshita Bhagia, Ian Helgi Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Evan Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hannaneh Hajishirzi, Noah A. Smith, and Jesse Dodge. 2024. What’s in my big data? In The Twelfth International Conference on Learning R… find
- Fatma Elsafoury. 2023. Thesis distillation: Investigating the impact of bias in NLP models on hate speech detection. In Proceedings of the Big Picture Workshop, pages 53–65, Singapore. Association for Computational Linguistics. find
- Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898, Melbourne, Australia. Association for Computational Linguistics. find
- Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. 2023. From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair NLP models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume… find our card
- Emilio Ferrara. 2023. Should chatgpt be biased? challenges and risks of bias in large language models. arXiv preprint arXiv:2304.03738. — [arXiv:2304.03738](https://arxiv.org/abs/2304.03738) find
- Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. 2023. Bias and fairness in large language models: A survey. arXiv preprint arXiv:2309.00770. find our card
- Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. 2018. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences, 115(16):E3635–E3644. find
- Seraphina Goldfarb-Tarrant, Rebecca Marchant, Ricardo Muñoz Sánchez, Mugdha Pandya, and Adam Lopez. 2021. Intrinsic bias metrics do not correlate with application bias. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confere… find our card
- Hila Gonen and Yoav Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In Proceedings of the 2019 Workshop on Widening NLP, pages 60–63, Florence, Italy. Association for Computational Linguistics. find our card
- Wei Guo and Aylin Caliskan. 2021. Detecting emergent intersectional biases: Contextualized word embeddings contain a distribution of human-like biases. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 122–133. find
- Vipul Gupta, Pranav Narayanan Venkit, Shomir Wilson, and Rebecca Passonneau. 2024. Sociodemographic bias in language models: A survey and forward path. In Proceedings of the 5th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 295–322, Bangkok, Thailand. Association for Computa… find
- Yuan He, Bailan He, Zifeng Ding, Alisia Lupidi, Yuqicheng Zhu, Shuo Chen, Caiqi Zhang, Jiaoyan Chen, Yunpu Ma, Volker Tresp, and Ian Horrocks. 2025. Supposedly equivalent facts that aren’t? entity frequency in pre-training induces asymmetry in llms. find
- David Hesmondhalgh and Sarah Baker. 2015. Sex, gender and work segregation in the cultural industries. The sociological review, 63(1_suppl):23–36. find
- Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations. find
- Tiancheng Hu, Yara Kyrychenko, Steve Rathje, Nigel Collier, Sander van der Linden, and Jon Roozenbeek. 2023. Generative language models exhibit social identity biases. arXiv preprint arXiv:2310.15819. — [arXiv:2310.15819](https://arxiv.org/abs/2310.15819) find
- Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack Rae, Vishal Maini, Dani Yogatama, and Pushmeet Kohli. 2019. Reducing sentiment bias in language models via counterfactual evaluation. arXiv preprint arXiv:1911.03064. find
- Hayate Iso, Pouya Pezeshkpour, Nikita Bhutani, and Estevam Hruschka. 2025. Evaluating bias in LLMs for job-resume matching: Gender, race, and education. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Techn… find
- Eugenie Jackson and Christina Mendoza. 2020. Setting the Record Straight: What the COM-PAS Core Risk and Need Assessment Is and Is Not. Harvard Data Science Review, 2(1). Https://hdsr.mitpress.mit.edu/pub/hzwo7ax4. find
- Sara Kassir, Lewis Baker, Jackson Dolphin, and Frida Polli. 2023. Ai for hiring in context: a perspective on overcoming the unique challenges of employment research to mitigate disparate impact. AI and Ethics, 3(3):845–868. find
- Abdullatif Köksal, Omer Yalcin, Ahmet Akbiyik, M. Kilavuz, Anna Korhonen, and Hinrich Schuetze. 2023. Language-agnostic bias detection in language models with bias probing. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12735–12747, Singapore. Association for Computa… find
- Anja Lambrecht and Catherine E Tucker. 2018. Algorithmic bias. An empirical study into apparent gender-based discrimination in the display of STEM career ads, 9. find
- Ehsan Latif, Xiaoming Zhai, and Lei Liu. 2023. Ai gender bias, disparities, and fairness: Does training data matter? arXiv preprint arXiv:2312.10833. — [arXiv:2312.10833](https://arxiv.org/abs/2312.10833) find
- Alina Leidinger and Richard Rogers. 2024. How are llms mitigating stereotyping harms? learning from search engine studies. find
- Alina Leidinger, Robert van Rooij, and Ekaterina Shutova. 2023. The language of prompting: What linguistic properties make a prompt successful? In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9210–9232, Singapore. Association for Computational Linguistics. find
- Yiran Liu, Ke Yang, Zehan Qi, Xiao Liu, Yang Yu, and Chengxiang Zhai. 2024. Prejudice and caprice: A statistical framework for measuring social discrimination in large language models. arXiv preprint arXiv:2402.15481. — [arXiv:2402.15481](https://arxiv.org/abs/2402.15481) find
- Abhishek Mandal, Suzanne Little, and Susan Leavy. 2023. Multimodal bias: Assessing gender bias in computer vision models with nlp techniques. In Proceedings of the 25th International Conference on Multimodal Interaction, pages 416–424. find
- Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and… find our card
- Safiya Umoja Noble. 2018. Algorithms of oppression: How search engines reinforce racism. In Algorithms of oppression. New York university press. find
- Daisuke Oba, Masahiro Kaneko, and Danushka Bollegala. 2024. In-contextual gender bias suppression for large language models. In Findings of the Association for Computational Linguistics: EACL 2024, pages 1722–1742, St. Julian’s, Malta. Association for Computational Linguistics. find
- Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. 2019. Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464):447– 453. find
- Hadas Orgad and Yonatan Belinkov. 2022. Choose your lenses: Flaws in gender bias evaluation. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 151–167, Seattle, Washington. Association for Computational Linguistics. find
- Karl Pearson. 1895. Note on Regression and Inheritance in the Case of Two Parents. Proceedings of the Royal Society of London Series I, 58:240– 242. find
- Matúš Pikuliak, Ivana Beňová, and Viktor Bachratý. 2023. In-depth look at word filling societal bias measures. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3648– 3665, Dubrovnik, Croatia. Association for Computational Linguisti… find
- Jo Anne Preston. 1999. Occupational gender segregation trends and explanations. The Quarterly Review of Economics and Finance, 39(5):611– 624. find
- Jennifer L Prewitt-Freilino, T Andrew Caswell, and Emmi K Laakso. 2012. The gendering of language: A comparison of gender equality in countries with gendered, natural gender, and genderless languages. Sex roles, 66(3):268–281. find
- Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems. find
- Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. find our card
- Abel Salinas, Parth Shah, Yuzhong Huang, Robert McCormack, and Fred Morstatter. 2023. The unequal opportunities of large language models: Examining demographic biases in job recommendations by chatgpt and llama. In Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms,… find
- Timo Schick, Sahana Udupa, and Hinrich Schütze. 2021. Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP. Transactions of the Association for Computational Linguistics, 9:1408–1424. find our card
- Emily Schnell and Yang Xu. 2021. A computational evaluation of gender asymmetry in semantic change. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 43. Retrieved from https://escholarship.org/uc/item/47c642f2. ↗
- Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2021. WikiMatrix: Mining 135M parallel sentences in 1620 language pairs from Wikipedia. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1… find our card
- Nikil Selvam, Sunipa Dev, Daniel Khashabi, Tushar Khot, and Kai-Wei Chang. 2023. The tail wagging the dog: Dataset construction biases of social bias benchmarks. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1373–1386, Toro… find
- Preethi Seshadri, Sameer Singh, and Yanai Elazar. 2024. The bias amplification paradox in text-to-image generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6367–63… find
- Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. The woman worked as a babysitter: On biases in language generation. arXiv preprint arXiv:1909.01326. find our card
- Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2021. Societal biases in language generation: Progress and challenges. arXiv preprint arXiv:2105.04054. find our card
- Irene Solaiman. 2023. The gradient of generative ai release: Methods and considerations. In Proceedings of the 2023 ACM conference on fairness, accountability, and transparency, pages 111–122. find
- Bernard Spolsky. 1998. Sociolinguistics. Oxford University Press, Oxford. find
- Karolina Stanczak and Isabelle Augenstein. 2021. A survey on gender bias in natural language processing. arXiv preprint arXiv:2112.14168. — [arXiv:2112.14168](https://arxiv.org/abs/2112.14168) find
- Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating gender bias in natural language processing: Literature review. In Proceedings of the 57th Annual Meeting of the Association for Computati… find our card
- Zeerak Talat, Aurélie Névéol, Stella Biderman, Miruna Clinciu, Manan Dey, Shayne Longpre, Sasha Luccioni, Maraim Masoud, Margaret Mitchell, Dragomir Radev, Shanya Sharma, Arjun Subramonian, Jaesung Tae, Samson Tan, Deepak Tunuguntla, and Oskar Van Der Wal. 2022. You reap what you sow: On the challen… find
- Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Bria… find
- Stefanie Urchs, Veronika Thurner, Matthias Aßenmacher, Christian Heumann, and Stephanie Thiemichen. 2023. How prevalent is gender bias in chatgpt?–exploring german and english chatgpt responses. arXiv preprint arXiv:2310.03031. find
- U.S. Census Bureau. 2019. ACS 5-Year Estimates – Public Use Microdata Sample, 2019. Last accessed February 2024. find
- Claudia Wagner, David Garcia, Mohsen Jadidi, and Markus Strohmaier. 2021. It’s a man’s wikipedia? assessing gender inequality in an online encyclopedia. Proceedings of the International AAAI Conference on Web and Social Media, 9(1):454– 463. find
- Laura Weidinger, John F. J. Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zachary Kenton, Sande Minnich Brown, William T. Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendrick… find our card
- Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling… find
- Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. 2019. Gender bias in contextualized word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1… find our card
- Jinman Zhao, Yitian Ding, Chen Jia, Yining Wang, and Zifan Qian. 2024. Gender bias in large language models across multiple languages. arXiv preprint arXiv:2403.00277. 10. — [arXiv:2403.00277](https://arxiv.org/abs/2403.00277) find
- Language Resource References Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel,… find
- Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Cr… find
This page in Markdown → https://md.research.elamur.ai/free/en/papers/thaler-2026-how-far-can-bias-go/