Elamur Research⌕
☰
home / papers / 2024 / Dialect prejudice predicts AI decisions about people's character, employability, and criminality

Dialect prejudice predicts AI decisions about people's character, employability, and criminality

Paper · 2024 · Valentin Hofmann, Pratyusha Ria Kalluri, Dan Jurafsky et al.

Summary

Earlier bias research on language models mostly asked about racial groups by name. This paper asks whether models also hold hidden racial prejudice that is triggered only by how a person writes, specifically African American English (AAE) compared with Standardized American English (SAE). The authors adapt the sociolinguistic matched guise technique into Matched Guise Probing. They place AAE or SAE texts, such as tweets, inside prompts that ask about the writer, and they never mention race. They then compare the model's predicted traits across the two dialects. Twelve models from five families were tested, including GPT2, RoBERTa, T5, GPT3.5 and GPT4.

The models linked AAE writers to very negative, old-fashioned stereotypes, even though their stated views of African Americans were positive. In hypothetical decisions, AAE writers were matched to less prestigious jobs, were convicted more often, and received the death penalty more often. Neither larger model size nor human-feedback training removed this effect. Human feedback actually widened the gap between what models say openly and what they associate implicitly. This matters because a model can pass explicit bias checks and still discriminate through dialect.

Key points

  • Models and method. Matched Guise Probing was applied to 12 models (GPT2, RoBERTa, T5, GPT3.5, GPT4) in two settings:
    • meaning-matched, where the SAE text is a translation of the AAE text;
    • non-meaning-matched, where the texts differ in content as well.
  • Human stereotype baseline. Covert stereotypes were measured with adjectives from the Princeton Trilogy studies. They were harsher than any stereotypes of African Americans that human participants have shown in experiments, and were closest to pre-civil-rights results.
  • IQ analysis. Models labeled AAE writers as low-IQ at a rate of 67.0%, compared with 60.3% for SAE writers (χ²(1, N=240)=547.2, p<.001). The gap held for every model family, for example GPT3.5 at 36.7% vs 16.1% and GPT4 at 94.5% vs 83.0%. Holm-Bonferroni correction was applied.
  • Generic noise is not the explanation. An invented feature that AAE does not have ("he am going") produced no significant stereotype effect (t(89)=−1.7, p=1.0). Randomly perturbed SAE text evoked weaker stereotypes than AAE, even though models understood it worse (higher perplexity).
  • Scale and feedback. Smaller models showed less prejudice, with GPT2 base as the one exception to the pattern. Human feedback hid overt racism but did not reduce the covert form.

Interesting ideas

  • I1. Probing with pairs that differ only in dialect, and attributing any output difference to that one variable, gives a clean causal test of sensitivity to surface form.
  • I2. An invented feature that belongs to no real dialect acts as a negative control, separating dialect-specific effects from a general reaction to non-standard text.
  • I3. Before comparing label choices, the prior probability of each label token is calibrated away (Zhao et al.), so the model's preference for a token does not pass as a decision.
  • I4. Two evaluation settings are run side by side: one rigorous (meaning-matched translations) and one realistic (natural texts whose topics also differ).
  • I5. Measuring both what a system states openly and what it implicitly associates reveals failures that one-sided checks miss.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Build AAE/SAE minimal-pair suites and require that frames and the world of the text stay identical across each pair. Treat any divergence as a regression.
  • I2 (learn): When evaluating dialect robustness, add synthetic ungrammatical variants. This shows whether parser errors come from real dialect features or from noise in general.
  • I3 (learn): Calibrate label priors in this way whenever an LLM teacher tier picks between candidate labels or parses.
  • I4 (tree): Test UD dialect conversion both on aligned translations and on naturally occurring dialect text. The second shows how topic shifts interact with dialect features.
  • I5 (frames): Compare explicit statements about a group in the text with the attributes that frames and implicature assign to a speaker. This flags inferences driven only by speech style rather than by the content of the text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States434 works · 314,419 citations🔬Allen Institute for Artificial IntelligenceSeattle · United States275 works · 97,634 citations🎓Ludwig-Maximilians-Universität MünchenMunich · Germany107 works · 56,478 citations🎓University of ChicagoChicago · United States45 works · 50,617 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Arts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 worksSociology and Political Science · 577 works

Cite

Cite as Valentin Hofmann, Pratyusha Ria Kalluri, Dan Jurafsky and Sharese King. 2024. Dialect prejudice predicts AI decisions about people's character, employability, and criminality. arXiv:2403.00742.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2403.00742 · arXiv 2403.00742

@misc{hofmann2024dialect,
  title = {{Dialect prejudice predicts AI decisions about people's character, employability, and criminality}},
  author = {Valentin Hofmann, Pratyusha Ria Kalluri, Dan Jurafsky and Sharese King},
  year = {2024},
  doi = {10.48550/arxiv.2403.00742},
  eprint = {2403.00742},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.00742}
}

References

110 references, 35 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abubakar Abid, Maheen Farooqi, and James Zou. Persistent anti-muslim bias in large language models. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 298–306, 2021.
  2. Nikolaos Aletras, Dimitrios Tsarapatsanis, Daniel Preoţiuc-Pietro, and Vasileios Lampos. Predicting judicial decisions of the European Court of Human Rights: A natural language processing perspective. PeerJ Computer Science, 2:e93, 2016. find
  3. Carolyn P. Atkins. Do employment recruiters discriminate on the basis of nonstandard dialect? Counseling, 30(3):108–118, 1993. ↗
  4. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv 2204.05862, 2022. ↗
  5. Peter Ball. Stereotypes of Anglo-Saxon and non-Anglo-Saxon accents: Some exploratory Australian studies with the matched guise technique. Language Sciences, 5(2):163–183, 1983. ↗
  6. Christine Basta, Marta R. Costa-jussà, and Noe Casas. Evaluating the underlying gender bias in contextualized word embeddings. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 33–39, 2019.
  7. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623, 2021. find
  8. Hilary B. Bergsieker, Lisa M. Leslie, Vanessa S. Constantine, and Susan T. Fiske. Stereotyping by omission: Eliminate the negative, accentuate the positive. Journal of Personality and Social Psychology, 102(6):1214–1238, 2012. ↗
  9. Andrew Billings. Beyond the Ebonics debate: Attitudes about Black and Standard American English. Journal of Black Studies, 36(1):68–81, 2005. find
  10. J. Stewart Black and Patrick van Esch. AI-enabled recruiting: What is it and how should a manager use it? Business Horizons, 63(2):215–226, 2020. ↗
  11. Su Lin Blodgett and Brendan O’Connor. Racial disparity in natural language processing: A case study of social media African-American English. arXiv 1707.00061, 2017. ↗
  12. Su Lin Blodgett, Lisa Green, and Brendan O’Connor. Demographic dialectal variation in social media: A case study of African-American English. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1119–1130, 2016.
  13. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. Language (technology) is power: A critical survey of “bias” in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454–5476, 2020.
  14. Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam T. Kalai. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In Advances in Neural Information Processing Systems 29, 2016.
  15. Eduardo Bonilla-Silva. The new racism: Racial structure in the United States, 1960s-1990s. In Race, Ethnicity, and Nationality in the United States: Toward the Twenty-First Century, pages 55–101. find
  16. Westview Press, Boulder, CO, 1999. find
  17. Eduardo Bonilla-Silva. Racism without Racists. Rowman & Littlefield, Plymouth, UK, 2014. find
  18. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv 2005.14165, 2020. ↗
  19. Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186, 2017. ↗
  20. Myra Cheng, Esin Durmus, and Dan Jurafsky. Marked personas: Using natural language prompts to measure stereotypes in language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, pages 1504–1532, 2023.
  21. Journal of Employment Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. PaLM: Scaling language modeling with pathways. arXiv 2204.02311, 2022. ↗
  22. Paul F. Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems 30, 2017. ↗
  23. Faye Crosby, Stephanie Bromley, and Leonard Saxe. Recent unobtrusive studies of Black and White discrimination and prejudice: A literature review. Psychological Bulletin, 87(3):546–563, 1980. ↗
  24. Alexander M. Czopp and Margo J. Monteith. Thinking well of African Americans: Measuring complimentary stereotypes and negative prejudice. Basic and Applied Social Psychology, 28(3):233–250, 2006. ↗
  25. Mark Davies. The Corpus of Contemporary American English as the first reliable monitor corpus of English. Literary and Linguistic Computing, 25(4):447–464, 2010. ↗
  26. Pieter Delobelle, Ewoenam Tokpo, Toon Calders, and Bettina Berendt. Measuring fairness with biased rulers: A comparative study on bias metrics for pre-trained language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1693–1706, 2022. find
  27. Patricia Devine and Andrew Elliot. Are racial stereotypes really fading? The Princeton Trilogy revisited. Personality and Social Psychology Bulletin, 21(11):1139–1150, 1995. find
  28. Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the Colossal Clean Crawled Corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1286–1305, 2021. find
  29. John F. Dovidio and Samuel L. Gaertner. Aversive racism. Advances in Experimental Social Psychology, 36:1–52, 2004. find
  30. Gabriel Doyle. Mapping dialectal variation by querying social media. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 98–106, 2014. find
  31. Jacob Eisenstein. What to do about bad language on the internet. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 359–369, 2013.
  32. Jacob Eisenstein. Systematic patterning in phonologically-motivated orthographic variation. Journal of Sociolinguistics, 19 (2):161–188, 2015. ↗
  33. Jacob Eisenstein, Brendan O’Connor, Noah A. Smith, and Eric P. Xing. A latent variable model for geographic lexical variation. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 1277–1287, 2010.
  34. Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair NLP models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, pages 11737–11762, 2023.
  35. Xavier Ferrer, Tom van Nuenen, Jose M. Such, and Natalia Criado. Discovering and categorising language biases in Reddit. arXiv 2008.02754, 2020. ↗
  36. W. Nelson Francis and Henry Kucera. Brown Corpus manual, 1979. find
  37. Stephen J. Gaies and Jacqueline D. Beebe. The matched-guise technique for measuring attitudes and their implications for language education: A critical assessment, 1991. find
  38. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800GB dataset of diverse text for language modeling. arXiv 2101.00027, 2021. ↗
  39. Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences, 115(16):E3635–E3644, 2018. ↗
  40. Negin Ghavami and Letitia A. Peplau. An intersectional analysis of gender and ethnic stereotypes. Psychology of Women Quarterly, 37(1):113–127, 2013. ↗
  41. Gustave M. Gilbert. Stereotype persistance and change among college students. Journal of Abnormal and Social Psychology, 46(2):245–254, 1951. find
  42. Tanya Golash-Boza. A critical and comprehensive sociological theory of race and racism. Sociology of Race and Ethnicity, 2 (2):129–141, 2016. ↗
  43. Lisa J. Green. African American English: A Linguistic Introduction. Cambridge University Press, Cambridge, UK, 2002. find
  44. Sophie Groenwold, Lily Ou, Aesha Parekh, Samhita Honnavalli, Sharon Levy, Diba Mirza, and William Yang Wang. Investigating African-American Vernacular English in transformer-based text generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5877–5883, 2020.
  45. Jeffrey Grogger. Speech patterns and racial wage inequality. The Journal of Human Resources, 46(1):1–25, 2011. ↗
  46. Lisa Anne Hendricks and Aida Nematzadeh. Probing image-language transformers for verb understanding. arXiv 2106.09141, 2021. ↗
  47. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. arXiv 2203.15556, 2022. ↗
  48. Yuan Huang, Diansheng Guo, Alice Kasakoff, and Jack Grieve. Understanding U.S. regional linguistic variation with Twitter data analysis. Computers, Environment and Urban Systems, 59:244–255, 2016. ↗
  49. Richard A. Hudson. Sociolinguistics. Cambridge University Press, Cambridge, UK, 1996. find
  50. Bradley T. Hughes, Sanjay Srivastava, Magdalena Leszko, and David M. Condon. Occupational prestige: The status component of socioeconomic status, 2022. find
  51. Anna Lena Hunkenschroer and Christoph Luetge. Ethics of AI-enabled recruiting and selection: A review and research agenda. Journal of Business Ethics, 178(4):977–1007, 2022. ↗
  52. Mary R. Jackman and Michael J. Muha. Education and intergroup attitudes: Moral enlightenment, superficial democratic commitment, or ideological refinement? American Sociological Review, 49(6):751–769, 1984. ↗
  53. Lavender Yao Jiang, Xujin Chris Liu, Nima Pour Nejatian, Mustafa Nasir-Moin, Duo Wang, Anas Abidin, Kevin Eaton, Howard Antony Riina, Ilya Laufer, Paawan Punjabi, Madeline Miceli, Nora C. Kim, Cordelia Orillac, Zane Schnurman, Christopher Livia, Hannah Weiss, David Kurland, Sean Neifert, Yosef Dastagirzada, Douglas Kondziolka, Alexander T. M. Cheung, Grace Yang, Ming Cao, Mona Flores, Anthony B. Costa, Yindalon Aphinyanaphongs, Kyunghyun Cho, and Eric Karl Oermann. Health system-scale language models are all-purpose prediction engines. Nature, 619(7969):357–362, 2023. ↗
  54. Taylor Jones. Toward a description of African American Vernacular English dialect regions using “Black Twitter”. American Speech, 90(4):403–440, 2015. find
  55. Anna Jørgensen, Dirk Hovy, and Anders Søgaard. Challenges of studying and processing dialects in social media. In Proceedings of the Workshop on Noisy User-generated Text, pages 9–18, 2015. find
  56. Anna Jørgensen, Dirk Hovy, and Anders Søgaard. Learning a POS tagger for AAVE-like language. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1115–1120, 2016.
  57. Dan Jurafsky and James H. Martin. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Prentice Hall, Upper Saddle River, NJ, 2000. find
  58. Marvin Karlins, Thomas Coffman, and Gary Walters. On the fading of social stereotypes: Studies in three generations of college students. Journal of Personality and Social Psychology, 13(1):1–16, 1969. ↗
  59. Enkelejda Kasneci, Kathrin Sessler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, Stepha Krusche, Gitta Kutyniok, Tilman Michaeli, Claudia Nerdel, Jürgen Pfeffer, Oleksandra Poquet, Michael Sailer, Albrecht Schmidt, Tina Seidel, Matthias Stadler, Jochen Weller, Jochen Kuhn, and Gjergji Kasneci. ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103:102274, 2023.
  60. Daniel Katz and Kenneth Braly. Racial stereotypes of one hundred college students. Journal of Abnormal and Social Psychology, 28(3):280–290, 1933. ↗
  61. Sharese King, Charlotte Vaughn, and Adam Dunbar. Dialect on trial: Raciolinguistic ideologies in perceptions of AAVE and MAE codeswitching. University of Pennsylvania Working Papers in Linguistics, 28(2):51–59, 2022. find
  62. Abdullatif Köksal, Omer Yalcin, Ahmet Akbiyik, M. Kilavuz, Anna Korhonen, and Hinrich Schuetze. Language-agnostic bias detection in language models with bias probing. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12735–12747, 2023. find
  63. Courtney Kurinec and Charles Weaver. “Sounding Black”: Speech stereotypicality activates racial stereotypes and expectations about appearance. Frontiers in Psychology, 12:785283, 2021. find
  64. Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W. Black, and Yulia Tsvetkov. Measuring bias in contextualized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 166–172, 2019.
  65. Wallace E. Lambert, Richard C. Hodgson, Robert C. Gardner, and Stanley Fillenbaum. Evaluational reactions to spoken languages. Journal of Abnormal and Social Psychology, 60:44–51, 1960. ↗
  66. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models. arXiv 2211.09110, 2022. ↗
  67. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv 1907.11692, 2019. ↗
  68. Li Lucy and David Bamman. Gender and representation bias in GPT-3 generated stories. In Proceedings of the Third Workshop on Narrative Understanding, pages 48–55, 2021.
  69. Reid Luhman. Appalachian English stereotypes: Language attitudes in Kentucky. Language in Society, 19(3):331–348, 1990. ↗
  70. Stephanie Madon, Kathy Aboufadel, Eulices Montiel, Alison Smith, Polly Palumbo, and Lee Jussim. Ethnic and national stereotypes: The Princeton Trilogy revisited and revised. Personality and Social Psychology Bulletin, 27(8):996–1010, 2001. ↗
  71. Douglas S. Massey and Garvey Lundy. Use of Black English and racial discrimination in urban housing markets: New methods and findings. Urban Affairs Review, 36(4):452–469, 2001. ↗
  72. Justus Mattern, Zhijing Jin, Mrinmaya Sachan, Rada Mihalcea, and Bernhard Schölkopf. Understanding stereotypes in language models: Towards robust measurement and zero-shot debiasing. arXiv 2212.10678, 2022. ↗
  73. Masha Medvedeva, Michel Vols, and Martijn Wieling. Using machine learning to predict decisions of the European Court of Human Rights. Artificial Intelligence and Law, 28(2):237–266, 2020. ↗
  74. Moin Nadeem, Anna Bethke, and Siva Reddy. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5356–5371, 2021.
  75. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. CrowS-Pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1953–1967, 2020.
  76. John Nay, David Karamardian, Sarah B. Lawsky, Wenting Tao, Meghana Bhat, Raghav Jain, Aaron Travis Lee, Jonathan H. Choi, and Jungo Kasai. Large language models as tax attorneys: A case study in legal capabilities emergence. arXiv 2306.07075, 2023. ↗
  77. Dong Nguyen and Jack Grieve. Do word embeddings capture spelling variation? In Proceedings of the 28th International Conference on Computational Linguistics, pages 870–881, 2020. find
  78. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, C. J. Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. GPT-4 technical report. arXiv 2303.08774, 2023. ↗
  79. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. arXiv 2203.02155, 2022. ↗
  80. Kay Payne, Joe Downing, and John Christopher Fleming. Speaking Ebonics in a professional context: The role of ethos/source credibility and perceived sociability of the speaker. Journal of Technical Writing and Communication, 30(4):367–383, 2000. ↗
  81. Geoffrey Pullum. African American Vernacular English is not standard English with mistakes. In The Workings of Language: From Prescriptions to Perspectives, pages 39–58. Praeger Publishers, Westport, CT, 1999. find
  82. Thomas Purnell, William Idsardi, and John Baugh. Perceptual and phonetic experiments on American English dialect identification. Journal of Language and Social Psychology, 18(1):10–30, 1999. ↗
  83. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. find
  84. Jack Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d’Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis & insights from training Gopher. arXiv 2112.11446, 2021. ↗
  85. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020. ↗
  86. Siva Reddy, Danqi Chen, and Christopher D. Manning. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266, 2019. ↗
  87. John R. Rickford. African American Vernacular English: Features, Evolution, Educational Implications. Blackwell, Malden, MA, 1999. find
  88. John R. Rickford and Sharese King. Language and linguistics on trial: Hearing Rachel Jeantel (and other vernacular speakers) in the courtroom and beyond. Language, 92(4):948–988, 2016. ↗
  89. José Rodriguez, Aaron Cargile, and Marc Rich. Reactions to African-American Vernacular English: Do more phonological features matter? The Western Journal of Black Studies, 28(3):407–414, 2004. find
  90. Maggie Ronkin and Helen E. Karn. Mock Ebonics: Linguistic racism in parodies of Ebonics on the internet. Journal of Sociolinguistics, 3(3):360–380, 1999. ↗
  91. Jonathan Rosa and Nelson Flores. Unsettling race and language: Toward a raciolinguistic perspective. Language in Society, 46(5):621–647, 2017. ↗
  92. Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. Masked language model scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699–2712, 2020.
  93. Bahar Salehi, Dirk Hovy, Eduard Hovy, and Anders Søgaard. Huntsville, hospitals, and hockey teams: Names can reveal your location. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 116–121, 2017. find
  94. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? arXiv 2303.17548, 2023. ↗
  95. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668–1678, 2019.
  96. Howard Schuman, Charlotte Steeh, Lawrence Bobo, and Maria Krysan, editors. Racial Attitudes in America: Trends and Interpretations. Harvard University Press, Cambridge, MA, 1997. find
  97. Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. The woman worked as a babysitter: On biases in language generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3407–3412, 2019. ↗
  98. Tom Smith and Jaesok Son. Measuring occupational prestige on the 2012 General Social Survey, 2014. find
  99. Ryan Steed, Swetasudha Panda, Ari Kobren, and Michael Wick. Upstream mitigation is not all you need: Testing the bias transfer hypothesis in pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 3524–3542, 2022. find
  100. Ian Stewart. Now we stronger than ever: African-American English syntax in Twitter. In Proceedings of the Student Research Workshop at the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 31–37, 2014.
  101. Harry Surden. Artificial intelligence and law: An overview. Georgia State University Law Review, 35(4):1305–1337, 2019. find
  102. Studs Terkel. Race: How Blacks and Whites Think and Feel about the American Obsession. New Press, New York City, NY, 1992. find
  103. Erik R. Thomas and Jeffrey Reaser. Delimiting perceptual cues used for the ethnic labeling of African American and European American voices. Journal of Sociolinguistics, 8(1):54–87, 2004. ↗
  104. Nancy T. Tippins, Frederick L. Oswald, and S. Morton McPhail. Scientific, legal, and ethical concerns about AI-based personnel selection tools: A call to action. Personnel Assessment and Decisions, 7(2), 2021. ↗
  105. Ashwani Kumar Upadhyay and Komal Khandelwal. Applying artificial intelligence: Implications for recruitment. Strategic HR Review, 17(5):255–258, 2018. ↗
  106. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William Isaac, Sean Legassick, Geoffrey Irving, and Iason Gabriel. Ethical and social risks of harm from language models. arXiv 2112.04359, 2021. ↗
  107. Ethan Zhang and Yi Zhang. Average precision. In Encyclopedia of Database Systems, pages 192–193. Springer, Boston, MA, 2009. find
  108. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 15–20, 2018. ↗
  109. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate Before Use: Improving Few-shot Performance of Language Models. In Proceedings of the 38th International Conference on Machine Learning, pages 12697–12706. PMLR, July 2021. ↗
  110. Caleb Ziems, Jiaao Chen, Camille Harris, Jessica Anderson, and Diyi Yang. VALUE: Understanding dialect disparity in NLU. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 3701–3720, 2022.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hofmann-2024-dialect-prejudice-predicts-ai-decisions/