Elamur Research⌕
☰
home / papers / 2022 / You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings

You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings

Paper · 2022 · Zeerak Talat, Aurélie Névéol, Stella Biderman et al.

Summary

This BigScience position paper asks how social bias should be evaluated in large language models that cover many languages. The authors argue that current practice falls short. Definitions of "bias" are often vague, and benchmarks frequently measure something other than what they claim. Many methods only work with particular model architectures, and most resources are English-only and shaped by Anglo-American culture. The paper reviews measurement approaches such as WEAT and CrowS-Pairs and draws on social science work from fields like STS, critical race theory, sociolinguistics and gender studies. It also surveys the 25 largest dense pretrained models, which are mostly built in the US, trained on English, and seldom audited for bias by their creators. From this analysis the authors propose four directions:

  • transparent scoping and documentation of evaluations
  • coverage of stereotypes beyond gender, including intersectional ones
  • datasets curated with awareness of each culture
  • bias measures that do not depend on model architecture but still reflect the task context

The paper matters because multilingual models inherit these gaps, so the languages and cultures they serve bear the harms without anyone measuring them.

Key points

  • Table 1 lists the 25 largest dense pretrained LMs, from 6B parameters (GPT-J-6B) to 530B (MT-NLG). By the table's own listing, only about seven had bias evaluations reported by their creators, including Gopher, GPT-3, LaMDA and GPT-NeoX-20B.
  • Most models in that table come from US-based teams and are trained on English. A few target Chinese or Korean, such as PanGu-α, Yuan 1.0 and HyperCLOVA, and only a handful are multilingual, such as mT5 and XGLM.
  • An estimate cited from the BigScience biomedical group finds that 82% of biomedical and clinical evaluation datasets are in English.
  • The paper groups the weaknesses of existing measures into four problems:
    • an underspecified notion of bias
    • dependence on particular architectures, for example static embeddings versus contextual ones
    • weak relevance across cultures
    • difficulty of validation and interpretation
  • The authors recommend recording the scope of an evaluation, the demographics of the speakers involved, and the annotators. They also urge weighing the social and environmental costs of building ever-larger English models.

Interesting ideas

  • I1. Audit table of models with a "bias eval" column. Listing each model's size, organization, language and access level next to whether a bias evaluation exists quickly exposes reporting gaps. The same format could be used to track evaluation coverage in any model zoo or library.
  • I2. Stereotypes are culture-relative, so localize instead of translating. Assuming parental leave belongs to mothers counts as a stereotype in the US, but it is not one in Sweden, where both parents share leave. Bias datasets for a new language, such as Ukrainian, should therefore be rebuilt with local input rather than machine-translated from English benchmarks.
  • I3. Minimal pairs as a probing design. CrowS-Pairs-style probing contrasts the same sentence aimed at different social groups, so any systematic difference in model behavior signals bias. The template is simple and could be adapted to other probing tasks.
  • I4. Document the people, not just the data. Recording who built and annotated an evaluation, including their background and demographics, makes its validity easier to judge. Researcher-level metadata of this kind would usefully extend datasheets for any annotated corpus.
  • I5. Separate the bias metric from the model architecture. A measure tied to the task rather than to a specific model's internals can be compared across models. This idea transfers to evaluation frameworks generally, beyond bias.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States353 works · 364,806 citations🎓University of EdinburghEdinburgh · United Kingdom255 works · 68,771 citations🎓Tsinghua UniversityBeijing · China137 works · 83,262 citations🎓University of AmsterdamAmsterdam · Netherlands89 works · 81,337 citations🎓National University of SingaporeSingapore · Singapore84 works · 32,204 citations🏢AmazonSeattle · United States70 works · 39,977 citations🎓Yale UniversityNew Haven · United States54 works · 11,964 citations🎓University of California, Los AngelesLos Angeles · United States54 works · 53,688 citations🔬Centre National de la Recherche ScientifiqueParis · France104 works · 60,877 citations🎓Heriot-Watt UniversityEdinburgh · United Kingdom10 works · 5,081 citations🔬Beijing Academy of Artificial IntelligenceBeijing · China16 works · 7,166 citations🏢NvidiaSanta Clara · United States16 works · 4,820 citations🎓Université Paris-SaclayGif-sur-Yvette · France34 works · 5,733 citations🎓Simon Fraser UniversityBurnaby · Canada10 works · 4,261 citations🎓Trinity College DublinDublin · Ireland17 works · 1,222 citations🏢Systems, Applications & Products in Data Processing (Germany)Walldorf · Germany4 works · 2,374 citations🔬Laboratoire Interdisciplinaire des Sciences du NumériqueGif-sur-Yvette · France10 works · 422 citations

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksSocial Sciences · 744 worksSafety Research · 221 works

🧭 Topics 1Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 467 works

Cite

Cite as Zeerak Talat, Aurélie Névéol, Stella Biderman, Miruna Clinciu, Manan Dey, Shayne Longpre, Sasha Luccioni, Maraim Masoud, Margaret Mitchell, Dragomir Radev, Shanya Sharma, Arjun Subramonian, Jaesung Tae, Samson Tan, Deepak Tunuguntla and Oskar Van Der Wal. 2022. You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings. Proceedings of BigScience Episode \#5 – Workshop on Challenges & Perspectives in Creating Large Language Models.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.bigscience-1.3

@inproceedings{talat2022you,
  title = {{You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings}},
  author = {Zeerak Talat, Aurélie Névéol, Stella Biderman, Miruna Clinciu, Manan Dey, Shayne Longpre, Sasha Luccioni, Maraim Masoud, Margaret Mitchell, Dragomir Radev, Shanya Sharma, Arjun Subramonian, Jaesung Tae, Samson Tan, Deepak Tunuguntla and Oskar {Van Der Wal}},
  booktitle = {Proceedings of BigScience Episode \#5 – Workshop on Challenges \& Perspectives in Creating Large Language Models},
  year = {2022},
  doi = {10.18653/v1/2022.bigscience-1.3},
  url = {https://doi.org/10.18653/v1/2022.bigscience-1.3}
}

References

126 references, 40 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. dangers of stochastic parrots: Can language models be too big? . In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 610–623, New York, NY, USA. Association for Computing Machinery. find
  2. Abubakar Abid, Maheen Farooqi, and James Zou. 2021. Persistent Anti-Muslim Bias in Large Language Models. In AIES 2021 - Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society.
  3. Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185–5198, Online. Association for Computational Linguistics.
  4. Daniela Agostinho, Catherine D’Ignazio, Annie Ring, Nanna Bonde Thylstrup, and Kristin Veel. 2019. Uncertain Archives: Approaching the Unknowns, Errors, and Vulnerabilities of Big Data through Cultural Theories of the Archive. Surveillance & Society, 17(3/4):422–441. find
  5. Ruha Benjamin. 2019. Race after technology: abolitionist tools for the new Jim code. Polity, Medford, MA. find
  6. Blaise Aguera y Arcas, Alexander Todorov, and Margaret Mitchell. 2018. Do algorithms reveal sexual orientation or just expose our stereotypes? find
  7. Stella Biderman, Kieran Bicheno, and Leo Gao. 2022. Datasheet for the Pile. arXiv:2201.07311 [cs]. ArXiv: 2201.07311. — [arXiv:2201.07311](https://arxiv.org/abs/2201.07311) find
  8. Hala Al Kuwatly, Maximilian Wich, and Georg Groh. 2020. Identifying and measuring annotator bias based on annotators’ demographic characteristics. In Proceedings of the Fourth Workshop on Online Abuse and Harms, pages 184–190, Online. Association for Computational Linguistics.
  9. Stella Biderman and Walter Scheirer. 2020. Pitfalls in machine learning research: Reexamining the development cycle. In “I Can’t Believe It’s Not Better!” NeurIPS 2020 workshop. find
  10. Liz Allen, Alison O’Connell, and Veronique Kiermer. 2019. How can we ensure visibility and diversity in research contributions? How the Contributor Role Taxonomy (CRediT) is helping the shift from authorship to contributorship. Learned Publishing, 32(1):71–74. find
  11. Abeba Birhane, Pratyusha Kalluri, Dallas Card, William Agnew, Ravit Dotan, and Michelle Bao. 2021. The Values Encoded in Machine Learning Research. arXiv:2106.15590 [cs]. ArXiv: 2106.15590. — [arXiv:2106.15590](https://arxiv.org/abs/2106.15590) find
  12. Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022. GPT-NeoX-20B: An Open-Source Autoregr…
  13. Saleema Amershi, Ece Kamar, Kristin Lauter, Jenn Wortman Vaughan, and Hanna Wallach. 2020. Research Supporting Responsible AI. find
  14. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam Mc-Candlish, Chris Olah,… find
  15. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (Technology) is Power: A Critical Survey of “Bias” in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454–5476, Online. Association for Computational Linguistics.
  16. Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021. Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint… find
  17. Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017. The problem with bias: from allocative to representational harms in machine learning. special interest group for computing. Information and Society (SIGCIS), 2. find
  18. Solon Barocas and Andrew D. Selbst. 2016. Big Data’s Disparate Impact. California Law Review, 104(3). find
  19. Emily Bender. 2019. The \#BenderRule: On Naming the Languages We Study and Why It Matters. find
  20. Emily M. Bender and Batya Friedman. 2018. Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science. Transactions of the Association for Computational Linguistics, 6:587–604. find
  21. Su Lin Blodgett, Johnny Wei, and Brendan O’Connor. 2018. Twitter Universal Dependency parsing for African-American and mainstream American English. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1415–1425, Melbourne, Australi…
  22. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in Neural Informati…
  23. Yan Chen, Christopher Mahoney, Isabella Grasso, Esma Wali, Abigail Matthews, Thomas Middleton, Mariama Njie, and Jeanna Matthews. 2021. Gender Bias and Under-Representation in Natural Language Processing Across Human Languages. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Societ… find
  24. Shikha Bordia and Samuel R. Bowman. 2019. Identifying and Reducing Gender Bias in Word-Level Language Models. In Proceedings of the 2019 Conference of the North, pages 7–15, Minneapolis, Minnesota. Association for Computational Linguistics.
  25. Lu Cheng, Ahmadreza Mosallanezhad, Paras Sheth, and Huan Liu. 2021. Causal Learning for Socially Responsible AI. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence. find
  26. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chr…
  27. Jenny Cheshire. 2007. An untitled review of “style and sociolinguistic variation”. Language, 83(2):432–435. find
  28. Wendy Hui Kyong Chun. 2021. Discriminating data: correlation, neighborhoods, and the new politics of recognition. The MIT Press, Cambridge, Massachusetts. find
  29. Sasha Costanza-Chock. 2018. Design Justice, A.I., and Escape from the Matrix of Domination. Journal of Design and Science. find
  30. Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency, volume 81 of Proceedings of Machine Learning Research, pages 77–91, New York, NY, USA. PML… find
  31. Kimberle Crenshaw. 1991. Mapping the margins: Intersectionality, identity politics, and violence against women of color. Stanford Law Review, 43(6):1241– 1299. find
  32. Paula Czarnowska, Yogarshi Vyas, and Kashif Shah. 2021. Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics. Transactions of the Association for Computational Linguistics, 9:1249–1267.
  33. Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334). find
  34. Debajyoti Datta, Jason A. Fries, Michael McKenna, Aurélie Névéol, Vassilina Nikoulina, and Maya Varma. 2021. Challenges in language modelling for biomedicine. find
  35. Alex Campolo, Madelyn Sanfilippo, Meredith Whittaker, and Kate Crawford. 2018. AI now 2017 report. In AI now 2017 symposium and workshop. AI Now Institute at New York University. find
  36. Edition: AI Now 2017 Symposium and Workshop. find
  37. Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting. In Proceedings of the Conference on Fairnes… find
  38. Yang Trista Cao and Hal Daumé III. 2020. Toward Gender-Inclusive Coreference Resolution. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4568–4595, Online. Association for Computational Linguistics.
  39. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646. find
  40. Daniel de Vassimon Manela, David Errington, Thomas Fisher, Boris van Breugel, and Pasquale Minervini. 2021. Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language Models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational L… find
  41. Alan Chan, Chinasa T. Okolo, Zachary Terner, and Angelina Wang. 2021. The Limits of Global Inclusion in AI Development. arXiv:2102.01265 [cs]. ArXiv: 2102.01265. — [arXiv:2102.01265](https://arxiv.org/abs/2102.01265) find
  42. John Chen, Ian Berlot-Attwell, Xindi Wang, Safwan Hossain, and Frank Rudzicz. 2020. Exploring text specific and blackbox fairness algorithms in multimodal clinical NLP. In Proceedings of the 3rd Clinical Natural Language Processing Workshop, pages 301–312, Online. Association for Computational Lingu… find
  43. Sunipa Dev, Tao Li, Jeff M. Phillips, and Vivek Srikumar. 2020. On measuring and mitigating biased inferences of word embeddings. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7659– 7666. find
  44. Sunipa Dev, Masoud Monajatipoor, Anaelia Ovalle, Arjun Subramonian, Jeff Phillips, and Kai-Wei Chang. 2021a. Harms of gender exclusivity and challenges in non-binary representation in language technologies. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pa…
  45. Sorelle A. Friedler, Carlos Scheidegger, and Suresh Venkatasubramanian. 2021. The (Im)possibility of fairness: different value systems require different mechanisms for fair decision making. Communications of the ACM, 64(4):136–143. find
  46. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv:2101.00027 [cs]. ArXiv: 2101.00027. find
  47. Sunipa Dev, Emily Sheng, Jieyu Zhao, Jiao Sun, Yu Hou, Mattie Sanseverino, Jiin Kim, Nanyun Peng, and Kai-Wei Chang. 2021b. What do bias measures measure? find
  48. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pa…
  49. Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 862–872… find
  50. Catherine D’Ignazio and Lauren F. Klein. 2020. Data feminism. Strong ideas series. The MIT Press, Cambridge, Massachusetts. find
  51. Aparna Garimella, Akhash Amarnath, Kiran Kumar, Akash Pramod Yalla, Anandhavelu N, Niyati Chhaya, and Balaji Vasan Srinivasan. 2021. He is very intelligent, she is very beautiful? On Mitigating Social Biases in Language Modelling and Generation. In Findings of the Association for Computational Lingu… find
  52. Virginia Dignum. 2017. Responsible artificial intelligence: Designing ai for human values. ICT Discoveries. find
  53. Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language P… find
  54. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2018. Datasheets for Datasets. arXiv:1803.09010 [cs]. ArXiv: 1803.09010. find
  55. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, Online. Association for Computational Linguisti…
  56. Mary Douglas. 1978. Purity and danger: an analysis of the concepts of pollution and taboo, repr edition. Routledge, London. OCLC: 248038797. find
  57. Jonathan Dunn. 2020. Mapping languages: the Corpus of Global Language Use. Language Resources and Evaluation, 54(4):999–1018. find
  58. Lisa Gitelman. 2013. Raw data is an oxymoron. MIT press. find
  59. Joel Escudé Font and Marta R. Costa-jussà. 2019. Equalizing gender bias in neural machine translation with word embeddings techniques. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 147–154, Florence, Italy. Association for Computational Linguistics. find
  60. Hila Gonen and Yoav Goldberg. 2019. Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them. In Proceedings of the 2019 Conference of the North, pages 609–614, Minneapolis, Minnesota. Association for Computational Linguistics.
  61. Anjalie Field, Su Lin Blodgett, Zeerak Waseem, and Yulia Tsvetkov. 2021. A Survey of Race, Racism, and Anti-Racism in NLP. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1:… find
  62. Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, and Meredith Ringel Morris. 2019. Toward fairness in ai for people with disabilities: A research roadmap. find
  63. Wei Guo and Aylin Caliskan. 2021. Detecting Emergent Intersectional Biases: Contextualized Word Embeddings Contain a Distribution of Human-like Biases. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 122–133, Virtual Event USA. ACM. find
  64. Nancy Fraser. 1990. Rethinking the Public Sphere: A Contribution to the Critique of Actually Existing Democracy. Social Text, (25/26):56. find
  65. Suchin Gururangan, Dallas Card, Sarah K. Dreier, Emily K. Gade, Leroy Z. Wang, Zeyu Wang, Luke Zettlemoyer, and Noah A. Smith. 2022. Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection. arXiv:2201.10474 [cs]. ArXiv: 2201.10474. of Machine Learning Research, pa… find
  66. Erin M. Kerrison, Jennifer Cobbina, and Kimberly Bender. 2018. “Your Pants Won’t Save You”: Why Black Youth Challenge Race-Based Police Surveillance and the Demands of Black Respectability Politics. Race and Justice, 8(1):7–26. find
  67. Donna Haraway. 1988. Situated Knowledges: The Science Question in Feminism and the Privilege of Partial Perspective. Feminist Studies, 14(3):575–599. find
  68. Os Keyes, Chandler May, and Annabelle Carrell. 2021. You keep using that word: Ways of thinking about gender in computing research. Proc. ACM Hum.-Comput. Interact., 5(CSCW1). find
  69. Dirk Hovy and Shrimai Prabhumoye. 2021. Five sources of bias in natural language processing. Language and Linguistics Compass, 15(8):e12432. find
  70. Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro…
  71. Po Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack W. Rae, Vishal Maini, Dani Yogatama, and Pushmeet Kohli. 2020. Reducing sentiment bias in language models via counterfactual evaluation. In Findings of the Association for Computational Linguistics Findings of ACL: EMNLP 202… find
  72. Ben Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, and Stephen Denuyl. 2020. Social biases in NLP models as barriers for persons with disabilities. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5491–5501, Online. Associ…
  73. Abigail Z. Jacobs and Hanna Wallach. 2021. Measurement and Fairness. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 375–385, Virtual Event Canada. ACM. find
  74. Jigsaw. 2017. Kaggle’s Toxicity Comment Classification competition. find
  75. Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. Measuring Bias in Contextualized Word Representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 166–172, Florence, Italy. Association for Computational Linguistics.
  76. Xisen Jin, Francesco Barbieri, Brendan Kennedy, Aida Mostafazadeh Davani, Leonardo Neves, and Xiang Ren. 2021. On transferability of bias mitigation effects in language model fine-tuning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguis… find
  77. William Labov. 1986. The social stratification of (r) in new york city department stores. In Harold B. Allen and Michael D. Linn, editors, Dialect and Language Variation, pages 304–329. Academic Press, Boston. find
  78. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Comp…
  79. Mandy Lau. 2021. Artificial intelligence language models and the false fantasy of participatory language policies. Working papers in Applied Linguistics and Linguistics at York, 1:4–15. find
  80. Connor Leahy and Stella Biderman. 2021. The hard problem of aligning AI to human values. In The State of AI Ethics Report (Volume 4). The Montreal AI Ethics Institute. find
  81. Nikhil Kandpal, Eric Wallace, and Colin Raffel. 2022. Deduplicating Training Data Mitigates Privacy Risks in Language Models. arXiv:2202.06539 [cs]. ArXiv: 2202.06539.
  82. Josh Lepawsky. 2019. No insides on the outsides. Discard Studies, 0(0). find
  83. Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. 2018. Preventing fairness gerrymandering: Auditing and learning for subgroup fairness. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings Shahar Levy, Koren Lazar, and Gabriel Stanovsky. 2021.… find
  84. Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2021. Towards Understanding and Mitigating Social Biases in Language Models. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6565–6576. PM…
  85. Mike Monteiro and Vivianne Castillo. 2019. Ruined by design: how designers destroyed the world, and what we can do to fix it. Mule Design, Fresno. find
  86. Q. Vera Liao and Michael Muller. 2019. Enabling value sensitive ai systems through participatory design fictions. find
  87. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953–1967, Online. Association for…
  88. Moin Nadeem, Anna Bethke, and Siva Reddy. 2020. Stereoset: Measuring stereotypical bias in pretrained language models.
  89. Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. TruthfulQA: Measuring How Models Mimic Human Falsehoods. arXiv:2109.07958 [cs]. ArXiv: 2109.07958.
  90. Wilhelmina Nekoto, Vukosi Marivate, Tshinondiwa Matsila, Timi Fasubaa, Taiwo Fagbohungbe, Solomon Oluwole Akinola, Shamsuddeen Muhammad, Salomon Kabongo Kabenamualu, Salomey Osei, Freshia Sackey, Rubungo Andre Niyongabo, Ricky Macharm, Perez Ogayo, Orevaoghene Ahia, Musie Meressa Berhe, Mofetoluwa A…
  91. Nishtha Madaan, Inkit Padhi, Naveen Panwar, and Diptikalyan Saha. 2021. Generate your counterfactuals: Towards controlled counterfactual generation for text. Proceedings of the AAAI Conference on Artificial Intelligence, 35(15):13516–13524. find
  92. Liam Magee, Lida Ghahremanlou, Karen Soldatic, and Shanthi Robertson. 2021. Intersectional Bias in Causal Language Models. arXiv:2107.07691 [cs]. ArXiv: 2107.07691. — [arXiv:2107.07691](https://arxiv.org/abs/2107.07691) find
  93. Vijit Malik, Sunipa Dev, Akihiro Nishi, Nanyun Peng, and Kai-Wei Chang. 2021. Socially aware bias measurements for hindi language representations. find
  94. Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and…
  95. Angelina McMillan-Major, Zaid Alyafeai, Stella Biderman, Kimbo Chen, Francesco De Toni, Gérard Dupont, Hady Elsahar, Chris Emezue, Alham Fikri Aji, Suzana Ilić, Nurulaqilla Khamis, Colin Leong, Maraim Masoud, Aitor Soroa, Pedro Ortiz Suarez, Zeerak Talat, Daniel van Strien, and Yacine Jernite. 2022… — [arXiv:2201.10066](https://arxiv.org/abs/2201.10066) find
  96. Safiya Umoja Noble. 2018. Algorithms of oppression: how search engines reinforce racism. New York University Press, New York. find
  97. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R. Bowman. 2021. Bbq: A hand-built bias benchmark for question answering.
  98. Inioluwa Deborah Raji, Emily M. Bender, Amandalynne Paullada, Emily Denton, and Alex Hanna. 2021. AI and the Everything in the Whole Wide World Benchmark. arXiv:2111.15366 [cs]. ArXiv: 2111.15366. find
  99. Milagros Miceli, Julian Posada, and Tianling Yang. 2022. Studying Up Machine Learning Data: Why Talk About Bias When We Mean Power? Proceedings of the ACM on Human-Computer Interaction, 6(GROUP):1–14. find
  100. Inioluwa Deborah Raji and Joy Buolamwini. 2019. Actionable Auditing: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 429–435, Honolulu HI USA. ACM. find
  101. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model Cards for Model Reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT* ’19, pages 220–229, Atlant… find
  102. Micah Rajunov and Scott Duane. 2019. Nonbinary: Memoirs of Gender and Identity. Columbia University Press. find
  103. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 8–1…
  104. Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwa…
  105. Nithya Sambasivan, Erin Arnesen, Ben Hutchinson, Tulsee Doshi, and Vinodkumar Prabhakaran. 2021. Re-imagining Algorithmic Fairness in India and Beyond. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 315–328, Virtual Event Canada. ACM. find
  106. Dean Spade. 2015. Normal Life: Administrative Violence, Critical Trans Politics, and the Limits of Law. Duke University Press. find
  107. Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M. Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti D…
  108. Karolina Stanczak and Isabelle Augenstein. 2021. A Survey on Gender Bias in Natural Language Processing. arXiv:2112.14168 [cs]. ArXiv: 2112.14168. — [arXiv:2112.14168](https://arxiv.org/abs/2112.14168) find
  109. Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1679–1684, Florence, Italy. Association for Computational Linguistics.
  110. Arjun Subramonian. 2021. Allennlp: Fairness and bias mitigation. find
  111. Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2021. Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection. arXiv:2111.07997 [cs]. ArXiv: 2111.07997.
  112. Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating Gender Bias in Natural Language Processing: Literature Review. In Proceedings of the 57th Annual Meeting of the Association for Computati…
  113. Daniel Schiff. 2020. Principles to Practices for Responsible AI: Closing the Gap. 2020 European Conference on AI Workshop on Advancing Towards the SDGs. find
  114. Zeerak Talat, Smarika Lulz, Joachim Bingel, and Isabelle Augenstein. 2021. Disembodied Machine Learning: On the Illusion of Objectivity in NLP. ArXiv: 2101.11974. — [arXiv:2101.11974](https://arxiv.org/abs/2101.11974) find
  115. Shanya Sharma, Manan Dey, and Koustuv Sinha. 2021. Evaluating gender bias in natural language inference. find
  116. Samson Tan, Shafiq Joty, Min-Yen Kan, and Richard Socher. 2020. It’s morphin’ time! Combating linguistic discrimination with inflectional perturbations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2920– 2935, Online. Association for Computational…
  117. Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. 2021. Societal Biases in Language Generation: Progress and Challenges. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (V…
  118. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzho…
  119. Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. 2022. LaMD…
  120. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  121. Nanna Thylstrup and Zeerak Talat. 2020. Detecting ‘Dirt’ and ‘Toxicity’: Rethinking Content Moderation as Pollution Behaviour. SSRN Electronic Journal. find
  122. Pei Zhou, Weijia Shi, Jieyu Zhao, Kuan-Hao Huang, Muhao Chen, Ryan Cotterell, and Kai-Wei Chang. 2019. Examining Gender Bias in Languages with Grammatical Gender. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on N… find
  123. Nenad Tomasev, Kevin R. McKee, Jackie Kay, and Shakir Mohamed. 2021. Fairness for Unobserved Characteristics: Insights from Technological Impacts on Queer Communities, page 254–265. Association for Computing Machinery, New York, NY, USA. find
  124. Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019. Multilingual is not enough: BERT for Finnish. arXiv:1912.07076 [cs]. ArXiv: 1912.07076.
  125. Ran Zmigrod, Sabrina Mielke, Hanna Wallach, and Ryan Cotterell. 2019. Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1651–1661, Florence, Italy. As…
  126. Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing NLP. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proce… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/talat-2022-you-reap-what-you-sow/