Elamur Research⌕
☰
home / papers / 2021 / Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics

Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics

Paper · 2021 · Paula Czarnowska, Yogarshi Vyas and Kashif Shah

Summary

Many metrics have been proposed to measure social bias in NLP models, but it has been unclear how they relate and whether they capture the same thing. The authors reviewed 146 papers on bias in NLP. They show that the extrinsic fairness metrics used in downstream tasks are special cases of three generalized metrics: the Pairwise Comparison Metric (PCM), the Background Comparison Metric (BCM) and the Multi-group Comparison Metric (MCM). Each metric is defined by two parameters: a scoring function φ and a comparison function d. Each also comes in a group-fairness and a counterfactual-fairness version.

They then tested many existing metrics on three transformer models: two for sentiment analysis and one for NER. The models were evaluated across seven sensitive attributes. Different metrics often give different pictures of bias, and the authors trace these differences systematically to the choices of φ and d.

The work matters because it helps practitioners choose metrics on principled grounds rather than inventing near-duplicates. It also comes with released code that is easy to extend.

Key points

  • Survey scope: 146 papers were surveyed. The paper covers only tasks that do not predict a sensitive attribute, because intrinsic (embedding-level) bias reportedly does not correlate with downstream bias.
  • The three generalized metrics:
    • PCM averages the distance between pairs of groups.
    • BCM compares each group against a "background." The background can be all examples, a privileged group, or an unperturbed sentence. A vector-valued variant (VBCM) keeps per-group results instead of averaging them.
    • MCM compares all groups at once to detect any overall effect of an attribute.
  • Experimental setup: The seven attributes examined in the experiments are Race, Religion, Age, Disability, Gender, Nationality and Sexual Orientation. These attributes are protected under US federal anti-discrimination law. The data are English, but the framework does not depend on language or model.
  • Recommendations:
    • Follow a three-step choice process: pick the question and matching generalized metric, then a harm-relevant φ (e.g., FNR for sentiment), then d.
    • Use at least one prediction-based and one probability-based metric.
    • Report all chosen metrics, even when they show no bias.
  • Code: Released at github.com/amazon-research/generalized-fairness-metrics. It is built on CheckList, so CheckList-style tests can run alongside the fairness metrics.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom155 works · 115,562 citations🏢AmazonSeattle · United States70 works · 39,977 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Social Sciences · 744 works Safety Research · 221 works

🧭 Topics 1 Bias, fairness & safety · 209 works

Cite

Cite as Paula Czarnowska, Yogarshi Vyas and Kashif Shah. 2021. Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00425

@article{czarnowska2021quantifying,
  title = {{Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics}},
  author = {Paula Czarnowska, Yogarshi Vyas and Kashif Shah},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.1162/tacl\_a\_00425},
  url = {https://doi.org/10.1162/tacl_a_00425}
}

References

78 references, 33 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pinkesh Badjatiya, Manish Gupta, and Vasudeva Varma. 2019. Stereotypical bias removal for hate speech detection task using knowledge-based generalizations. In The World Wide Web Conference, pages 49–59. https://doi.org/10.1145/3308558.3313504 — [doi:10.1145/3308558.3313504](https://doi.org/10.1145/3308558.3313504) ↗
  2. Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov. 2019. Differential privacy has disparate impact on model accuracy. In Advances in Neural Information Processing Systems, pages 15479–15488. find
  3. David Bamman, Sejal Popat, and Sheng Shen. 2019. An annotated dataset of literary entities. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2138–2144, Minneapoli… our card
  4. Alex Beutel, J. Chen, Zhe Zhao, and Ed Huai hsin Chi. 2017. Data decisions and theoretical implications when adversarially learning fair representations. Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT/ML 2017). find
  5. Alex Beutel, Jilin Chen, Tulsee Doshi, Hai Qian, Allison Woodruff, Christine Luu, Pierre Kreitmann, Jonathan Bischof, and Ed H. Chi. 2019. Putting fairness principles into practice: Challenges, metrics, and improvements. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, page… — [doi:10.1145/3306618](https://doi.org/10.1145/3306618) find
  6. Jayadev Bhaskaran and Isha Bhallamudi. 2019. Good secretaries, bad truck drivers? Occupational gender stereotypes in sentiment analysis. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 62–68, Florence, Italy. Association for Computational Linguistics. https:… find
  7. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of ‘‘bias’’ in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454–5476, Online. Association for Computational Linguistic… our card
  8. Su Lin Blodgett and Brendan T. O’Connor. 2017. Racial Disparity in Natural Language Processing: A case study of social media African-American English. Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT/ML 2017). our card
  9. Su Lin Blodgett, Johnny Wei, and Brendan O’Connor. 2018. Twitter universal dependency parsing for African-American and mainstream American English. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1415–1425, Melbourne, Australi… our card
  10. Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, pages 4356–4364, Red Hook, N… our card
  11. Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2019. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion Proceedings of The 2019 World Wide Web Conference, pages 491–500, San Francisco USA. ACM. https://doi.org /10.1145/… — [doi:10.1145/3308560.3317593](https://doi.org/10.1145/3308560.3317593) ↗
  12. Aylin Caliskan, Joanna Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356:183–186. https://doi.org/10.1126/science.aal4230, PubMed: 28408601 ↗
  13. Yang Trista Cao and Hal Daumé III. 2020. Toward gender-inclusive coreference resolution. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4568–4595, Online. Association for Computational Linguistics. our card
  14. Kaytlin Chaloner and Alfredo Maldonado. 2019. Measuring gender bias in word embeddings across domains and discovering new gender bias word categories. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 25–32, Florence, Italy. Association for Computational Lingu… find
  15. Alexandra Chouldechova and Aaron Roth. 2020. A snapshot of the frontiers of fairness in machine learning. Communications of the ACM, 63(5):82–89. https://doi.org/10.1145/3376898 ↗
  16. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre-training text encoders as discriminators rather than generators. The International Conference on Learning Representations (ICLR). find
  17. Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. 2019. Racial bias in hate speech and abusive language detection datasets. In Proceedings of the Third Workshop on Abusive Language Online, pages 25–35, Florence, Italy. Association for Computational Linguistics. https://doi.org/10.18653/v1/W… ↗ our card
  18. Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In Proceedings of the Conference on Fairnes… — [doi:10.1145/3287560](https://doi.org/10.1145/3287560) find
  19. Sunipa Dev and Jeff Phillips. 2019. Attenuating bias in word vectors. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 879–887. PMLR. find
  20. Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ ACM Conference on AI, Ethics, and Society, AIES ’18, pages 67–73, New York, NY, USA. Association for Computing Machinery.… — [doi:10.1145/3278721](https://doi.org/10.1145/3278721) find
  21. Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, ITCS ’12, pages 214–226, New York, NY, USA. Association for Computing Machinery. https://doi.org/10 .114… find
  22. Lachlan Forrow, William C. Taylor, and Robert M. Arnold. 1992. Absolutely relative: How research results are summarized can affect treatment decisions. The American Journal of Medicine, 92(2):121–124. https://doi .org/10.1016/0002-9343(92)90100-P Sahaj Garg, Vincent Perot, Nicole Limtiaco, Ankur Tal… — [doi:10.1016/0002-9343(92)90100-p](https://doi.org/10.1016/0002-9343%2892%2990100-p) find
  23. Aparna Garimella, Carmen Banea, Dirk Hovy, and Rada Mihalcea. 2019. Women’s syntactic resilience and men’s grammatical luck: Gender-bias in part-of-speech tagging and dependency parsing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3493–3498, Flor… find
  24. Andrew Gaut, Tony Sun, Shirlyn Tang, Yuxin Huang, Jing Qian, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2020. Towards understanding gender bias in relation extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational… find
  25. Oguzhan Gencoglu. 2021. Cyberbullying detection with fairness constraints. IEEE Internet Computing, 25(01):20–29. https://doi.org/10.1109/MIC.2020.3032461 — [doi:10.1109/mic.2020.3032461](https://doi.org/10.1109/mic.2020.3032461) ↗
  26. Seraphina Goldfarb-Tarrant, Rebecca Marchant, Ricardo Muñoz Sanchez, Mugdha Pandya, and Adam Lopez. 2021. Intrinsic bias metrics do not correlate with application bias. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguist… our card
  27. Hila Gonen and Yoav Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol… our card
  28. Hila Gonen and Kellie Webster. 2020. Automatically identifying gender issues in machine translation using perturbations. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1991–1995, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020 .fin… ↗
  29. Rowan Hall Maudslay, Hila Gonen, Ryan Cotterell, and Simone Teufel. 2019. It’s all in the name: Mitigating gender bias with name-based counterfactual data substitution. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferenc… our card
  30. Moritz Hardt, Eric Price, and Nathan Srebro. 2016. Equality of opportunity in supervised learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, pages 3323–3331, Red Hook, NY, USA. Curran Associates Inc. our card
  31. Dirk Hovy and Anders Søgaard. 2015. Tagging performance correlates with author age. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 483–488, Beijing, C… our card
  32. Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack Rae, Vishal Maini, Dani Yogatama, and Pushmeet Kohli. 2020a. Reducing sentiment bias in language models via counterfactual evaluation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 65–83, On… find
  33. Xiaolei Huang, Linzi Xing, Franck Dernoncourt, and Michael J. Paul. 2020b. Multilingual twitter corpus and baselines for evaluating demographic bias in hate speech recognition. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 1440–1448, Marseille, France. European Langu… find
  34. Ben Hutchinson and Margaret Mitchell. 2019. 50 Years of test (un)fairness: Lessons for machine learning. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 49–58. https://doi.org/10.1145/3287560.3287600 and Vibhav Gogate, editors, In Proceedings of The 35th Uncert… — [doi:10.1145/3287560.3287600](https://doi.org/10.1145/3287560.3287600) ↗
  35. Svetlana Kiritchenko and Saif Mohammad. 2018. Examining gender and race bias in two hundred sentiment analysis systems. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 43–53, New Orleans, Louisiana. Association for Computational Linguistics. https://doi.o… ↗ our card
  36. Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W. Black, and Yulia Tsvetkov. 2019. Measuring bias in contextualized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 166–172, Florence, Italy. Association for Computational Linguistics. https… our card
  37. Matt J. Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. 2017. Counterfactual fairness. In Advances in Neural Information Processing Systems, volume 30, pages 4066–4076. Curran Associates, Inc. find
  38. Ben Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, and Stephen Denuyl. 2020. Social biases in NLP models as barriers for persons with disabilities. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5491–5501, Online. Associ… our card
  39. John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML ’01, pages 282–289, San Francisco, CA, USA. Morgan Kaufm… find
  40. Abigail Z. Jacobs, Su Lin Blodgett, Solon Barocas, Hal Daumé, and Hanna Wallach. 2020. The meaning and measurement of bias: Lessons from natural language processing. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, FAT* ’20, page 706, New York, NY, USA. Associati… — [doi:10.1145/3351095.3375671](https://doi.org/10.1145/3351095.3375671) find
  41. Paul Pu Liang, Irene Mengze Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2020. Towards debiasing sentence representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5502–5515, Online. Association for Computa… our card
  42. Ray Jiang, Aldo Pacchiano, Tom Stepleton, Heinrich Jiang, and Silvia Chiappa. 2020. Wasserstein fair classification. find
  43. Ryan P. Adams Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. Version 1. find
  44. H. B. Mann and D. R. Whitney. 1947. On a test of whether one of two random variables is stochastically larger than the other. Ann. Math. Statist., 18(1):50–60. https://doi.org/10.1214/aoms/1177730491 — [doi:10.1214/aoms/1177730491](https://doi.org/10.1214/aoms/1177730491) ↗
  45. Ninareh Mehrabi, Thamme Gowda, Fred Morstatter, Nanyun Peng, and Aram Galstyan. 2020. Man is to person as woman is to location: Measuring gender bias in named entity recognition. In Proceedings of the 31st ACM Conference on Hypertext and Social Media, HT ’20, pages 231–232, New York, NY, USA. Associ… — [doi:10.1145/3372923](https://doi.org/10.1145/3372923) find
  46. Ninareh Mehrabi, Fred Morstatter, N. Saxena, Kristina Lerman, and A. Galstyan. 2019. A survey on bias and fairness in machine learning. CoRR, abs/1908.09635. Version 2. our card
  47. Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. SemEval-2018 Task 1: Affect in Tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 1–17, New Orleans, Louisiana. Association for Computational Linguistics. https://doi.org/10.18… our card
  48. Moin Nadeem, Anna Bethke, and Siva Reddy. 2020. StereoSet: Measuring stereotypical bias in pretrained language models. CoRR, abs/2004 .09456. Version 1. https://doi.org/10 .18653/v1/2021.acl-long.416 our card
  49. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowS-Pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953–1967, Online. Association for… our card
  50. M. Noordzij, M. van Diepen, F. Caskey, and K. Jager. 2017. Relative risk versus absolute risk: One cannot be interpreted without the other: Clinical epidemiology in nephrology. Nephrology Dialysis Transplantation, 32:ii13–ii18. https://doi.org/10.1093/ndt/gfw465, PubMed: 28339913 — [doi:10.1093/ndt/gfw465](https://doi.org/10.1093/ndt/gfw465) ↗
  51. Alexandra Olteanu, Kartik Talamadupula, and Kush R. Varshney. 2017. The limits of abstract evaluation metrics: The case of hate speech detection. In Proceedings of the 2017 ACM on Web Science Conference, pages 405–406. https://doi.org/10.1145/3091478 .3098871 — [doi:10.1145/3091478](https://doi.org/10.1145/3091478) ↗
  52. Orestis Papakyriakopoulos, Simon Hegelich, Juan Carlos Medina Serrano, and Fabienne Marco. 2020. Bias in word embeddings. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, FAT* ’20, pages 446–457, New York, NY, USA. Association for Computing Machinery. https://doi… — [doi:10.1145/3351095.3372843](https://doi.org/10.1145/3351095.3372843) find
  53. Radomir Popović, Florian Lemmerich, and Markus Strohmaier. 2020. Joint multiclass debiasing of word embeddings. In Foundations of Intelligent Systems, pages 79–89, Cham. Springer International Publishing. https://doi.org/10.1007/978-3-030-59491 -6 8 — [doi:10.1007/978-3-030-59491](https://doi.org/10.1007/978-3-030-59491) ↗
  54. Vinodkumar Prabhakaran, Ben Hutchinson, and Margaret Mitchell. 2019. Perturbation sensitivity analysis to detect unintended model biases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processin… find
  55. Flavien Prost, Nithum Thain, and Tolga Bolukbasi. 2019. Debiasing embeddings for reduced gender bias in text classification. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 69–75, Florence, Italy. Association for Computational Linguistics. https://doi .org/1… find
  56. Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009), pages 147–155, Boulder, Colorado. Association for Computational Linguistics. https://doi.org/10 .3115… our card
  57. Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with Checklist. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902–4912, Online. Association for Computational Linguistic… find
  58. Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2021. Gender bias in machine translation. Transactions of the Association for Computational Linguistics. our card
  59. Anthony Rios. 2020. FuzzE: Fuzzy fairness evaluation of offensive language classifiers on African-American English. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 881–889. https://doi.org/10.1609/aaai.v34i01.5434 — [doi:10.1609/aaai.v34i01.5434](https://doi.org/10.1609/aaai.v34i01.5434) ↗
  60. Anthony Rios and Brandon Lwowski. 2020. An empirical study of the downstream reliability of pre-trained word embeddings. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3371–3388. International Committee on Computational Linguistics, Barcelona, Spain (Online).… find
  61. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 8–1… our card
  62. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668–1678, Florence, Italy. Association for Computational Linguistics. our card
  63. Danielle Saunders and Bill Byrne. 2020. Reducing gender bias in neural machine translation as a João Sedoc and Lyle Ungar. 2019. The role of protected class word lists in bias identification of contextualized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Langu… find
  64. Deven Santosh Shah, H. Andrew Schwartz, and Dirk Hovy. 2020. Predictive biases in natural language processing models: A conceptual framework and overview. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5248–5264, Online. Association for Computationa… our card
  65. Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. 2020. Towards controllable biases in language generation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3239–3254, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020 .findi… ↗ our card
  66. Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2021. Societal biases in language generation: Progress and challenges. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics. our card
  67. Seungjae Shin, Kyungwoo Song, JoonHo Jang, Hyemi Kim, Weonyoung Joo, and Il-Chul Moon. 2020. Neutralizing gender bias in word embeddings with latent disentanglement and counterfactual generation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3126–3140, Online. find
  68. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020 .findings-emnlp.280 Association for Computational Linguistics. https://doi.org/10.18653/v1/P19 -1159 — [doi:10.18653/v1/2020](https://doi.org/10.18653/v1/2020) ↗
  69. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–… find
  70. Samson Tan, Shafiq Joty, Min-Yen Kan, and Richard Socher. 2020. It’s morphin’ time! Combating linguistic discrimination with inflectional perturbations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2920–2935, Online. Association for Computational… our card
  71. Artūrs Stafanovičs, Toms Bergmanis, and Mārcis Pinnis. 2020. Mitigating gender bias in machine translation with target gender annotations. In Proceedings of the 5th Conference on Machine Translation (WMT), pages 629–638. Association for Computational Linguistics. find
  72. Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating Gender Bias in Machine Translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1679–1684, Florence, Italy. Association for Computational Linguistics. https://doi.org/10 .18… our card
  73. Jacob Stegenga. 2015. Measuring effectiveness. Studies in History and Philosophy of Science Part C: Studies in History and Philosophy of Biological and Biomedical Sciences, 54:62–71. https://doi.org/10.1016/j.shpsc .2015.06.003, PubMed: 26199055 — [doi:10.1016/j.shpsc](https://doi.org/10.1016/j.shpsc) ↗
  74. Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating gender bias in natural language processing: Literature review. In Proceedings of the 57th Annual Meeting of the Association for Computati… our card
  75. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 Shared Task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142–147. https://doi.org/10.3115/1119176.1119195 ↗ our card
  76. Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018. Mind the GAP: A balanced corpus of gendered ambiguous pronouns. Transactions of the Association for Computational Linguistics, 6:605–617. https://doi.org/10.1162/tacl a 00240 ↗ our card
  77. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… our card
  78. A. Zhiltsova, S. Caton, and Catherine Mulway. 2019. Mitigation of unintended biases against non-native english texts in sentiment analysis. In Proceedings for the 27th AIAI Irish Conference on Artificial Intelligence and Cognitive Science. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/czarnowska-2021-quantifying-social-biases-nlp-generalization/