Elamur Research
☰
home / papers / 2019 / A Survey on Bias and Fairness in Machine Learning

A Survey on Bias and Fairness in Machine Learning

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1908.09635

Summary

Machine learning now informs high-stakes choices such as lending, hiring and pretrial detention, but these systems can systematically disadvantage particular groups. This survey by Mehrabi and colleagues at USC-ISI brings order to the scattered literature on algorithmic bias and fairness. It opens with documented failures. The most prominent is COMPAS, a US recidivism-risk tool whose false positives fell disproportionately on African-American defendants. It also points to auditing toolkits that measure such disparities. The authors then classify kinds of bias by where they enter a cycle connecting data, algorithms and user interaction, and they show how skewed outputs can feed back into future training data. They also build a taxonomy of formal fairness definitions. Finally, they review how unfairness and its remedies appear across general ML, deep learning, representation learning and NLP, and they close with open research directions. An appendix catalogs commonly used benchmark datasets. The survey is valuable as a shared reference: it gives practitioners vocabulary and a map for finding where unfairness originates before they deploy sensitive systems.

Key points

  • COMPAS relies on 137 features. A cited study found it no more accurate than non-expert people who saw only 7 of those features, or than a plain logistic regression.
  • The bias categories are arranged around the data → algorithm → user loop. Examples include measurement bias, aggregation bias, longitudinal data fallacy, ranking bias, emergent bias, behavioral bias and content production bias.
  • The survey highlights two fairness-auditing toolkits: Aequitas, which produces subgroup bias reports, and IBM's AI Fairness 360 (AIF360).
  • The appendix datasets include:
    • UCI Adult: 48,842 income records
    • German Credit: 1,000 records
    • COMPAS: 18,610 records from Broward County, 2013–2014
    • Communities and Crime: 1,994 records
    • Recidivism in Juvenile Justice: 4,753 records from Catalonia
  • The face and language benchmarks are:
    • WinoBias: 3,160 sentences covering 40 occupations, used for coreference bias
    • Pilot Parliaments Benchmark (PPB): 1,270 faces of parliamentarians from three European and three African countries
    • Diversity in Faces (DiF): about one million face annotations

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

180 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 172 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1908.09635 · arXiv 1908.09635

@misc{mehrabi2019survey,
  title = {{A Survey on Bias and Fairness in Machine Learning}},
  author = {Mehrabi, Ninareh and Morstatter, Fred and Saxena, Nripsuta and Lerman, Kristina and Galstyan, Aram},
  year = {2019},
  doi = {10.48550/arxiv.1908.09635},
  eprint = {1908.09635},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1908.09635}
}

References

170 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alekh Agarwal, Miroslav Dudik, and Zhiwei Steven Wu. 2019. Fair Regression: Quantitative Definitions and Reduction-Based Algorithms. In International Conference on Machine Learning. 120–129. find
  2. Sina Aghaei, Mohammad Javad Azizi, and Phebe Vayanos. 2019. Learning optimal and fair decision trees for non-discriminative decision-making. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 1418–1426. find
  3. Nazanin Alipourfard, Peter G Fennell, and Kristina Lerman. 2018. Can you Trust the Trend?: Discovering Simpson’s Paradoxes in Social Data. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining. ACM, 19–27. find
  4. Nazanin Alipourfard, Peter G Fennell, and Kristina Lerman. 2018. Using Simpson’s Paradox to Discover Interesting Patterns in Behavioral Data. In Twelfth International AAAI Conference on Web and Social Media. find
  5. Alexander Amini, Ava Soleimany, Wilko Schwarting, Sangeeta Bhatia, and Daniela Rus. 2019. Uncovering and Mitigating Algorithmic Bias through Learned Latent Structure. (2019). find
  6. Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine Bias: there’s software used across the country to predict future criminals. And it’s biased against blacks. ProPublica 2016. find
  7. A. Asuncion and D.J. Newman. 2007. UCI Machine Learning Repository. http://www.ics.uci.edu/$\\sim$mlearn/{MLR}epository.html ↗
  8. Arturs Backurs, Piotr Indyk, Krzysztof Onak, Baruch Schieber, Ali Vakilian, and Tal Wagner. 2019. Scalable Fair Clustering. In Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 97), Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.).… find
  9. Ricardo Baeza-Yates. 2018. Bias on the Web. Commun. ACM 61, 6 (May 2018), 54–61. https://doi.org/10.1145/3209581 — [doi:10.1145/3209581](https://doi.org/10.1145/3209581) ↗
  10. Samuel Barbosa, Dan Cosley, Amit Sharma, and Roberto M. Cesar-Jr. 2016. Averaging Gone Wrong: Using Time-Aware Analyses to Better Understand Behavior. (April 2016), 829–841. find
  11. Rachel KE Bellamy, Kuntal Dey, Michael Hind, Samuel C Hoffman, Stephanie Houde, Kalapriya Kannan, Pranay Lohia, Jacquelyn Martino, Sameep Mehta, Aleksandra Mojsilovic, et al. 2018. Ai fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias. arXiv pr… — [arXiv:1810.01943](https://arxiv.org/abs/1810.01943) find
  12. Emily M. Bender and Batya Friedman. 2018. Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science. Transactions of the Association for Computational Linguistics 6 (2018), 587–604. https://doi.org/10.1162/tacl\_a\_00041 ↗
  13. Misha Benjamin, Paul Gagnon, Negar Rostamzadeh, Chris Pal, Yoshua Bengio, and Alex Shee. [n.d.]. TOWARDS STANDARDIZATION OF DATA LICENSES: THE MONTREAL DATA LICENSE. ([n. d.]). find
  14. Richard Berk, Hoda Heidari, Shahin Jabbari, Matthew Joseph, Michael Kearns, Jamie Morgenstern, Seth Neel, and Aaron Roth. 2017. A Convex Framework for Fair Regression. arXiv:1706.02409 [cs.LG] — [arXiv:1706.02409](https://arxiv.org/abs/1706.02409) find
  15. Richard Berk, Hoda Heidari, Shahin Jabbari, Michael Kearns, and Aaron Roth. [n.d.]. Fairness in criminal justice risk assessments: The state of the art. Sociological Methods & Research ([n. d.]), 0049124118782533. find
  16. Peter J Bickel, Eugene A Hammel, and J William O’Connell. 1975. Sex bias in graduate admissions: Data from Berkeley. Science 187, 4175 (1975), 398–404. find
  17. RDP Binns. 2018. Fairness in machine learning: Lessons from political philosophy. Journal of Machine Learning Research (2018). find
  18. Colin R Blyth. 1972. On Simpson’s paradox and the sure-thing principle. J. Amer. Statist. Assoc. 67, 338 (1972), 364–366. find
  19. Miranda Bogen and Aaron Rieke. 2018. Help wanted: an examination of hiring algorithms, equity. Technical Report. and bias. Technical report, Upturn. find
  20. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in neural information processing systems. 4349–4357. find our card
  21. Shikha Bordia and Samuel Bowman. 2019. Identifying and Reducing Gender Bias in Word-Level Language Models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop. 7–15. find our card
  22. Avishek Bose and William Hamilton. 2019. Compositional Fairness Constraints for Graph Embeddings. In International Conference on Machine Learning. 715–724. find
  23. Marc-Etienne Brunet, Colleen Alkalay-Houlihan, Ashton Anderson, and Richard Zemel. 2019. Understanding the Origins of Bias in Word Embeddings. In Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 97), Kamalika Chaudhuri and Ruslan Sa… find
  24. Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research, Vol. 81), Sorelle A. Friedler and Christo Wilson… find
  25. Toon Calders and Sicco Verwer. 2010. Three naive Bayes approaches for discrimination-free classification. Data Mining and Knowledge Discovery 21, 2 (2010), 277–292. find
  26. Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science 356, 6334 (2017), 183–186. find
  27. Flavio Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ramamurthy, and Kush R Varshney. 2017. Optimized Pre-Processing for Discrimination Prevention. In Advances in Neural Information Processing Systems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and… find
  28. Manel Capdevila, Marta Ferrer, and Eulália Luque. 2005. La reincidencia en el delito en la justicia de menores. Centro de estudios jurídicos y formación especializada, Generalitat de Catalunya. Documento no publicado (2005). find
  29. Allison JB Chaney, Brandon M Stewart, and Barbara E Engelhardt. 2018. How algorithmic confounding in recommendation systems increases homogeneity and decreases utility. In Proceedings of the 12th ACM Conference on Recommender Systems. ACM, 224–232. find
  30. Jiahao Chen, Nathan Kallus, Xiaojie Mao, Geoffry Svacha, and Madeleine Udell. 2019. Fairness Under Unawareness: Assessing Disparity When Protected Class Is Unobserved. In Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM, 339–348. find
  31. Xingyu Chen, Brandon Fain, Liang Lyu, and Kamesh Munagala. 2019. Proportionally Fair Clustering. In International Conference on Machine Learning. 1032–1041. find
  32. S. Chiappa. 2019. Path-specific Counterfactual Fairness. In Thirty-Third AAAI Conference on Artificial Intelligence. 7801–7808. find
  33. S. Chiappa and W. S. Isaac. 2019. A Causal Bayesian Networks Viewpoint on Fairness. In E. Kosta, J. Pierson, D. Slamanig, S. Fischer-Hübner, S. Krenn (eds) Privacy and Identity Management. Fairness, Accountability, and Transparency in the Age of Big Data. Privacy and Identity 2018. IFIP Advances in… find
  34. Alexandra Chouldechova. 2017. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data 5, 2 (2017), 153–163. find
  35. Alexandra Chouldechova, Diana Benavides-Prado, Oleksandr Fialko, and Rhema Vaithianathan. 2018. A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Mach… find
  36. Alexandra Chouldechova and Aaron Roth. 2018. The frontiers of fairness in machine learning. arXiv preprint arXiv:1810.08810 (2018). — [arXiv:1810.08810](https://arxiv.org/abs/1810.08810) find
  37. John S Chuang, Olivier Rivoire, and Stanislas Leibler. 2009. Simpson’s paradox in a synthetic microbial system. Science 323, 5911 (2009), 272–275. find
  38. Kevin A Clarke. 2005. The phantom menace: Omitted variable bias in econometric research. Conflict management and peace science 22, 4 (2005), 341–352. find
  39. Lee Cohen, Zachary C. Lipton, and Yishay Mansour. 2019. Efficient candidate screening under multiple tests and implications for fairness. arXiv:1905.11361 [cs.LG] — [arXiv:1905.11361](https://arxiv.org/abs/1905.11361) find
  40. United States. Equal Employment Opportunity Commission. [n.d.]. EEOC compliance manual. [Washington, D.C.] : U.S. Equal Employment Opportunity Commission, [1992]. find
  41. Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. 2017. Algorithmic decision making and the cost of fairness. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 797–806. find
  42. Elliot Creager, David Madras, Joern-Henrik Jacobsen, Marissa Weis, Kevin Swersky, Toniann Pitassi, and Richard Zemel. 2019. Flexibly Fair Representation Learning by Disentanglement. In International Conference on Machine Learning. 1436–1445. find
  43. Brian d’Alessandro, Cathy O’Neil, and Tom LaGatta. 2017. Conscientious classification: A data scientist’s guide to discrimination-aware classification. Big data 5, 2 (2017), 120–134. find
  44. David Danks and Alex John London. 2017. Algorithmic Bias in Autonomous Systems.. In IJCAI. 4691–4697. find
  45. Shai Danziger, Jonathan Levav, and Liora Avnaim-Pesso. 2011. Extraneous factors in judicial decisions. Proceedings of the National Academy of Sciences 108, 17 (2011), 6889–6892. find
  46. Julia Dressel and Hany Farid. 2018. The accuracy, fairness, and limits of predicting recidivism. Science Advances 4, 1 (2018). https://doi.org/10.1126/sciadv.aao5580 arXiv:https://advances.sciencemag.org/content/4/1/eaao5580.full.pdf — [doi:10.1126/sciadv.aao5580](https://doi.org/10.1126/sciadv.aao5580) ↗
  47. Dheeru Dua and Casey Graff. 2017. UCI Machine Learning Repository. http://archive.ics.uci.edu/ml ↗
  48. Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012. Fairness Through Awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference (Cambridge, Massachusetts) (ITCS ’12). ACM, New York, NY, USA, 214–226. https://doi.org/10.1145/2090236.2… — [doi:10.1145/2090236.2090255](https://doi.org/10.1145/2090236.2090255) ↗
  49. Cynthia Dwork, Nicole Immorlica, Adam Tauman Kalai, and Max Leiserson. 2018. Decoupled Classifiers for Group-Fair and Efficient Machine Learning. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research, Vol. 81), Sorelle A. Friedler… find
  50. Golnoosh Farnadi, Behrouz Babaki, and Lise Getoor. 2018. Fairness in Relational Domains. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society (New Orleans, LA, USA) (AIES ’18). ACM, New York, NY, USA, 108–114. https://doi.org/10.1145/3278721.3278733 — [doi:10.1145/3278721.3278733](https://doi.org/10.1145/3278721.3278733) ↗
  51. Michael Feldman, Sorelle A. Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. 2015. Certifying and Removing Disparate Impact. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (Sydney, NSW, Australia) (KDD ’15). Association… — [doi:10.1145/2783258.2783311](https://doi.org/10.1145/2783258.2783311) find
  52. Joel Escudé Font and Marta R Costa-jussà. 2019. Equalizing Gender Biases in Neural Machine Translation with Word Embeddings Techniques. arXiv preprint arXiv:1901.03116 (2019). — [arXiv:1901.03116](https://arxiv.org/abs/1901.03116) find
  53. Batya Friedman and Helen Nissenbaum. 1996. Bias in Computer Systems. ACM Trans. Inf. Syst. 14, 3 (July 1996), 330–347. https://doi.org/10.1145/230538.230561 — [doi:10.1145/230538.230561](https://doi.org/10.1145/230538.230561) ↗
  54. Anna Fry, Thomas J Littlejohns, Cathie Sudlow, Nicola Doherty, Ligia Adamska, Tim Sprosen, Rory Collins, and Naomi E Allen. 2017. Comparison of Sociodemographic and Health-Related Characteristics of UK Biobank Participants With Those of the General Population. American Journal of Epidemiology 186, 9… — [doi:10.1093/aje/kwx246](https://doi.org/10.1093/aje/kwx246) find
  55. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. [n.d.]. Datasheets for Datasets. ([n. d.]). find
  56. C. E. Gehlke and Katherine Biehl. 1934. Certain effects of grouping upon the size of the correlation coefficient in census tract material. J. Amer. Statist. Assoc. 29, 185A (1934), 169–170. https://doi.org/10.2307/2277827 — [doi:10.2307/2277827](https://doi.org/10.2307/2277827) ↗
  57. Naman Goel, Mohammad Yaghini, and Boi Faltings. 2018. Non-discriminatory machine learning through convex fairness criteria. In Thirty-Second AAAI Conference on Artificial Intelligence. find
  58. Hila Gonen and Yoav Goldberg. 2019. Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them. arXiv preprint arXiv:1903.03862 (2019). find our card
  59. Sandra González-Bailón, Ning Wang, Alejandro Rivero, Javier Borge-Holthoefer, and Yamir Moreno. 2014. Assessing the bias in samples of large online networks. Social Networks 38 (2014), 16–27. find
  60. Susan T Gooden. 2015. Race and social equity: A nervous area of government. Routledge. find
  61. Nina Grgic-Hlaca, Muhammad Bilal Zafar, Krishna P Gummadi, and Adrian Weller. 2016. The case for process fairness in learning: Feature selection for fair decision making. In NIPS Symposium on Machine Learning and the Law, Vol. 1. 2. find
  62. S. Hajian and J. Domingo-Ferrer. 2013. A Methodology for Direct and Indirect Discrimination Prevention in Data Mining. IEEE Transactions on Knowledge and Data Engineering 25, 7 (July 2013), 1445–1459. https: //doi.org/10.1109/TKDE.2012.72 — [doi:10.1109/tkde.2012.72](https://doi.org/10.1109/tkde.2012.72) find
  63. Moritz Hardt, Eric Price, Nati Srebro, et al. 2016. Equality of opportunity in supervised learning. In Advances in neural information processing systems. 3315–3323. find our card
  64. Eszter Hargittai. 2007. Whose Space? Differences among Users and Non-Users of Social Network Sites. Journal of Computer-Mediated Communication 13, 1 (10 2007), 276–297. https://doi.org/10.1111/j.1083-6101.2007.00396.x arXiv:http://oup.prod.sis.lan/jcmc/article-pdf/13/1/276/22317170/jjcmcom0276.pdf — [doi:10.1111/j.1083-6101.2007.00396.x](https://doi.org/10.1111/j.1083-6101.2007.00396.x) ↗
  65. Yuzi He, Keith Burghardt, and Kristina Lerman. 2020. A Geometric Solution to Fair Representations. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. 279–285. find
  66. Sarah Holland, Ahmed Hosny, Sarah Newman, Joshua Joseph, and Kasia Chmielinski. 2018. The dataset nutrition label: A framework to drive higher data quality standards. arXiv preprint arXiv:1805.03677 (2018). find
  67. Ayanna Howard and Jason Borenstein. 2018. The ugly truth about ourselves and our robot creations: the problem of bias and social inequity. Science and engineering ethics 24, 5 (2018), 1521–1536. find
  68. Gary B Huang, Marwan Mattar, Tamara Berg, and Eric Learned-Miller. 2008. Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. find
  69. Lingxiao Huang and Nisheeth Vishnoi. 2019. Stable and Fair Classification. In International Conference on Machine Learning. 2879–2890. find
  70. Ben Hutchinson and Margaret Mitchell. 2019. 50 Years of Test (Un) fairness: Lessons for Machine Learning. In Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM, 49–58. find
  71. L. Introna and H. Nissenbaum. 2000. Defining the Web: the politics of search engines. Computer 33, 1 (Jan 2000), 54–62. https://doi.org/10.1109/2.816269 — [doi:10.1109/2.816269](https://doi.org/10.1109/2.816269) ↗
  72. Ayush Jaiswal, Yue Wu, Wael AbdAlmageed, and Premkumar Natarajan. 2018. Unsupervised Adversarial Invariance. arXiv:1809.10083 [cs.LG] — [arXiv:1809.10083](https://arxiv.org/abs/1809.10083) find
  73. Ray Jiang, Aldo Pacchiano, Tom Stepleton, Heinrich Jiang, and Silvia Chiappa. [n.d.]. Wasserstein Fair Classification. ([n. d.]). find
  74. F. Kamiran and T. Calders. 2009. Classifying without discriminating. In 2009 2nd International Conference on Computer, Control and Communication. 1–6. https://doi.org/10.1109/IC4.2009.4909197 — [doi:10.1109/ic4.2009.4909197](https://doi.org/10.1109/ic4.2009.4909197) ↗
  75. Faisal Kamiran and Toon Calders. 2010. Classification with no discrimination by preferential sampling. In Proc. 19th Machine Learning Conf. Belgium and The Netherlands. Citeseer, 1–6. find
  76. Faisal Kamiran and Toon Calders. 2012. Data preprocessing techniques for classification without discrimination. Knowledge and Information Systems 33, 1 (01 Oct 2012), 1–33. https://doi.org/10.1007/s10115-011-0463-8 — [doi:10.1007/s10115-011-0463-8](https://doi.org/10.1007/s10115-011-0463-8) ↗
  77. Faisal Kamiran and Indrė Žliobaitė. 2013. Explainable and Non-explainable Discrimination in Classification. Springer Berlin Heidelberg, Berlin, Heidelberg, 155–170. https://doi.org/10.1007/978-3-642-30487-3\_8 — [doi:10.1007/978-3-642-30487-3\_8](https://doi.org/10.1007/978-3-642-30487-3_8) ↗
  78. Toshihiro Kamishima, Shotaro Akaho, Hideki Asoh, and Jun Sakuma. 2012. Fairness-aware classifier with prejudice remover regularizer. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 35–50. find
  79. Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. 2018. Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness. In International Conference on Machine Learning. 2569–2577. find
  80. Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. 2019. An empirical study of rich subgroup fairness for machine learning. In Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM, 100–109. find
  81. Rogier Kievit, Willem Eduard Frankenhuis, Lourens Waldorp, and Denny Borsboom. 2013. Simpson’s paradox in psychological science: a practical guide. Frontiers in psychology 4 (2013), 513. find
  82. Niki Kilbertus, Mateo Rojas Carulla, Giambattista Parascandolo, Moritz Hardt, Dominik Janzing, and Bernhard Schölkopf. 2017. Avoiding discrimination through causal reasoning. In Advances in Neural Information Processing Systems. 656–666. find
  83. Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. 2016. Inherent trade-offs in the fair determination of risk scores. arXiv preprint arXiv:1609.05807 (2016). — [arXiv:1609.05807](https://arxiv.org/abs/1609.05807) find
  84. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In MT summit, Vol. 5. 79–86. find
  85. Emmanouil Krasanakis, Eleftherios Spyromitros-Xioufis, Symeon Papadopoulos, and Yiannis Kompatsiaris. 2018. Adaptive Sensitive Reweighting to Mitigate Bias in Fairness-aware Classification. In Proceedings of the 2018 World Wide Web Conference (Lyon, France) (WWW ’18). International World Wide Web Co… — [doi:10.1145/3178876.3186133](https://doi.org/10.1145/3178876.3186133) find
  86. Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Andreas Veit, et al. 2017. Openimages: A public dataset for large-scale multi-label and multi-class image classification. Dataset available from https://github. co… find
  87. Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. 2017. Counterfactual Fairness. In Advances in Neural Information Processing Systems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). Curran Associates, Inc., 4066–4076. http://papers… find
  88. Anja Lambrecht and Catherine E Tucker. 2018. Algorithmic bias? An empirical study into apparent gender-based discrimination in the display of STEM career ads. An Empirical Study into Apparent Gender-Based Discrimination in the Display of STEM Career Ads (March 9, 2018) (2018). find
  89. J Larson, S Mattu, L Kirchner, and J Angwin. 2016. Compas analysis. GitHub, available at: https://github. com/propublica/compas-analysis[Google Scholar] (2016). find
  90. Blake Lemoine, Brian Zhang, and M Mitchell. 2018. Mitigating Unwanted Biases with Adversarial Learning. (2018). find
  91. Kristina Lerman. 2018. Computational social scientist beware: Simpson’s paradox in behavioral data. Journal of Computational Social Science 1, 1 (2018), 49–58. find
  92. Kristina Lerman and Tad Hogg. 2014. Leveraging position bias to improve peer recommendation. PLoS One 9, 6 (2014), e98914. http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0098914 ↗
  93. Kristina Lerman and Tad Hogg. 2014. Leveraging position bias to improve peer recommendation. PloS one 9, 6 (2014), e98914. find
  94. Zachary C Lipton, Alexandra Chouldechova, and Julian McAuley. 2017. Does mitigating ML’s disparate impact require disparate treatment? stat 1050 (2017), 19. find
  95. Lydia T Liu, Sarah Dean, Esther Rolf, Max Simchowitz, and Moritz Hardt. 2018. Delayed Impact of Fair Machine Learning. In Proceedings of the 35th International Conference on Machine Learning. find
  96. Joshua R Loftus, Chris Russell, Matt J Kusner, and Ricardo Silva. 2018. Causal reasoning for algorithmic fairness. arXiv preprint arXiv:1805.05859 (2018). — [arXiv:1805.05859](https://arxiv.org/abs/1805.05859) find
  97. Christos Louizos, Kevin Swersky, Yujia Li, Max Welling, and Richard Zemel. 2016. THE VARIATIONAL FAIR AUTOENCODER. stat 1050 (2016), 4. find
  98. Arjun K. Manrai, Birgit H. Funke, Heidi L. Rehm, Morten S. Olesen, Bradley A. Maron, Peter Szolovits, David M. Margulies, Joseph Loscalzo, and Isaac S. Kohane. 2016. Genetic Misdiagnoses and the Potential for Health Disparities. New England Journal of Medicine 375, 7 (2016), 655–665. https://doi.org… — [doi:10.1056/nejmsa1507092](https://doi.org/10.1056/nejmsa1507092) ↗
  99. Ray Marshall. 1974. The economics of racial discrimination: A survey. Journal of Economic Literature 12, 3 (1974), 849–871. find
  100. Chandler May, Alex Wang, Shikha Bordia, Samuel R Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. arXiv preprint arXiv:1903.10561 (2019). find our card
  101. Ninareh Mehrabi, Thamme Gowda, Fred Morstatter, Nanyun Peng, and Aram Galstyan. 2019. Man is to Person as Woman is to Location: Measuring Gender Bias in Named Entity Recognition. arXiv preprint arXiv:1910.10872 (2019). — [arXiv:1910.10872](https://arxiv.org/abs/1910.10872) find
  102. Ninareh Mehrabi, Umang Gupta, Fred Morstatter, Greg Ver Steeg, and Aram Galstyan. 2021. Attributing Fair Decisions with Attention Interventions. arXiv preprint arXiv:2109.03952 (2021). find
  103. Ninareh Mehrabi, Yuzhong Huang, and Fred Morstatter. 2020. Statistical Equity: A Fairness Classification Objective. arXiv preprint arXiv:2005.07293 (2020). — [arXiv:2005.07293](https://arxiv.org/abs/2005.07293) find
  104. Ninareh Mehrabi, Fred Morstatter, Nanyun Peng, and Aram Galstyan. 2019. Debiasing Community Detection: The Importance of Lowly-Connected Nodes. arXiv preprint arXiv:1903.08136 (2019). — [arXiv:1903.08136](https://arxiv.org/abs/1903.08136) find
  105. Ninareh Mehrabi, Pei Zhou, Fred Morstatter, Jay Pujara, Xiang Ren, and Aram Galstyan. 2021. Lawyers are Dishonest? Quantifying Representational Harms in Commonsense Knowledge Resources. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computa… find
  106. Aditya Krishna Menon and Robert C Williamson. 2018. The cost of fairness in binary classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research, Vol. 81), Sorelle A. Friedler and Christo Wilson (Eds.). PMLR, New York, NY,… find
  107. Michele Merler, Nalini Ratha, Rogerio S Feris, and John R Smith. 2019. Diversity in Faces. arXiv preprint arXiv:1901.10436 (2019). — [arXiv:1901.10436](https://arxiv.org/abs/1901.10436) find
  108. Hannah Jean Miller, Jacob Thebault-Spieker, Shuo Chang, Isaac Johnson, Loren Terveen, and Brent Hecht. 2016. “Blissfully Happy” or “Ready toFight”: Varying Interpretations of Emoji. In Tenth International AAAI Conference on Web and Social Media. find
  109. I Minchev, G Matijevic, DW Hogg, G Guiglion, M Steinmetz, F Anders, C Chiappini, M Martig, A Queiroz, and C Scannapieco. 2019. Yule-Simpson’s paradox in Galactic Archaeology. arXiv preprint arXiv:1902.01421 (2019). — [arXiv:1902.01421](https://arxiv.org/abs/1902.01421) find
  110. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model Cards for Model Reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency (Atlanta, GA, USA) (FAT* ’19). A… find
  111. Fred Morstatter, Jürgen Pfeffer, Huan Liu, and Kathleen M Carley. 2013. Is the sample good enough? Comparing data from twitter’s streaming API with Twitter’s firehose. In 7th International AAAI Conference on Weblogs and Social Media, ICWSM 2013. AAAI press. find
  112. Daniel Moyer, Shuyang Gao, Rob Brekelmans, Aram Galstyan, and Greg Ver Steeg. 2018. Invariant Representations without Adversarial Training. In Advances in Neural Information Processing Systems. 9084–9093. find
  113. Amitabha Mukerjee, Rita Biswas, Kalyanmoy Deb, and Amrit P Mathur. 2002. Multi–objective evolutionary algorithms for the risk–return trade–off in bank loan management. International Transactions in operational research 9, 5 (2002), 583–597. find
  114. David B Mustard. 2003. Reexamining criminal behavior: the importance of omitted variable bias. Review of Economics and Statistics 85, 1 (2003), 205–211. find
  115. Razieh Nabi, Daniel Malinsky, and Ilya Shpitser. 2018. Learning Optimal Fair Policies. arXiv preprint arXiv:1809.02244 (2018). — [arXiv:1809.02244](https://arxiv.org/abs/1809.02244) find
  116. Razieh Nabi and Ilya Shpitser. 2018. Fair inference on outcomes. In Thirty-Second AAAI Conference on Artificial Intelligence. find
  117. Azadeh Nematzadeh, Giovanni Luca Ciampaglia, Filippo Menczer, and Alessandro Flammini. 2017. How algorithmic popularity bias hinders or promotes quality. arXiv preprint arXiv:1707.00574 (2017). — [arXiv:1707.00574](https://arxiv.org/abs/1707.00574) find
  118. Dong-Phuong Nguyen, Rilana Gravel, Rudolf Berend Trieschnigg, and Theo Meder. 2013. "How old do you think I am?": A study of language and age in Twitter. In Proceedings of the Seventh International AAAI Conference on Weblogs and Social Media, ICWSM 2013. AAAI Press, 439–448. eemcs-eprint-23604. find
  119. Anne O’Keeffe and Michael McCarthy. 2010. The Routledge handbook of corpus linguistics. Routledge. find
  120. Alexandra Olteanu, Carlos Castillo, Fernando Diaz, and Emre Kiciman. 2016. Social data: Biases, methodological pitfalls, and ethical boundaries. (2016). find
  121. Cathy O’Neil. 2016. Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. Crown Publishing Group, New York, NY, USA. find
  122. Luca Oneto, Michele Doninini, Amon Elders, and Massimiliano Pontil. 2019. Taking advantage of multitask learning for fair classification. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. 227–237. find
  123. Osonde A Osoba and William Welser IV. 2017. An intelligence in our image: The risks of bias and errors in artificial intelligence. Rand Corporation. find
  124. Edmund S Phelps. 1972. The statistical theory of racism and sexism. The american economic review 62, 4 (1972), 659–661. find
  125. Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, and Kilian Q Weinberger. 2017. On Fairness and Calibration. In Advances in Neural Information Processing Systems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). Curran Associates, Inc., 568… find
  126. Marcelo OR Prates, Pedro H Avelar, and Luís C Lamb. 2018. Assessing gender bias in machine translation: a case study with Google Translate. Neural Computing and Applications (2018), 1–19. find
  127. Bilal Qureshi, Faisal Kamiran, Asim Karim, and Salvatore Ruggieri. 2016. Causal discrimination discovery through propensity score analysis. arXiv preprint arXiv:1608.03735 (2016). — [arXiv:1608.03735](https://arxiv.org/abs/1608.03735) find
  128. Inioluwa Deborah Raji and Joy Buolamwini. 2019. Actionable auditing: Investigating the impact of publicly naming biased performance results of commercial ai products. find
  129. M Redmond. 2011. Communities and crime unnormalized data set. UCI Machine Learning Repository. In website: http://www. ics. uci. edu/mlearn/MLRepository. html (2011). find
  130. Willy E Rice. 1996. Race, Gender, Redlining, and the Discriminatory Access to Loans, Credit, and Insurance: An Historical and Empirical Analysis of Consumers Who Sued Lenders and Insurers in Federal and State Courts, 1950-1995. San Diego L. Rev. 33 (1996), 583. find
  131. Stephanie K Riegg. 2008. Causal inference and omitted variable bias in financial aid research: Assessing solutions. The Review of Higher Education 31, 3 (2008), 329–354. find
  132. Lauren A Rivera. 2012. Hiring as cultural matching: The case of elite professional service firms. American sociological review 77, 6 (2012), 999–1022. find
  133. Andrea Romei and Salvatore Ruggieri. 2011. A multidisciplinary survey on discrimination analysis. find
  134. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender Bias in Coreference Resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). Associati… find our card
  135. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015. Imagenet large scale visual recognition challenge. International journal of computer vision 115, 3 (2015), 211–252. find our card
  136. Pedro Saleiro, Benedict Kuester, Abby Stevens, Ari Anisfeld, Loren Hinkson, Jesse London, and Rayid Ghani. 2018. Aequitas: A Bias and Fairness Audit Toolkit. arXiv preprint arXiv:1811.05577 (2018). — [arXiv:1811.05577](https://arxiv.org/abs/1811.05577) find
  137. Samira Samadi, Uthaipon Tantipongpipat, Jamie Morgenstern, Mohit Singh, and Santosh Vempala. 2018. The Price of Fair PCA: One Extra Dimension. In Proceedings of the 32Nd International Conference on Neural Information Processing Systems (Montr&\#233;al, Canada) (NIPS’18). Curran Associates Inc., USA,… find
  138. Nripsuta Ani Saxena. 2019. Perceptions of Fairness. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society (Honolulu, HI, USA) (AIES ’19). ACM, New York, NY, USA, 537–538. https://doi.org/10.1145/3306618.3314314 — [doi:10.1145/3306618.3314314](https://doi.org/10.1145/3306618.3314314) ↗
  139. Nripsuta Ani Saxena, Karen Huang, Evan DeFilippis, Goran Radanovic, David C Parkes, and Yang Liu. 2019. How Do Fairness Definitions Fare?: Examining Public Attitudes Towards Algorithmic Definitions of Fairness. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. ACM, 99–106. find
  140. Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. 2016. Recommendations as Treatments: Debiasing Learning and Evaluation. In International Conference on Machine Learning. 1670–1679. find
  141. Andrew D Selbst, Danah Boyd, Sorelle A Friedler, Suresh Venkatasubramanian, and Janet Vertesi. 2019. Fairness and abstraction in sociotechnical systems. In Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM, 59–68. find
  142. Shreya Shankar, Yoni Halpern, Eric Breck, James Atwood, Jimbo Wilson, and D Sculley. 2017. No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing World. stat 1050 (2017), 22. find
  143. Richard Shaw and Manuel Corpas. [n.d.]. Further bias in personal genomics? ([n. d.]). find
  144. Harini Suresh and John V Guttag. 2019. A Framework for Understanding Unintended Consequences of Machine Learning. arXiv preprint arXiv:1901.10002 (2019). — [arXiv:1901.10002](https://arxiv.org/abs/1901.10002) find
  145. Songül Tolan, Marius Miron, Emilia Gómez, and Carlos Castillo. 2019. Why Machine Learning May Lead to Unfairness: Evidence from Risk Assessment for Juvenile Justice in Catalonia. (2019). find
  146. Zeynep Tufekci. 2014. Big questions for social media big data: Representativeness, validity and other methodological pitfalls. In Eighth International AAAI Conference on Weblogs and Social Media. find
  147. Berk Ustun, Yang Liu, and David Parkes. 2019. Fairness without Harm: Decoupled Classifiers with Preference Guarantees. In Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 97), Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR… find
  148. Eva Vanmassenhove, Christian Hardmeier, and Andy Way. 2018. Getting gender right in neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 3003–3008. find our card
  149. Sahil Verma and Julia Rubin. 2018. Fairness definitions explained. In 2018 IEEE/ACM International Workshop on Software Fairness (FairWare). IEEE, 1–7. find
  150. Selwyn Vickers, Mona Fouad, and Moon S Chen Jr. 2014. Enhancing Minority Participation in Clinical Trials (EMPaCT): laying the groundwork for improving minority clinical trial accrual. Cancer 120 (2014), vi–vii. find
  151. Ting Wang and Dashun Wang. 2014. Why Amazon’s ratings might mislead you: The story of herding effects. Big data 2, 4 (2014), 196–204. find
  152. Steven L Willborn. 1984. The disparate impact model of discrimination: Theory and limits. Am. UL Rev. 34 (1984), 799. find
  153. Christo Wilson, Bryce Boe, Alessandra Sala, Krishna PN Puttaswamy, and Ben Y Zhao. 2009. User interactions in social networks and their implications. In Proceedings of the 4th ACM European conference on Computer systems. Acm, 205–218. find
  154. Blake Woodworth, Suriya Gunasekar, Mesrob I Ohannessian, and Nathan Srebro. 2017. Learning non-discriminatory predictors. arXiv preprint arXiv:1702.06081 (2017). — [arXiv:1702.06081](https://arxiv.org/abs/1702.06081) find
  155. Yongkai Wu, Lu Zhang, and Xintao Wu. 2018. Fairness-aware Classification: Criterion, Convexity, and Bounds. arXiv:1809.04737 [cs.LG] — [arXiv:1809.04737](https://arxiv.org/abs/1809.04737) find
  156. Depeng Xu, Shuhan Yuan, Lu Zhang, and Xintao Wu. 2018. Fairgan: Fairness-aware generative adversarial networks. In 2018 IEEE International Conference on Big Data (Big Data). IEEE, 570–575. find
  157. Irene Y Chen, Peter Szolovits, and Marzyeh Ghassemi. 2019. Can AI Help Reduce Disparities in General Medical and Mental Health Care? AMA journal of ethics 21 (02 2019), E167–179. https://doi.org/10.1001/amajethics.2019.167 — [doi:10.1001/amajethics.2019.167](https://doi.org/10.1001/amajethics.2019.167) ↗
  158. Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P Gummadi. 2017. Fairness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment. In Proceedings of the 26th international conference on world wide web. 1171–1180. find
  159. Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P Gummadi. 2015. Fairness constraints: Mechanisms for fair classification. arXiv preprint arXiv:1507.05259 (2015). — [arXiv:1507.05259](https://arxiv.org/abs/1507.05259) find
  160. Lu Zhang and Xintao Wu. 2017. Anti-discrimination learning: a causal modeling-based framework. International Journal of Data Science and Analytics 4, 1 (01 Aug 2017), 1–16. https://doi.org/10.1007/s41060-017-0058-x — [doi:10.1007/s41060-017-0058-x](https://doi.org/10.1007/s41060-017-0058-x) ↗
  161. Lu Zhang, Yongkai Wu, and Xintao Wu. 2016. On Discrimination Discovery Using Causal Networks. In Social, Cultural, and Behavioral Modeling, Kevin S. Xu, David Reitter, Dongwon Lee, and Nathaniel Osgood (Eds.). Springer International Publishing, Cham, 83–93. find
  162. Lu Zhang, Yongkai Wu, and Xintao Wu. 2016. Situation Testing-based Discrimination Discovery: A Causal Inference Approach. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (New York, New York, USA) (IJCAI’16). AAAI Press, 2718–2724. http://dl.acm.org/citati… ↗
  163. Lu Zhang, Yongkai Wu, and Xintao Wu. 2017. Achieving non-discrimination in data release. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 1335–1344. find
  164. Lu Zhang, Yongkai Wu, and Xintao Wu. 2017. A Causal Framework for Discovering and Removing Direct and Indirect Discrimination. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17. 3929–3935. https://doi.org/10.24963/ijcai.2017/549 — [doi:10.24963/ijcai.2017/549](https://doi.org/10.24963/ijcai.2017/549) ↗
  165. L. Zhang, Y. Wu, and X. Wu. 2018. Causal Modeling-Based Discrimination Discovery and Removal: Criteria, Bounds, and Algorithms. IEEE Transactions on Knowledge and Data Engineering (2018), 1–1. https://doi.org/10.1109/TKDE.2018.2872988 — [doi:10.1109/tkde.2018.2872988](https://doi.org/10.1109/tkde.2018.2872988) ↗
  166. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. 2019. Gender Bias in Contextualized Word Embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1… find our card
  167. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. find our card
  168. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods. arXiv:1804.06876 [cs.CL] find our card
  169. Jieyu Zhao, Yichao Zhou, Zeyu Li, Wei Wang, and Kai-Wei Chang. 2018. Learning Gender-Neutral Word Embeddings. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 4847–4853. find our card
  170. James Zou and Londa Schiebinger. 2018. AI can be sexist and racist it’s time to make it fair. Nature Publishing Group. find