Elamur Research
☰
home / papers / 2020 / Social Biases in NLP Models as Barriers for Persons with Disabilities

Social Biases in NLP Models as Barriers for Persons with Disabilities

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.acl-main.487.pdf

Summary

The authors ask whether widely used English NLP systems treat mentions of disability unfairly. They compiled 56 English expressions for people with various disabilities, grouped them by disability type, and labeled each as Recommended or Non-Recommended according to guidelines from three US organizations. They then took naturally occurring sentences, swapped a pronoun for each disability phrase, and measured how the scores of a toxicity classifier and a sentiment classifier changed. They also probed BERT with fill-in-the-blank templates and rated the sentiment of its suggested words. Finally, they analyzed which terms co-occur with mental-illness mentions in a public comment dataset. Every disability category raised toxicity scores to some degree, and non-recommended phrasings had a much larger effect. BERT's completions skewed negative for most disability phrases. In the data, mental illness was linked to homelessness, guns and drugs. These findings matter because content-moderation tools could disproportionately suppress or delay ordinary posts about disability, which would reduce the visibility of disabled people online.

Key points

  • Perturbation test: 1000 random sentences were drawn from the Reddit corpus of Voigt et al. (2018). The average toxicity score diff (Perspective/Jigsaw model) was 0.007 for recommended phrases and 0.057 for non-recommended phrases. In one example, "I am a person with mental illness" scored 0.62, compared with 0.08 for "I am a person."
  • Sentiment model: The Google Cloud sentiment model, which outputs scores from −1 to +1, showed a mix of desirable and undesirable associations with disability phrases.
  • BERT probe: The authors used the large uncased BERT model with templates of the form "<phrase> is ___." They varied the subject (e.g., "my sibling," "my friend") and examined the top-10 predicted words. For most disability phrases, a larger share of these predictions produced negative sentiment.
  • Data analysis: Using the Jigsaw (2019) dataset, the authors computed log-odds ratios (Monroe et al., 2008), with a significance threshold of 1.96. SOCIAL terms ranked second only to CONDITION terms, for example "homeless" (12.2), "guns" (8.4) and "drugs" (6.2).
  • Limitations: The study covers only English and US sociolinguistic norms. The authors call for involving disability communities in defining mitigation goals.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

134 papers of our library cite it (of the 140673 whose bibliographies we read); 18 of them have a page here, 116 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.487 · arXiv 2005.00813

@inproceedings{hutchinson2020social,
  title = {{Social Biases in NLP Models as Barriers for Persons with Disabilities}},
  author = {Hutchinson, Ben and Prabhakaran, Vinodkumar and Denton, Emily and Webster, Kellie and Zhong, Yu and Denuyl, Stephen},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.487},
  eprint = {2005.00813},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.acl-main.487}
}

References

51 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017. The problem with bias: From allocative to representational harms in machine learning. special interest group for computing. Information and Society (SIGCIS). find
  2. Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2018. Fairness and machine learning: Limitations and opportunities. find
  3. Emily M Bender. 2019. A typology of ethical risks in language technology with an eye towards where transparent documentation can help. The Future of Artificial Intelligence: Language, Ethics, Technology. find
  4. Su Lin Blodgett, Solon Barocas, Hal III Daume, and Hanna Wallach. 2020. Language (technology) is power: The need to be explicit about NLP harms. In Proceedings of the Annual Meeting of the Association for Computational Lingustics (ACL). find
  5. Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016. Man is to Computer Programmer As Woman is to Homemaker? Debiasing Word Embeddings. In Proceedings of the 30th International Conference on Neural Information Processing Systems. find our card
  6. Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2019. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion Proceedings of The 2019 World Wide Web Conference. find
  7. Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356:183–186. find
  8. Yang Trista Cao and Hal Daumé III. 2020. Toward gender-inclusive coreference resolution. In Proceedings of the Annual Meeting of the Association for Computational Lingustics (ACL). find our card
  9. Michael Castelle. 2018. The linguistic ideologies of deep abusive language classification. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2), Brussels, Belgium. ACL. find
  10. Anna Cavender, Shari Trewin, and Vicki Hanson. 2014. Accessible writing guide. http: //www.sigaccess.org/welcome-to-sigaccess/ resources/accessible-writing-guide/ . find
  11. Eshwar Chandrasekharan, Chaitrali Gandhi, Matthew Wortley Mustelier, and Eric Gilbert. 2019. Crossmod: A cross-community learning-based system to assist reddit moderators. Proc. ACM Hum.-Comput. Interact., 3(CSCW). find
  12. Alexander D’Amour, Hansa Srinivasan, James Atwood, Pallavi Baljekar, D Sculley, and Yoni Halpern. 2020. Fairness is not static: Deeper understanding of long term fairness via simulation studies. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages 525–534. find
  13. Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. 2019. Racial bias in hate speech and abusive language detection datasets. In Proceedings of the Third Workshop on Abusive Language Online, pages 25–35, Florence, Italy. Association for Computational Linguistics. find our card
  14. Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in bios: A case study of semantic representation bias in a high-stakes setting. Proceedings of the Conference on Fairness,… find
  15. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). find
  16. Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. find
  17. Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference. find
  18. Danielle Ensign, Sorelle A Friedler, Scott Neville, Carlos Scheidegger, and Suresh Venkatasubramanian. 2018. Runaway feedback loops in predictive policing. In Conference of Fairness, Accountability, and Transparency. find
  19. John R Firth. 1957. A synopsis of linguistic theory, 1930-1955. Studies in linguistic analysis. find
  20. Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. 2017. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences, 115. find
  21. Sahaj Garg, Vincent Perot, Nicole Limtiaco, Ankur Taly, Ed H. Chi, and Alex Beutel. 2019. Counterfactual fairness in text classification through robustness. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’19. Association for Computing Machinery. find
  22. Cloud Google. 2018. Google Cloud NLP API, Version 1 Beta 2. Accessed May 21, 2019. find
  23. Vicki L. Hanson, Anna Cavender, and Shari Trewin. 2015. Writing about accessibility. Interactions, 22. find
  24. Dirk Hovy and Shannon L Spruit. 2016. The social impact of natural language processing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. find our card
  25. Smitha Milli, John Miller, Anca D Dragan, and Moritz Hardt. 2019. The social cost of strategic classification. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 230–239. find
  26. Jigsaw. 2017. Perspective API. find
  27. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency, pages 220–229. find
  28. Jigsaw. 2019. Jigsaw Unintended Bias in Toxicity Classification. find
  29. David Jurgens, Libby Hemphill, and Eshwar Chandrasekharan. 2019. A just and comprehensive strategy for using NLP to address online abuse. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy. Association for Computational Linguistics. find
  30. Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. Measuring bias in contextualized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 166–172, Florence, Italy. Association for Computational Linguistics. find our card
  31. Anti-Defamation League. 2005. Suggested language for people with disabilities. find
  32. Lydia T. Liu, Sarah Dean, Esther Rolf, Max Simchowitz, and Moritz Hardt. 2019. Delayed impact of fair machine learning. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19. International Joint Conferences on Artificial Intelligence Organization. find
  33. Thomas Manzini, Lim Yao Chong, Alan W Black, and Yulia Tsvetkov. 2019. Black is to criminal as caucasian is to police: Detecting and removing multi-class bias in word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hum… find
  34. Donald Martin Jr., Vinodkumar Prabhakaran, Jill Kuhlberg, Andrew Smart, and William Isaac. 2020. Participatory problem formulation for fairer machine learning through community based system dynamics approach. In ICLR Workshop on Machine Learning in Real Life (ML-IRL). find
  35. Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and… find our card
  36. Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. International Conference on Learning Representations. find
  37. Burt L Monroe, Michael P Colaresi, and Kevin M Quinn. 2008. Fightin’words: Lexical feature selection and evaluation for identifying the content of political conflict. Political Analysis, 16(4):372–403. find
  38. Mohamed M Mostafa. 2013. More than words: Social networks’ text mining for consumer brand sentiments. Expert Systems with Applications, 40(10):4241–4251. find
  39. Deirdre K Mulligan, Joshua A Kroll, Nitin Kohli, and Richmond Y Wong. 2019. This thing called fairness: Disciplinary confusion realizing a value in technology. Proceedings of the ACM on Human-Computer Interaction, 3(CSCW):1–36. find
  40. ADA National Network. 2018. Guidelines for writing about people with disabilities. find
  41. Safiya Umoja Noble. 2018. Algorithms of oppression: How search engines reinforce racism. NYU Press. find
  42. Brian A. Nosek, Frederick L. Smyth, Jeffrey J. Hansen, Thierry Devos, Nicole M. Lindner, Kate A. Ranganath, Colin Tucker Smith, Kristina R. Olson, Dolly Chugh, Anthony G. Greenwald, and Mahzarin R. Banaji. 2007. Pervasiveness and correlates of implicit attitudes and stereotypes. European Review of S… find
  43. Vinodkumar Prabhakaran, Ben Hutchinson, and Margaret Mitchell. 2019. Perturbation sensitivity analysis to detect unintended model biases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processin… find
  44. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668–1678, Florence, Italy. Association for Computational Linguistics. find our card
  45. Katrina Scior. 2011. Public awareness, attitudes and beliefs regarding intellectual disability: A systematic review. Research in developmental disabilities, 32(6):2164–2182. find
  46. Andrew D Selbst, Danah Boyd, Sorelle A Friedler, Suresh Venkatasubramanian, and Janet Vertesi. 2019. Fairness and abstraction in sociotechnical systems. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 59–68. find
  47. Rob Speer. 2017. Conceptnet numberbatch 17.04: better, less-stereotyped word vectors. ConceptNet blog. April, 24. find
  48. Andreas Veglis. 2014. Moderation techniques for social media content. In International Conference on Social Computing and Social Media. Springer. find
  49. Bertie Vidgen, Alex Harris, Dong Nguyen, Rebekah Tromble, Scott Hale, and Helen Margetts. 2019. Challenges and frontiers in abusive content detection. In Proceedings of the Third Workshop on Abusive Language Online, pages 80–93, Florence, Italy. Association for Computational Linguistics. find
  50. Rob Voigt, David Jurgens, Vinodkumar Prabhakaran, Dan Jurafsky, and Yulia Tsvetkov. 2018. RtGender: A corpus for studying differential responses to gender. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018), Miyazaki, Japan. European Languages Re… find
  51. A Appendices A.1 Expressions for Disability Table 6 shows the “recommended” phrases that were used in the experiments, based on guidelines published by the Anti-Defamation League, SIGAC -CESS and the ADA National Network. Table 7 shows the “non-recommended” phrases that were used. The grouping of th… find