Elamur Research⌕
☰
home / papers / 2023 / NLPositionality: Characterizing Design Biases of Datasets and Models

NLPositionality: Characterizing Design Biases of Datasets and Models

Paper · 2023 · Sebastin Santy, Jenny Liang, Ronan Le Bras et al.

Summary

NLP datasets and models often work better for some populations than others. These design biases frequently trace back to the positionality of their creators, meaning the identity, background and lived experience that shape their choices. Such choices are rarely documented, and many systems are only reachable through APIs, so the skew is hard to measure. The authors propose NLPositionality, a framework that works with any dataset or model. It has volunteers recruited through LabintheWild re-annotate a sample of task instances and report their demographics. It then measures, using Pearson's r, how closely each demographic group's judgments match the original labels or a model's outputs.

The study covers two tasks, social acceptability (e.g., Social Chemistry 101) and hate speech detection. It examines datasets, supervised classifiers and GPT-4. All of these agree most with Western, White, college-educated, younger and English-speaking people. Non-binary people and non-native English speakers show the weakest alignment. Volunteers take part to learn about themselves rather than for payment, so data collection can continue indefinitely at no cost. This gives an ongoing way to audit whose perspectives NLP systems reflect.

Key points

  • Scale: By May 25, 2023, the framework had gathered 16,299 annotations for 600 instances. These came from 1,096 annotators in 87 countries, averaging about 38 annotations per day.
  • Method: For each dataset, 300 instances were sampled and shown in LabintheWild studies. Annotators were unpaid, and the study was approved by the University of Washington IRB (STUDY00014813).
  • Main finding: Alignment is highest with a subset of "WEIRD" populations. Datasets also align closely with their own original annotators.
  • Most marginalized groups: Non-binary people and non-native English speakers rank lowest in alignment across all tasks.
  • Limitations and resources: The framework works best for classification tasks. It uses demographics as a proxy for positionality and does not model intersectionality. Updated results are published at nlpositionality.cs.washington.edu and github.com/liang-jenny/nlpositionality.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States365 works · 158,048 citations🎓University of WashingtonSeattle · United States263 works · 180,581 citations🔬Allen Institute for Artificial IntelligenceSeattle · United States219 works · 89,712 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksSocial Sciences · 744 worksSafety Research · 221 works

Cite

Cite as Sebastin Santy, Jenny Liang, Ronan Le Bras, Katharina Reinecke and Maarten Sap. 2023. NLPositionality: Characterizing Design Biases of Datasets and Models. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.acl-long.505

@inproceedings{santy2023nlpositionality,
  title = {{NLPositionality: Characterizing Design Biases of Datasets and Models}},
  author = {Sebastin Santy, Jenny Liang, Ronan {Le Bras}, Katharina Reinecke and Maarten Sap},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.505},
  url = {https://doi.org/10.18653/v1/2023.acl-long.505}
}

References

90 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AAAI. 2023. Reproducibility checklist. find
  2. ACL. 2017. ACL Diversity Statistics. 2022. Morality beyond the WEIRD: How the nomological network of morality varies across cultures. find
  3. Tal August and Katharina Reinecke. 2019. Pay attention, please: Formal language improves attention in volunteer and paid online experiments. In ACM SIGCHI Conference on Human Factors in Computing Systems, page 1–11. find
  4. Edmond Awad, Sohan Dsouza, Richard Kim, Jonathan Schulz, Joseph Henrich, Azim Shariff, Jean-François Bonnefon, and Iyad Rahwan. 2018. The moral machine experiment. Nature, 563(7729):59–64. find
  5. Edmond Awad, Sohan Dsouza, Azim Shariff, Iyad Rahwan, and Jean-François Bonnefon. 2020. Universals and variations in moral decisions made in 42 countries by 70,000 participants. National Academy of Sciences, 117(5):2332–2337. find
  6. Valerio Basile, Federico Cabitza, Andrea Campagner, and Michael Fell. 2021. Toward a perspectivist turn in ground truthing for predictive computing. arXiv preprint arXiv:2109.04270.
  7. Emily Bender. 2019. The\# benderrule: On naming the languages we study and why it matters. The Gradient, 14. find
  8. Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604. find
  9. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In ACM Conference on Fairness, Accountability, and Transparency, page 610–623. find
  10. Abeba Birhane, Pratyusha Kalluri, Dallas Card, William Agnew, Ravit Dotan, and Michelle Bao. 2022. The values encoded in machine learning research. In ACM Conference on Fairness, Accountability, and Transparency, pages 173–184. find
  11. David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D’souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, et al. 2021. Masakhaner: Named entity recognition for african languages. Transactions of the Association for Computational Li…
  12. Damian Blasi, Antonios Anastasopoulos, and Graham Neubig. 2022. Systematic inequalities in language technology performance across the world’s languages. In Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5486–5505.
  13. Arnav Arora, Lucie-Aimée Kaffee, and Isabelle Augenstein. 2023. Probing pre-trained language models for cross-cultural differences in values. In Workshop on Cross-Cultural Considerations in NLP, page 114–130.
  14. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of "bias" in NLP. In Annual Meeting of the Association for Computational Linguistics, pages 5454–5476.
  15. Lora Aroyo and Chris Welty. 2015. Truth is a lie: Crowd truth and the seven myths of human annotation. AI Magazine, 36(1):15–24.
  16. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. volume 29.
  17. Mohammad Atari, Jonathan Haidt, Jesse Graham, Sena Koleva, Sean T Stevens, and Morteza Dehghani. find
  18. Vivien Burr. 2015. Social Constructionism. Routledge. find
  19. Andrew Caines. 2021. The geographic diversity of NLP conferences. find
  20. Scott Allen Cambo and Darren Gergle. 2022. Model positionality and computational reflexivity: Promoting reflexivity in data science. In ACM SIGCHI Conference on Human Factors in Computing Systems, pages 1–19. find
  21. Sasha Costanza-Chock. 2018. Design justice, AI, and escape from the matrix of domination. Journal of Design and Science, 3(5). find
  22. Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. 2022. Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Transactions of the Association for Computational Linguistics, 10:92–110.
  23. Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. 2019. Racial bias in hate speech and abusive language detection datasets.
  24. Emily Denton and Timnit Gebru. 2020. Tutorial on fairness, accountability, transparency, and ethics in computer vision at CVPR 2020. find
  25. Sunipa Dev, Masoud Monajatipoor, Anaelia Ovalle, Arjun Subramonian, Jeff Phillips, and Kai-Wei Chang. 2021. Harms of gender exclusivity and challenges in non-binary representation in language technologies. In Conference on Empirical Methods in Natural Language Processing, pages 1968–1994.
  26. Kaustubh D Dhole, Varun Gangal, Sebastian Gehrmann, Aadesh Gupta, Zhenhao Li, Saad Mahamood, Abinaya Mahendiran, Simon Mille, Ashish Srivastava, Samson Tan, et al. 2021. NL-augmenter: A framework for task-sensitive natural language augmentation. arXiv preprint arXiv:2112.02721.
  27. Djellel Difallah, Elena Filatova, and Panos Ipeirotis. 2018. Demographics and dynamics of mechanical turk workers. In ACM International Conference on Web Search and Data Mining, page 135–143. find
  28. Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Conference on Empirical Methods in Natural Language Processing, pages 1286–13… find
  29. Bonaventure F.P. Dossou, Atnafu Lambebo Tonja, Oreen Yousuf, Salomey Osei, Abigail Oppong, Iyanuoluwa Shode, Oluwabusayo Olufunke Awoyomi, and Chris Emezue. 2022. Afrolm: A self-active learning-based multilingual pretrained language model for 23 African languages. In Workshop on Simple and Efficient… find
  30. Maeve Duggan, L Rainie, A Smith, C Funk, A Lenhart, and M Madden. 2014. Online harassment. Pew Research Center, Washington, DC, USA, Technical Rep. find
  31. Mary Q Foote and Tonya Gau Bartell. 2011. Pathways to equity in mathematics education: How life experiences impact researcher positionality. Educational Studies in Mathematics, 78(1):45–68. find
  32. Maxwell Forbes, Jena D. Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. 2020. Social chemistry 101: Learning to reason about social and moral norms. In Conference on Empirical Methods in Natural Language Processing, pages 653–670. find
  33. Batya Friedman. 1996. Value-sensitive design. Interactions, 3(6):16–23. find
  34. Batya Friedman and Helen Nissenbaum. 1996. Bias in computer systems. ACM Transactions on Information Systems, 14(3):330–347. find
  35. Aparna Garimella, Carmen Banea, Dirk Hovy, and Rada Mihalcea. 2019. Women’s syntactic resilience and men’s grammatical luck: Gender-bias in part-of-speech tagging and dependency parsing. In Annual Meeting of the Association for Computational Linguistics, pages 3493–3498. find
  36. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2021. Datasheets for datasets. Communications of the ACM, 64(12):86–92. find
  37. Sayan Ghosh, Dylan Baker, David Jurgens, and Vinodkumar Prabhakaran. 2021. Detecting cross-geographic biases in toxicity modeling on social media. In Workshop on Noisy User-generated Text, page 313–328. find
  38. Maharshi Gor, Kellie Webster, and Jordan Boyd-Graber. 2021. Toward deconfounding the effect of entity demographics for question answering accuracy. In Conference on Empirical Methods in Natural Language Processing, pages 5457–5473. find
  39. Sophie Groenwold, Lily Ou, Aesha Parekh, Samhita Honnavalli, Sharon Levy, Diba Mirza, and William Yang Wang. 2020. Investigating African-American Vernacular English in transformer-based text generation. In Conference on Empirical Methods in Natural Language Processing, pages 5877–5883.
  40. Suchin Gururangan, Dallas Card, Sarah K Drier, Emily K Gade, Leroy Z Wang, Zeyu Wang, Luke Zettlemoyer, and Noah A Smith. 2022. Whose language counts as high quality? measuring language ideologies in text data selection. "Conference on Empirical Methods in Natural Language Processing", pages 2562–25… find
  41. Christian W Haerpfer and Kseniya Kizilova. 2012. The world values survey. The Wiley-Blackwell Encyclopedia of Globalization, pages 1–5. find
  42. Melissa Hall, Laurens van der Maaten, Laura Gustafson, and Aaron Adcock. 2022. A systematic study of bias amplification. arXiv preprint arXiv:2201.11706. — [arXiv:2201.11706](https://arxiv.org/abs/2201.11706) find
  43. Katharina Hämmerl, Björn Deiseroth, Patrick Schramowski, Jindřich Libovickỳ, Constantin A Rothkopf, Alexander Fraser, and Kristian Kersting. 2022. Speaking multiple languages affects the moral bias of language models. arXiv preprint arXiv:2211.07733. find
  44. Alex Hanna, Emily Denton, Andrew Smart, and Jamila Smith-Loud. 2020. Towards a critical race methodology in algorithmic fairness. In ACM Conference on Fairness, Accountability, and Transparency, pages 501–512. find
  45. Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. Toxigen: Controlling language models to generate implied and adversarial toxicity. In Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). find
  46. Joseph Henrich, Steven J Heine, and Ara Norenzayan. 2010. The weirdest people in the world? Behavioral and Brain Sciences, 33(2-3):61–83. find
  47. Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. 2022. Challenges and strategies in cross-cultural N…
  48. Andrew Gary Darwin Holmes. 2020. Researcher positionality–A consideration of its influence and place in qualitative research–A new researcher guide. Shanlax International Journal of Education, 8(4):1– 10. find
  49. Dirk Hovy and Anders Søgaard. 2015. Tagging performance correlates with author age. In Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 483–488.
  50. Panagiotis G Ipeirotis. 2010. Demographics of Mechanical Turk. find
  51. Lilly Irani, Janet Vertesi, Paul Dourish, Kavita Philip, and Rebecca E. Grinter. 2010. Postcolonial computing: A lens on design and development. In ACM SIGCHI Conference on Human Factors in Computing Systems, page 1311–1320. find
  52. Liwei Jiang, Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Jenny Liang, Jesse Dodge, Keisuke Sakaguchi, Maxwell Forbes, Jon Borchardt, Saadia Gabriel, et al. 2021. Can machines learn morality? the delphi experiment. arXiv e-prints, pages arXiv– 2110. find
  53. Anders Johannsen, Dirk Hovy, and Anders Søgaard. 2015. Cross-lingual syntactic variation over age and gender. In Conference on Computational Natural Language Learning, pages 103–112. find
  54. Anna Jørgensen, Dirk Hovy, and Anders Søgaard. 2015. Challenges of studying and processing dialects in social media. In ACL Workshop on Noisy User-generated Text, pages 9–18. find
  55. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Annual Meeting of the Association for Computational Linguistics, pages 6282–6293.
  56. Os Keyes, Jevan Hutson, and Meredith Durbin. 2019. A mulching proposal: Analysing and improving an algorithmic system for turning the elderly into high-nutrient slurry. In Extended abstracts of the SIGCHI Conference on Human Factors in Computing Systems, pages 1–11. find
  57. András Kornai. 2013. Digital language death. PLOS ONE, 8(10):1–11. find
  58. Klaus Krippendorff. 2006. Reliability in content analysis: Some common misconceptions and recommendations. Human Communication Research, 30(3):411– 433. find
  59. Michael Li. 2020. To build less-biased AI, hire a more diverse team. Harvard Business Review. find
  60. Kim Halskov Madsen and Peter H. Aiken. 1993. Experiences using cooperative interactive storyboard prototyping. Communications of the ACM, 36(6):57–64. find
  61. Manuel Mager, Ximena Gutierrez-Vasques, Gerardo Sierra, and Ivan Meza-Ruiz. 2018. Challenges of language technologies for the indigenous languages of the Americas. In International Conference on Computational Linguistics, pages 55–69.
  62. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In ACM Conference on Fairness, Accountability, and Transparency, page 220–229. find
  63. Youssef Mohamed, Mohamed Abdelfattah, Shyma Alhuwaider, Feifan Li, Xiangliang Zhang, Kenneth Ward Church, and Mohamed Elhoseiny. 2022. Artelingo: A million emotion annotations of WikiArt with emphasis on diversity over language and culture. In Conference on Empirical Methods in Natural Language Proc… find
  64. NAACL. 2021. Reproducibility checklist. find
  65. Nigini Oliveira, Eunice Jun, and Katharina Reinecke. 2017. Citizen science opportunities in volunteer-based online experiments. In ACM SIGCHI Conference on Human Factors in Computing Systems, page 6800–6812. find
  66. OpenAI. 2023. Gpt-4 technical report. arXiv.
  67. Kiemute Oyibo. 2016. Designing culture-based persuasive technology to promote physical activity among university students. In Proceedings of the 2016 conference on user modeling adaptation and personalization, pages 321–324. find
  68. Barbara Plank. 2022. The ’problem’ of human label variation: On ground truth in data, modeling and evaluation. In Conference on Empirical Methods in Natural Language Processing, pages 10671–10682.
  69. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training Gopher. arXiv preprint arXiv:2112.11446. find
  70. Katharina Reinecke and Abraham Bernstein. 2011. Improving performance, perceived usability, and aesthetics with culturally adaptive user interfaces. ACM Transactions on Computer-Human Interaction, 18(2):1–29. find
  71. Katharina Reinecke and Abraham Bernstein. 2013. Knowing what a user likes: A design science approach to interfaces that automatically adapt to culture. Mis Quarterly, pages 427–453. find
  72. Katharina Reinecke and Krzysztof Z. Gajos. 2015. LabInTheWild: Conducting large-scale online experiments with uncompensated samples. In ACM Conference on Computer Supported Cooperative Work & Social Computing, pages 1364––1378. find
  73. Jill Walker Rettberg. 2022. ChatGPT is multilingual but monocultural, and it’s learning your values. https://jilltxt.net/right-now-chatgpt-is-multilingual-but-monocultural-but-its-learning-your-values/. Accessed: 2023-5-25. ↗
  74. Anna Rogers. 2021. Changing the world by changing the data. In Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2182–2194. find
  75. Wendy E Rowe. 2014. Positionality. The SAGE encyclopedia of action research, 628:627–628. find
  76. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Annual Meeting of the Association for Computational Linguistics, pages 1668– 1678.
  77. Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language…
  78. Maggi Savin-Baden and Claire Howell-Major. 2013. Qualititative research: The essential guide to theory and practice. Qualitative Research: The Essential Guide to Theory and Practice. Routledge. find
  79. Vered Shwartz. 2022. Good night at 4 pm?! time expressions in different cultures. In Findings of the Association for Computational Linguistics: ACL, pages 2842–2853. find
  80. Clay Spinuzzi. 2005. The methodology of participatory design. Technical Communication, 52(2):163–174. find
  81. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615.
  82. Zeerak Talat, Hagen Blix, Josef Valvoda, Maya Indira Ganesh, Ryan Cotterell, and Adina Williams. 2022. On the machine learning of ethical judgments from natural language. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages… find
  83. Rachael Tatman. 2017. Gender and dialect bias in YouTube’s automatic captions. In ACL Workshop on Ethics in Natural Language Processing, pages 53–59.
  84. Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. 2021. Learning from the worst: Dynamically generated datasets to improve online hate detection. In Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (Vol…
  85. Ruyuan Wan, Jaehyung Kim, and Dongyeop Kang. 2023. Everyone’s voice matters: Quantifying annotation disagreement using demographic information. arXiv preprint arXiv:2301.05036. — [arXiv:2301.05036](https://arxiv.org/abs/2301.05036) find
  86. William Warner and Julia Hirschberg. 2012. Detecting hate speech on the world wide web. In Workshop on Language in Social Media, pages 19–26.
  87. Zeerak Waseem, Smarika Lulz, Joachim Bingel, and Isabelle Augenstein. 2021. Disembodied machine learning: On the illusion of objectivity in NLP. arXiv preprint arXiv:2101.11974. — [arXiv:2101.11974](https://arxiv.org/abs/2101.11974) find
  88. Thomas D Wickens and Geoffrey Keppel. 2004. Design and Analysis: A Researcher’s Handbook. Prentice-Hall. find
  89. Langdon Winner. 2017. Do artifacts have politics? In Computer Ethics, pages 177–192. Routledge. find
  90. Albert Xu, Eshaan Pathak, Eric Wallace, Suchin Gururangan, Maarten Sap, and Dan Klein. 2021. Detoxifying language models risks marginalizing minority voices. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2390–2397.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/santy-2023-nlpositionality-characterizing-design-biases-datasets/