Elamur Research
☰
home / papers / 2022 / Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection

Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2022.naacl-main.431.pdf

Summary

Toxic language datasets usually merge a few raters' judgments into a single "correct" label. This ignores that perceived toxicity depends on who is reading. Sap et al. study how annotators' demographics (race, gender, political leaning) and attitudes, measured with established social-psychology scales, shape toxicity judgments. The posts they examine fall into three categories: vulgar language, text written in African American English (AAE), and content that is hostile toward Black people.

The authors ran two online studies with US participants. In the first, a large, varied crowd rated a small hand-picked set of posts. The second imitated ordinary dataset annotation, with many posts and a smaller but diverse rater pool.

Raters who scored high on racist beliefs found anti-Black posts less toxic. Conservatism was tied to seeing AAE as more toxic, and conservative or traditionalist views to harsher ratings of vulgarity. A pilot analysis showed that PERSPECTIVE API scores reflect some annotator perspectives more than others.

The authors conclude that toxicity labels should be documented and modeled together with annotator identities and beliefs, not treated as one aggregated truth.

Key points

  • Two studies: a breadth-of-workers study (641 annotators rating 15 curated posts) and a breadth-of-posts study (173 annotators rating about 600 posts). All participants were US-based, and posts were rated for both offensiveness and racism.
  • Seven attitude scales: FreeOffSpeech, HarmOfHateSpeech, RacistBeliefs (Modern Racism Scale items), Traditionalism, LingPurism (a new four-item scale), Empathy (abbreviated Interpersonal Reactivity Index) and Altruism.
  • Vulgarity types: following Zhou et al. (2021), vulgarity is split into non-identity-referring swearing (ONI) and identity-referring slurs (OI). The analyses mainly use ONI.
  • PERSPECTIVE API case study: for anti-Black posts, the system's scores matched raters with higher RacistBeliefs more closely. For AAE, its scores were closer to white annotators' perceptions than to Black annotators'.
  • Recommendations: report annotator demographics and attitudes (e.g., in datasheets) and model label distributions instead of majority votes. The authors also suggest moving toward explainable frameworks such as Social Bias Frames.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

208 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 200 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.naacl-main.431

@inproceedings{sap2022annotators,
  title = {{Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection}},
  author = {Sap, Maarten and Swayamdipta, Swabha and Vianna, Laura and Zhou, Xuhui and Choi, Yejin and Smith, Noah A.},
  booktitle = {Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-main.431},
  url = {https://doi.org/10.18653/v1/2022.naacl-main.431}
}

References

110 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sohail Akhtar, Valerio Basile, and Viviana Patti. 2021. Whose opinions matter? perspective-aware models to identify opinions of hate speech victims in abusive language detection. ArXiv preprint arXiv:2106.15896. — [arXiv:2106.15896](https://arxiv.org/abs/2106.15896) find
  2. Hala Al Kuwatly, Maximilian Wich, and Georg Groh. 2020. Identifying and measuring annotator bias based on annotators’ demographic characteristics. In Proceedings of the Fourth Workshop on Online Abuse and Harms. find
  3. Carolina Are. 2020. How instagram’s algorithm is censoring women and vulnerable users but helping online abusers. Feminist media studies, 20(5). find
  4. Kofi Arhin, Ioana Baldini, Dennis Wei, Karthikeyan Natesan Ramamurthy, and Moninder Singh. 2021. Ground-Truth, whose truth? – examining the challenges with annotating toxic text datasets. find
  5. Luke Breitfeller, Emily Ahn, David Jurgens, and Yulia Tsvetkov. 2019. Finding microaggressions in the wild: A case for locating elusive phenomena in social media posts. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferenc… find
  6. Martin J Burnham, Yen K Le, and Ralph L Piedmont. 2018. Who is mturk? personal characteristics and sample consistency of these online workers. Mental Health, Religion & Culture, 21(9-10). find
  7. Scott Allen Cambo. 2021. Model Positionality: A Novel Framework for Data Science with Subjective Target Concepts. Ph.D. thesis, Northwestern University. find
  8. Evelyn R Carter and Mary C Murphy. 2015. Group-based differences in perceptions of racism: What counts, to whom, and why? Social and personality psychology compass, 9(6). find
  9. Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017. The problem with bias: Allocative versus representational harms in machine learning. In SIGCIS. find
  10. Tempii B Champion, Deirdre Cobb-Roberts, and Linda Bland-Stewart. 2012. Future educators’ perceptions of african american vernacular english (AAVE). Online Journal of Education Research, 1(5). find
  11. Margaret Beneke and Gregory A Cheatham. 2015. Speaking up for african american english: Equity and inclusion in early childhood settings. Early Childhood Education Journal, 43(2). find
  12. Jamie Cleland. 2014. Racism, football fans, and online message boards: How social media has added a new dimension to racist discourse in English football. J. Sport Soc. Issues, 38(5). find
  13. Susan Benesch. 2020. Proposals for improved regulation of harmful online content. Technical report. find
  14. D Cole. 1996. Racist speech should be protected by the constitution. Hate crimes. find
  15. Abeba Birhane. 2021. Algorithmic injustice: a relational ethics approach. Patterns (New York, N.Y.), 2(2). find
  16. Gloria Cowan and Désirée Khatchadourian. 2003. Empathy, ways of knowing, and interdependence as mediators of gender differences in attitudes toward hate speech and freedom of speech. Psychology of Women Quarterly, 27(4). find
  17. Renée Blake and Cecilia Cutler. 2003. AAE and variation in teachers’ attitudes: A question of school philosophy? Linguistics and education, 14(2). find
  18. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of “bias” in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. find our card
  19. Su Lin Blodgett, Lisa Green, and Brendan O’Connor. 2016. Demographic dialectal variation in social media: A case study of African-American English. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. find our card
  20. Gloria Cowan, Miriam Resendez, Elizabeth Marshall, and Ryan Quist. 2002. Hate speech and constitutional protection: Priming values of equality and freedom. The Journal of social issues, 58(2). find
  21. Adam M Croom. 2011. Slurs. Language Sciences, 33(3). find
  22. Aida Mostafazadeh Davani, Mohammad Atari, Brendan Kennedy, and Morteza Dehghani. 2021. Hate speech classifiers learn human-like social stereotypes. find
  23. Eduardo Bonilla-Silva. 2006. Racism without racists: Color-blind racism and the persistence of racial inequality in the United States. find
  24. Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. 2019. Racial bias in hate speech and abusive language detection datasets. In Proceedings of the Third Workshop on Abusive Language Online. find our card
  25. Thomas J. Bouchard Jr. and Matt McGue. 2003. Genetic and environmental influences on human psychological differences. Journal of Neurobiology, 54(1). find
  26. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In ICWSM. find our card
  27. Melanie DeFrank and Patricia Kahlbaugh. 2019. Language choice matters: When profanity affects how people are judged. Journal of Language and Social Psychology, 38(1). find
  28. Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019. Build it break it fix it for dialogue safety: Robustness from adversarial human attack. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Nat… find our card
  29. Daniel M Downs and Gloria Cowan. 2012. Predicting the importance of freedom of speech and the perceived harm of hate speech. Journal of applied social psychology, 42(6). find
  30. John Duckitt and Kirstin Fisher. 2003. The impact of social threat on worldview and ideological attitudes. Political Psychology, 24(1). find
  31. Marta Dynel. 2012. Swearing methodologically : the (im)politeness of expletives in anonymous commentaries on youtube. Journal of English Studies, 10(0). find
  32. Walter F Edwards. 2004. African american vernacular english: phonology. In A Handbook of Varieties of English: Morphology and syntax. find
  33. Christine Helen Elers and Pooja Jayan. 2020. “this is us”: Free speech embedded in whiteness, racism and coloniality in aotearoa, new zealand. First Amendment Studies, 54(2). find
  34. S T Fiske. 1993. Controlling other people. the impact of power on stereotyping. The American psychologist, 48(6). find
  35. Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Barbara Plank, Dirk Hovy, and Massimo Poesio. 2021. Beyond black & white: Leveraging annotator disagreement via soft-label multi-task learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Lin… find
  36. Antigoni Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large scale crowdsourcing and characterization of twitter abusive behavior. In ICWSM. find
  37. Batya Friedman, Peter H Kahn, and Alan Borning. 2008. Value sensitive design and information systems. The handbook of information and computer ethics. find
  38. Sarah E Gaither, Ariel M Cohen-Goldberg, Calvin L Gidney, and Keith B Maddox. 2015. Sounding black or white: Priming identity and biracial speech. Frontiers in Psychology, 6. find
  39. Adam D Galinsky, Cynthia S Wang, Jennifer A Whitson, Eric M Anicich, Kurt Hugenberg, and Galen V Bodenhausen. 2013. The reappropriation of stigmatizing labels: the reciprocal relationship between power and self-labeling. Psychol. Sci., 24(10). find
  40. Sergey Gavrilets and Laura Fortunato. 2014. A solution to the collective action problem in between-group conflict with within-group inequality. Nature communications, 5(1). find
  41. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2018. Datasheets for datasets. In FAccT*. find
  42. Karen E Gerdes, Cynthia A Lietz, and Elizabeth A Segal. 2011. Measuring empathy in the 21st century: Development of an empathy index rooted in social cognitive neuroscience and social justice. Social Work Research, 35(2). find
  43. Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conf… find
  44. David Gillborn. 2009. Risk-free racism: Whiteness and so-called free speech. Wake Forest law review, 44. find
  45. Tarleton Gillespie, Patricia Aufderheide, Elinor Carmi, Ysabel Gerrard, Robert Gorwa, Ariadna Matamoros-Fernandez, Sarah T Roberts, Aram Sinnreich, and Sarah Myers West. 2020. Expanding the debate about content moderation: Scholarly research agendas for the coming policy debates. Internet Policy Rev… find
  46. Donna M Goldstein and Kira Hall. 2017. Postelection surrealism and nostalgic racism in the hands of donald trump. HAU: Journal of Ethnographic Theory, 7(1). find
  47. Mitchell L Gordon, Michelle S Lam, Joon Sung Park, Kayur Patel, Jeffrey T Hancock, Tatsunori Hashimoto, and Michael S Bernstein. 2022. Jury learning: Integrating dissenting voices into machine learning models. In CHI. find
  48. Ben Green. 2020. Data science as political action: Grounding data science in a politics of justice. SSRN Electronic Journal. find
  49. Lisa Green. 2002. African American English: A Linguistic Introduction, 8.3.2002 edition edition. find
  50. Gil Greengross and Geoffrey F Miller. 2008. Dissing oneself versus dissing rivals: effects of status, personality, and sex on the Short-Term and Long-Term attractiveness of Self-Deprecating and Other-Deprecating humor. Evolutionary Psychology, 6(3). find
  51. Xiaochuang Han and Yulia Tsvetkov. 2020. Fortifying toxic speech detectors against veiled toxicity. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  52. A G Hilliard. 1997. Language, culture and the assessment of african american children. Assessment for equity and inclusion: Embracing all our children. find
  53. Dirk Hovy and Diyi Yang. 2021. The importance of modeling social factors of language: Theory and practice. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. find our card
  54. Connor Huff and Dustin Tingley. 2015. “who are these people?” evaluating the demographic characteristics and political preferences of mturk survey respondents. Research & Politics, 2(3). find
  55. Christian Ilbury. 2020. “sassy queens”: Stylistic orthographic variation in twitter and the enregisterment of AAVE. Journal of sociolinguistics, 24(2). find
  56. Timothy Jay. 2018. Swearing, moral order, and online communication. Journal of Language Aggression and Conflict, 6(1). find
  57. Timothy Jay and Kristin Janschewitz. 2008. The pragmatics of swearing. Journal of Politeness Research, 4. find
  58. Björn H Jernudd and Michael J Shapiro. 1989. The politics of language purism. find
  59. J A Jiang. 2020. Identifying and addressing design and policy challenges in online content moderation. Extended Abstracts of the 2020 CHI Conference on. find
  60. Jialun Aaron Jiang, Morgan Klaus Scheuerman, Casey Fiesler, and Jed R Brubaker. 2021. Understanding international perceptions of the severity of harmful content online. PloS one, 16(8). find
  61. Darin G Johnson, Bradley D Mattan, Nelson Flores, Nina Lauharatanahirun, and Emily B Falk. 2022. Social-Cognitive and affective antecedents of code switching and the consequences of linguistic racism for black people and people of color. Affective Science, 3(1). find
  62. Eric Loepp and Jarrod T Kelly. 2020. Distinction without a difference? an assessment of mturk worker types. Research & Politics, 7(1). find
  63. Daniel S Lucks. 2020. Reconsidering Reagan: Racism, Republicans, and the Road to Trump. find
  64. Antonis Maronikolakis, Axel Wisiorek, Leah Nann, Haris Jabbar, Sahana Udupa, and Hinrich Schütze. 2022. Listening to affected communities to define extreme speech: Dataset and experiments. In ACL 2022 Findings. find
  65. John B McConahay. 1986. Modern racism, ambivalence, and the modern racism scale. In Prejudice, discrimination, and racism, volume 337. find
  66. Kris McGuffie and Alex Newhouse. 2020. The radicalization risks of gpt-3 and advanced neural language models. find
  67. Ashley Muddiman. 2021. Conservatives and incivility. In Conservative Political Communication. find
  68. Kevin L Nadal. 2018. Microaggressions and traumatic stress: Theory, research, and clinical treatment. find
  69. Jessica C Nelson, Glenn Adams, and Phia S Salter. 2013. The marley hypothesis: Denial of racism reflects ignorance of history. Psychological science, 24(2). find
  70. Michael I Norton and Samuel R Sommers. 2011. Whites see racism as a Zero-Sum game that they are now losing. Perspectives on psychological science: a journal of the Association for Psychological Science, 6(3). find
  71. Gwenn Schurgin O’Keeffe, Kathleen Clarke-Pearson, and Council on Communications and Media. 2011. The impact of social media on children, adolescents, and families. Pediatrics, 127(4). find
  72. Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7. find our card
  73. V Paul Poteat and Lisa B Spanierman. 2012. Modern racism attitudes among white students: The role of dominance and authoritarianism and the mediating effects of racial color-blindness. The Journal of Social Psychology, 152(6). find
  74. Stephen D Johnson and Joseph B Tamney. 2001. Social traditionalism and economic conservatism: Two conservative political ideologies in the united states. The Journal of social psychology, 141(2). find
  75. Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Díaz. 2021. On releasing Annotator-Level labels and information in datasets. In Proc. of LAW-DMR workshop at EMNLP. find
  76. Pratyusha Kalluri. 2020. Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature, 583(7815). find
  77. Steven Pulos, Jeff Elison, and Randy Lennon. 2004. The hierarchical structure of the interpersonal reactivity index. Social behavior and personality, 32(4). find
  78. Jonathan Knuckey. 2005. A new front in the culture war? moral traditionalism and voting behavior in us house elections. American Politics Research, 33(5). find
  79. Jacquelyn Rahman. 2008. Middle-class african americans: Reactions and attitudes toward african american english. American speech, 83(2). find
  80. K Sabeel Rahman. 2017. The new utilities: Private power, social infrastructure, and the revival of the public utility concept. Cardozo law review, 39. find
  81. Marc Riar, Benedikt Morschheuser, Juho Hamari, and Rüdiger Zarnekow. 2020. How game features give rise to altruism and collective action? implications for cultivating cooperation by gamification. In Proceedings of the 53rd Hawaii International Conference on System Sciences. find
  82. Sarah T Roberts. 2017. Social media’s silent filter. The Atlantic. find
  83. Jonathan Rosa. 2019. Looking Like a Language, Sounding Like a Race. find
  84. Jonathan Rosa and Nelson Flores. 2017. Unsettling race and language: Toward a raciolinguistic perspective. Language In Society, 46(5). find
  85. Björn Ross, Michael Rist, Guillermo Carbonell, Benjamin Cabrera, Nils Kurowsky, and Michael Wojatzki. 2017. Measuring the reliability of hate speech annotations: the case of the european refugee crisis. In NLP 4 CMC Workshop. find
  86. Gina Roussos and John F Dovidio. 2018. Hate speech is in the eye of the beholder: The influence of racial attitudes and freedom of speech beliefs on perceptions of racially motivated threats of violence. Social psychological and personality science, 9(2). find
  87. Elizabeth Sanders. 2002. From user-centered to participatory design approaches. find
  88. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. find our card
  89. Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power implications of language. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. find our card
  90. N Clayton Silver and William P Dunlap. 1987. Averaging correlation coefficients: should fisher’s z transformation be used? Journal of applied psychology, 72(1). find
  91. Wiktor Soral, Michał Bilewicz, and Mikołaj Winiewski. 2018. Exposure to hate speech increases prejudice through desensitization. Aggressive behavior, 44(2). find
  92. Arthur K Spears et al. 1998. African-american language use: Ideology and so-called obscenity. African-American English: Structure, history, and use. find
  93. Linda Steg, Goda Perlaviciute, Ellen Van der Werff, and Judith Lurvink. 2014. The significance of hedonic values for environmentally relevant attitudes, preferences, and actions. Environment and behavior, 46(2). find
  94. Miriah Steiger, Timir J Bharucha, Sukrit Venkatagiri, Martin J Riedl, and Matthew Lease. 2021. The psychological Well-Being of content moderators: The emotional labor of commercial moderation and avenues for improving support. In CHI, number Article 341 in CHI ’21. find
  95. Joanna Sterling, John T Jost, and Richard Bonneau. 2020. Political psycholinguistics: A comprehensive analysis of the language habits of liberal and conservative social media users. Journal of personality and social psychology, 118(4). find
  96. Ane Ueland. 2020. Language and identity: a study of African American Vernacular English and its status in American society. Ph.D. thesis, University of Stavanger, Norway. find
  97. Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. 2021. Learning from the worst: Dynamically generated datasets to improve online hate detection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Nat… find our card
  98. Graham F Wagstaff. 1998. Equity, justice, and altruism. Current Psychology, 17(2). find
  99. Barry S. Sapolsky, Daniel M. Shafer, and Barbara K. Kaye. 2010. Rating offensive words in three television program contexts. Mass Communication and Society, 14(1). find
  100. Zeerak Waseem. 2016. Are you a racist or am I seeing things? annotator influence on hate speech detection on Twitter. In Proceedings of the First Workshop on NLP and Computational Social Science. find
  101. Brian F Schaffner. 2020. The heightened importance of racism and sexism in the 2018 US midterm elections. British journal of political science. find
  102. Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? predictive features for hate speech detection on Twitter. In Proceedings of the NAACL Student Research Workshop. find
  103. Leandro Silva, Mainack Mondal, Denzil Correa, Fabrício Benevenuto, and Ingmar Weber. 2016. Analyzing the targets of hate in online social media. In Tenth international AAAI conference on web and social media. find
  104. Zeerak Waseem, Smarika Lulz, Joachim Bingel, and Isabelle Augenstein. 2021. Disembodied machine learning: On the illusion of objectivity in nlp. Anonymous preprint under review. find
  105. Mark H White and Christian S Crandall. 2017. Freedom of racist speech: Ego and expressive threats. Journal of personality and social psychology, 113(3). find
  106. Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex machina: Personal attacks seen at scale. In Proceedings of the 26th International Conference on World Wide Web, WWW 2017, Perth, Australia, April 3-7, 2017. find
  107. Danyaal Yasin. 2018. Black and banned: Who is free speech for? . Accessed: 2018-12-6. find
  108. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. Predicting the type and target of offensive posts in social media. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Languag… find our card
  109. Zuleyka Zevallos. 2017. Sociology of race. . Accessed: 2021-10-10. find
  110. Xuhui Zhou, Maarten Sap, Swabha Swayamdipta, Yejin Choi, and Noah Smith. 2021. Challenges in automated debiasing for toxic language detection. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. find