Elamur Research⌕
☰
home / papers / 2023 / Hate Speech Classifiers Learn Normative Social Stereotypes

Hate Speech Classifiers Learn Normative Social Stereotypes

Paper · 2023 · Aida Mostafazadeh Davani, Mohammad Atari, Brendan Kennedy et al.

Summary

This paper traces how social stereotypes enter each stage of building English hate speech classifiers: individual labeling, aggregated datasets, and model predictions. It draws on the Stereotype Content Model (SCM), which places social groups along two dimensions, warmth and competence. The authors ran three studies. In the first, a stratified US sample of novice annotators rated eight frequently targeted groups on stereotypical traits and then labeled contested posts from the Gab Hate Corpus (GHC). The authors then related each person's stereotypes to how often they chose the hate label and how often they disagreed with others. In the second, warmth and competence scores derived from language were computed for dozens of groups and tested against expert GHC labels and annotator disagreement. In the third, these normative stereotypes were linked to systematic errors made by a trained classifier. Warmth and competence affected judgments differently at each stage, and the classifier reproduced stereotypes about marginalized groups. The authors argue for more diverse annotator pools and for modeling disagreement directly instead of averaging it away.

Key points

  • Study 1 recruited 1,228 US adults via Qualtrics Panels, stratified by gender, age, ethnicity, and political ideology. After quality checks, 857 remained, split roughly evenly between Democrats and Republicans.
  • The stimuli were 56 high-disagreement posts from the GHC (27,665 Gab posts in total), seven for each of eight groups: women, immigrants, Muslims, Jews, communists, liberals, African Americans, and gay people.
  • Warmth was the mean of friendliness, helpfulness, and peacefulness ratings, and competence was the intelligence rating, all on an 8-point semantic differential scale. A Rasch model was used to estimate each annotator's labeling tendency.
  • Groups seen as more competent received more hate labels, and posts about groups seen as less competent produced more disagreement. In Study 2, which covered over 63 groups, higher warmth and higher competence both predicted lower disagreement, and competence predicted fewer hate labels.
  • The proposed remedies are to diversify annotation teams and to treat inter-annotator disagreement as a meaningful signal rather than noise.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Southern CaliforniaLos Angeles · United States162 works · 35,836 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 972 worksBias, fairness & safety · 517 works

Cite

Cite as Aida Mostafazadeh Davani, Mohammad Atari, Brendan Kennedy and Morteza Dehghani. 2023. Hate Speech Classifiers Learn Normative Social Stereotypes. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00550

@article{davani2023hate,
  title = {{Hate Speech Classifiers Learn Normative Social Stereotypes}},
  author = {Aida Mostafazadeh Davani, Mohammad Atari, Brendan Kennedy and Morteza Dehghani},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.1162/tacl\_a\_00550},
  url = {https://doi.org/10.1162/tacl_a_00550}
}

References

83 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Su Lin Blodgett and Brendan O’Connor. 2017. Racial disparity in natural language processing: A case study of social media African American English. In Fairness, Accountability, and Transparency in Machine Learning (FAT/ML) Workshop, KDD.
  2. Zo Ahmed, Bertie Vidgen, and Scott A. Hale. 2022. Tackling racial bias in automated online hate detection: Towards fair and accurate detection of hateful users with geometric deep learning. EPJ Data Science, 11(1):8. https://doi.org/10.1140/epjds/s13688 -022-00319-9 — [doi:10.1140/epjds/s13688](https://doi.org/10.1140/epjds/s13688) ↗
  3. Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam T. Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In Advances in Neural Information Processing Systems, pages 4349–4357.
  4. Sohail Akhtar, Valerio Basile, and Viviana Patti. 2021. Whose opinions matter? Perspective-aware models to identify opinions of hate speech victims in abusive language detection. arXiv preprint arXiv:2106.15896. — [arXiv:2106.15896](https://arxiv.org/abs/2106.15896) find
  5. Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2019. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion Proceedings of the 2019 World Wide Web Conference, pages 491–500. find
  6. Kofi Arhin, Ioana Baldini, Dennis Wei, Karthikeyan Natesan Ramamurthy, and Moninder Singh. 2021. Ground-truth, whose truth?–examining the challenges with annotating toxic text datasets. arXiv preprint arXiv:2112.03529. — [arXiv:2112.03529](https://arxiv.org/abs/2112.03529) find
  7. Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. https://doi.org/10.1126/science.aal4230, PubMed: 28408601 ↗
  8. Lora Aroyo, Lucas Dixon, Nithum Thain, Olivia Redfield, and Rachel Rosen. 2019. Crowdsourcing subjective tasks: The case study of understanding toxicity in online discussions. In Companion Proceedings of The 2019 World Wide Web Conference, pages 1100–1105. https://doi.org/10.1145/3308560 .3317083 — [doi:10.1145/3308560](https://doi.org/10.1145/3308560) ↗
  9. Evelyn R. Carter and Mary C. Murphy. 2015. Group-based differences in perceptions of racism: What counts, to whom, and why? Social and Personality Psychology Compass, 9(6):269–280. https://doi.org/10 .1111/spc3.12181 find
  10. Pinkesh Badjatiya, Manish Gupta, and Vasudeva Varma. 2019. Stereotypical bias removal for hate speech detection task using knowledge-based generalizations. In The World Wide Web Conference, pages 49–59. find
  11. Tessa E. S. Charlesworth, Victor Yang, Thomas C. Mann, Benedek Kurdi, and Mahzarin R. Banaji. 2021. Gender stereotypes in natural language: Word embeddings show robust consistency across child and adult language corpora of more than 65 million words. Psychological Science, 32:218–240. https://doi .o… — [doi:10.1177/0956797620963619](https://doi.org/10.1177/0956797620963619) find
  12. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623. https://doi.org/10.1145 /3442188.3445… find
  13. Yung-Sung Chuang, Mingye Gao, Hongyin Luo, James Glass, Hung-yi Lee, Yun-Nung Chen, and Shang-Wen Li. 2021. Mitigating biases in toxic language detection through invariant rationalization. In Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021), pages 114–120, Online. Association fo…
  14. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Eleventh international aaai conference on web and social media.
  15. Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine Learning, 20(3):273–297. https://doi.org/10.1007/BF00994018 ↗
  16. Gloria Cowan, Miriam Resendez, Elizabeth Marshall, and Ryan Quist. 2002. Hate speech and constitutional protection: Priming values of equality and freedom. Journal of Social Issues, 58(2):247–263. https://doi.org/10.1111/1540-4560.00259 — [doi:10.1111/1540-4560.00259](https://doi.org/10.1111/1540-4560.00259) ↗
  17. J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT.
  18. Kate Crawford. 2017. The trouble with bias. In Conference on Neural Information Processing Systems, invited speaker. find
  19. Amy J. C. Cuddy, Susan T. Fiske, and Peter Glick. 2007. The bias map: Behaviors from intergroup affect and stereotypes. Journal of Personality and Social Psychology, 92(4): 631–648. https://doi.org/10.1037/0022 -3514.92.4.631, PubMed: 17469949 — [doi:10.1037/0022](https://doi.org/10.1037/0022) ↗
  20. Amy J. C. Cuddy, Susan T. Fiske, and Peter Glick. 2008. Warmth and competence as universal dimensions of social perception: The stereotype content model and the bias map. Advances in Experimental Social Psychology, 40:61–149. https://doi.org/10.1016 /S0065-2601(07)00002-0 find
  21. Mark Dı́az, Ian Kivlichan, Rachel Rosen, Dylan Baker, Razvan Amironesei, Vinodkumar Prabhakaran, and Emily Denton. 2022. Crowdworksheets: Accounting for individual and collective identities underlying crowdsourced dataset annotation. In 2022 ACM Conference on Fairness, Accountability, and Transparen… find
  22. Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 67–73. https://doi.org/10 .1145/3278721.3278729 find
  23. Michael Feldman, Sorelle A. Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. 2015. Certifying and removing disparate impact. In proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pages 259–268. find
  24. Paula Czarnowska, Yogarshi Vyas, and Kashif Shah. 2021. Quantifying social biases in NLP: A generalization and empirical comparison of extrinsic fairness metrics. Transactions of the Association for Computational Linguistics, 9:1249–1267. https://doi.org/10 .1162/tacl_a_00425
  25. Susan T. Fiske, Amy J. C. Cuddy, P. Glick, and J. Xu. 2002. A model of (often mixed) stereotype content: Competence and warmth respectively follow from perceived status and competition. Journal of Personality and Social Psychology, 82(6):878. https://doi.org/10.1037/0022-3514.82.6.878 — [doi:10.1037/0022-3514.82.6.878](https://doi.org/10.1037/0022-3514.82.6.878) ↗
  26. Aida Mostafazadeh Davani, Mark Dı́az, and Vinodkumar Prabhakaran. 2022. Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Transactions of the Association for Computational Linguistics, 10:92–110. https://doi.org/10.1162/tacl_a_00449 ↗
  27. Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. https://doi.org/10.1037/h0031619 — [doi:10.1037/h0031619](https://doi.org/10.1037/h0031619) ↗
  28. Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. 2019. Racial bias in hate speech and abusive language detection Gavin Gaffney. 2018. Pushshift gab corpus. https://files.pushshift.io/gab/. Accessed: 2019-5-23. ↗
  29. Social Psychology Bulletin, 31(10):1369–1385. https://doi.org/10.1177/0146167205275613, PubMed: 16143669 — [doi:10.1177/0146167205275613](https://doi.org/10.1177/0146167205275613) ↗
  30. Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. 2018. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences, 115(16):E3635–E3644. https://doi.org/10.1073/pnas.1720347115, PubMed: 29615513 ↗
  31. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with mace. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1120–1130.
  32. Sahaj Garg, Vincent Perot, Nicole Limtiaco, Ankur Taly, Ed H. Chi, and Alex Beutel. 2019. Counterfactual fairness in text classification through robustness. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 219–226. https://doi.org/10.1145/3306618.3317950 — [doi:10.1145/3306618.3317950](https://doi.org/10.1145/3306618.3317950) ↗
  33. Dirk Hovy and Shrimai Prabhumoye. 2021. Five sources of bias in natural language processing. Language and Linguistics Compass, 15(8):e12432. https://doi.org/10 .1111/lnc3.12432, PubMed: 35864931 find
  34. L. Rowell Huesmann, Eric F. Dubow, Paul Boxer, Violet Souweidane, and Jeremy Ginges. 2012. Foreign wars and domestic prejudice: How media exposure to the Israeli-Palestinian conflict predicts ethnic stereotyping by Jewish and Arab American adolescents. Journal of Research on Adolescence, 22(3):556–5… — [doi:10.1111/j.1532-7795](https://doi.org/10.1111/j.1532-7795) find
  35. Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? An investigation of annotator bias in natural language understanding datasets. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conf… find
  36. Ben Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, and Stephen Denuyl. 2020. Social biases in NLP models as barriers for persons with disabilities. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5491–5501, Online. Associ…
  37. Lin Gong, Benjamin Haines, and Hongning Wang. 2017. Clustered model adaption for personalized sentiment analysis. In Proceedings of the 26th International Conference on World Wide Web, pages 937–946. find
  38. Jialun Aaron Jiang, Morgan Klaus Scheuerman, Casey Fiesler, and Jed R. Brubaker. 2021. Understanding international perceptions of the severity of harmful content online. PloS One, 16(8):e0256762. https://doi.org/10.1371/journal.pone.0256762, PubMed: 34449815 ↗
  39. Limor Gultchin, Genevieve Patterson, Nancy Baym, Nathaniel Swinger, and Adam Kalai. 2019. Humor in word embeddings: Cocka-mamie gobbledegook for nincompoops. In International Conference on Machine Learning, pages 2474–2483. PMLR. find
  40. Brendan Kennedy, Mohammad Atari, Aida Mostafazadeh Davani, Leigh Yeh, Ali Omrani, Yehsong Kim, Kris Coombs, Shreya Havaldar, Gwenyth Portillo-Wightman, Elaine Gonzalez, Joe Hoover, Aida Azatian, Alyzeh Hussain, Austin Lara, Gabriel Cardenas, Adam Omary, Christina Park, Xin Wang, Clarisa Wijaya, Yong… find
  41. Moritz Hardt, Eric Price, and Nati Srebro. 2016. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems, pages 3315–3323.
  42. Wilhelm Hofmann, Bertram Gawronski, Tobias Gschwendner, Huy Le, and Manfred Schmitt. 2005. A meta-analysis on the correlation between the implicit association test and explicit self-report measures. Personality and https://doi.org/10.1007/s10579-021 -09569-x Natural Language Processing, volume 2016,… — [doi:10.1007/s10579-021](https://doi.org/10.1007/s10579-021) ↗
  43. Brendan Kennedy, Xisen Jin, Aida Mostafazadeh Davani, Morteza Dehghani, and Xiang Ren. 2020. Contextualizing hate speech classifiers with post-hoc explanation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5435–5442, Online. Association for Computa…
  44. Thomas Manzini, Lim Yao Chong, Alan W. Black, and Yulia Tsvetkov. 2019. Black is to criminal as caucasian is to police: Detecting and removing multiclass bias in word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hum…
  45. Brendan Kennedy, Drew Kogon, Kris Coombs, Joseph Hoover, Christina Park, Gwenyth Portillo-Wightman, Aida Mostafazadeh Davani, Mohammad Atari, and Morteza Dehghani. 2018. A typology and coding manual for the study of hate-based rhetoric. PsyArXiv. July, 18. find
  46. Melissa D. McCradden, Shalmali Joshi, James A. Anderson, Mjaye Mazwi, Anna Goldenberg, and Randi Zlotnik Shaul. 2020. Patient safety and quality improvement: Ethical principles for a regulatory approach to bias in healthcare machine learning. Journal of the American Medical Informatics Association,… find
  47. Svetlana Kiritchenko, Isar Nejadgholi, and Kathleen C. Fraser. 2021. Confronting abusive language online: A survey from the ethical and human rights perspective. Journal of Artificial Intelligence Research, 71:431–478. https://doi.org/10.1613/jair.1.12590 ↗
  48. Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2021. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR), 54(6):1–35. https://doi.org/10.1145/3457607 ↗
  49. Alex Koch, Roland Imhoff, Ron Dotsch, Christian Unkelbach, and Hans Alves. 2016. The abc of stereotypes about groups: Agency/socioeconomic success, conservative–progressive beliefs, and communion. Journal of Personality and Social Psychology, 110(5):675. https://doi.org/10.1037/pspa0000046, PubMed:… — [doi:10.1037/pspa0000046](https://doi.org/10.1037/pspa0000046) ↗
  50. Aida Mostafazadeh Davani, Ali Omrani, Brendan Kennedy, Mohammad Atari, Xiang Ren, and Morteza Dehghani. 2021. Improving counterfactual generation for fair hate speech detection. In Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021), pages 92–101, Online. Association for Computatio… find
  51. Jan Kocoń, Marcin Gruza, Julita Bielaniewicz, Damian Grimling, Kamil Kanclerz, Piotr Miłkowski, and Przemysław Kazienko. 2021. Learning personal human biases and representations for subjective tasks in natural language processing. In 2021 IEEE International Conference on Data Mining (ICDM), pages 1… find
  52. Marzieh Mozafari, Reza Farahbakhsh, and Noël Crespi. 2020. Hate speech detection and racial bias mitigation in social media based on bert model. PloS ONE, 15(8):e0237861. https://doi.org/10.1371/journal.pone.0237861, PubMed: 32853205 ↗
  53. Irene Kwok and Yuzhou Wang. 2013. Locate the hate: Detecting tweets against blacks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 27. https://doi.org/10.1609/aaai.v27i1.8539 — [doi:10.1609/aaai.v27i1.8539](https://doi.org/10.1609/aaai.v27i1.8539) ↗
  54. Michael I. Norton and Samuel R. Sommers. 2011. Whites see racism as a zero-sum game that they are now losing. Perspectives on Psychological Science, 6(3):215–218. https://doi.org/10 .1177/1745691611406922, PubMed: 26168512 find
  55. John P. Lalor, Hao Wu, and Hong Yu. 2016. Building an evaluation scale using item response theory. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Debora Nozza, Claudia Volpetti, and Elisabetta Fersini. 2019. Unintended bias in… find
  56. Lisa Posch, Arnim Bleier, Fabian Flöck, and Markus Strohmaier. 2018. Characterizing the global crowd workforce: A cross-country comparison of crowdworker demographics. In Eleventh International AAAI Conference on Web and Social Media. find
  57. Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. 2019. Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464):447–453. https://doi.org/10 .1126/science.aax2342, PubMed: 31649194 find
  58. Vinodkumar Prabhakaran, Ben Hutchinson, and Margaret Mitchell. 2019. Perturbation sensitivity analysis to detect unintended model biases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processin… find
  59. Ji Ho Park, Jamin Shin, and Pascale Fung. 2018. Reducing gender bias in abusive language detection. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium. Association for Computational Linguistics.
  60. Desmond Patton, Philipp Blandfort, William Frey, Michael Gaskell, and Svebor Karaman. 2019. Annotating social media data from vulnerable populations: Evaluating disagreement between domain experts and graduate student annotators. In Proceedings of the 52nd Hawaii International Conference on System S… find
  61. Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz. 2021. On releasing annotator-level labels and information in datasets. In Proceedings of The Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop, pages 133–138, Punta Cana, Dominican…
  62. Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677–694. https://doi.org/10.1162/tacl_a_00293 ↗
  63. Ashwin Rajadesingan, Reza Zafarani, and Huan Liu. 2015. Sarcasm detection on twitter: A behavioral modeling approach. In Proceedings of the eighth ACM international conference on web search and data mining, pages 97–106. https://doi.org/10.1145/2684822 .2685316 — [doi:10.1145/2684822](https://doi.org/10.1145/2684822) ↗
  64. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, and David Cournapeau. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learni… find
  65. Georg Rasch. 1993. Probabilistic Models for Some Intelligence and Attainment Tests. ERIC. find
  66. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. https://doi.org/10.3115/v1/D14-1162 ↗
  67. Björn Ross, Michael Rist, Guillermo Carbonell, Benjamin Cabrera, Nils Kurowsky, and Michael Wojatzki. 2017. Measuring the reliability of hate speech annotations: The case of the european refugee crisis. In Proceedings of the Workshop on Natural Language Processing for Computer Mediated Communicatio… find
  68. Agnieszka Pietraszkiewicz, Magdalena Formanowicz, Marie Gustafsson Sendén, Ryan L. Boyd, Sverker Sikström, and Sabine Sczesny. 2019. The big two dictionaries: Capturing agency and communion in natural Paul Rottger, Bertie Vidgen, Dirk Hovy, and Janet Pierrehumbert. 2022. Two contrasting data annot…
  69. Ameya Vaidya, Feng Mai, and Yue Ning. 2020. Empirical analysis of multi-task learning for reducing identity bias in toxic comment detection. In Proceedings of the International AAAI Conference on Web and Social Media, volume 14, pages 683–693. find
  70. Bertie Vidgen, Tristan Thrush, Zeerak Talat, and Douwe Kiela. 2021. Learning from the worst: Dynamically generated datasets to improve online hate detection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natu…
  71. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The riskofracial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668–1678. find
  72. Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah Smith. 2022. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguis…
  73. Claudia Wagner, Markus Strohmaier, Alexandra Olteanu, Emre Kıcıman, Noshir Contractor, and Tina Eliassi-Rad. 2021. Measuring algorithmically infused societies. Nature, 595(7866):197–204. https://doi.org/10.1038/s41586 -021-03666-1, PubMed: 34194046 ↗
  74. Steven E. Stemler and Adam Naples. 2021. Rasch measurement v. item response theory: Knowing when to cross the line. Practical Assessment, Research & Evaluation, 26:11. find
  75. Maximilian Wich, Hala Al Kuwatly, and Georg Groh. 2020. Investigating annotator bias with a graph-based approach. In Proceedings of the fourth workshop on online abuse and harms, pages 191–199. find
  76. Nathaniel Swinger, Maria De-Arteaga, Neil Thomas Heffernan IV, Mark DM Leiserson, and Adam Tauman Kalai. 2019. What are the biases in my word embedding? In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 305–311. https://doi.org/10.1145/3306618.3314270 — [doi:10.1145/3306618.3314270](https://doi.org/10.1145/3306618.3314270) ↗
  77. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe… — Transformers: State-of-the-Art Natural Language Processing
  78. Zeerak Talat. 2016. Are you a racist or am I seeing things? Annotator influence on hate speech detection on twitter. In Proceedings of the first workshop on NLP and computational social science, pages 138–142.
  79. Zeerak Talat, Smarika Lulz, Joachim Bingel, and Isabelle Augenstein. 2021. Disembodied machine learning: On the illusion of objectivity in NLP. Anonymous preprint under review. find
  80. Alexandra N. Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385–1470. https://doi.org/10.1613/jair.1.12752 — [doi:10.1613/jair.1.12752](https://doi.org/10.1613/jair.1.12752) ↗
  81. Mengzhou Xia, Anjalie Field, and Yulia Tsvetkov. 2020. Demoting racial bias in hate speech detection. In Proceedings of the Eighth International Workshop on Natural Language Processing for Social Media, pages 7–14, Online. Association for Computational Linguistics. find
  82. Huhhot, China. Chinese Information Processing Society of China. find
  83. Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. 2021. A robustly optimized BERT pre-training approach with post-training. In Proceedings of the 20th Chinese National Conference on Computational Linguistics, pages 1218–1227, Linda X. Zou and Sapna Cheryan. 2017. Two axes of subordination: A new model of…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/davani-2023-hate-speech-classifiers-learn-normative/