Elamur Research
☰
home / papers / 2022 / ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection

ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2203.09509

Summary

Classifiers that detect toxic language often treat any mention of a minority group as a sign of hate. As a result, they flag harmless statements and miss subtle hate that contains no slurs or profanity. To address this, the authors built ToxiGen, a corpus of 274,186 statements about 13 minority groups, generated with GPT-3. They prompted GPT-3 with example sets of hateful or benign sentences. They also introduced ALICE, a decoding method that places a toxicity classifier inside constrained beam search. ALICE steers the output so that toxic text looks harmless to the classifier and benign text looks hateful. The resulting data is almost entirely implicit and has equal numbers of toxic and benign statements for each group, so group mentions alone do not predict the label. In human evaluation, annotators usually could not tell the generated text from human writing. Fine-tuning existing classifiers on ToxiGen improved their results on three human-written benchmarks. The work shows that large language models can be used to build better hate speech detectors, despite their own tendency to produce toxic text.

Key points

  • Dataset scale and balance: ToxiGen contains 274,186 statements, with roughly 135k toxic and 135k benign, covering 13 groups such as Native Americans and LGBTQ+ people. Of these, 98.2% are implicit, and the hate class makes up about 50%.
  • ALICE attacks: ALICE (Adversarial Language Imitation with Constrained Exemplars) fooled five public classifiers: Perspective API, HateBERT, the OpenAI content filter, AI2 Delphi and a RoBERTa model.
  • Human study: In a sample of 792 generated sentences, about 90% were judged to be human-written. Annotators labeled 94.5% of the toxic examples as hate speech.
  • Downstream gains: Fine-tuning HateBERT and ToxDectRoBERTa on ToxiGen raised performance by 7–19% on ImplicitHateCorpus, SocialBiasFrames and DynaHate. It also helped on the human-validated ToxiGen evaluation subset.
  • Release and caveats: The code, data, two fine-tuned models and the human annotations are available at github.com/microsoft/ToxiGen. The authors warn that the data may be noisy and that the methods could be misused to produce hate speech.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

316 papers of our library cite it (of the 140673 whose bibliographies we read); 22 of them have a page here, 294 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2203.09509 · arXiv 2203.09509

@misc{hartvigsen2022toxigen,
  title = {{ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection}},
  author = {Hartvigsen, Thomas and Gabriel, Saadia and Palangi, Hamid and Sap, Maarten and Ray, Dipankar and Kamar, Ece},
  year = {2022},
  doi = {10.48550/arxiv.2203.09509},
  eprint = {2203.09509},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2203.09509}
}

References

68 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2017. Guided open vocabulary image captioning with constrained beam search. In EMNLP. find
  2. Carolina Are. 2020. How instagram’s algorithm is censoring women and vulnerable users but helping online abusers. Feminist media studies, 20(5):741– 744. find
  3. Elizabeth Behm-Morawitz and Dana E Mastro. 2008. Mean girls? the influence of gender portrayals in teen movies on emerging adults’ Gender-Based attitudes and beliefs. Journalism & mass communication quarterly, 85(1):131–146. find
  4. Susan Benesch. 2020. Proposals for improved regulation of harmful online content. Technical report. find
  5. Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow. If you use this software, please cite it using these metadata. find
  6. Luke Breitfeller, Emily Ahn, David Jurgens, and Yulia Tsvetkov. 2019. Finding microaggressions in the wild: A case for locating elusive phenomena in social media posts. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferenc… find
  7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… find
  8. Tommaso Caselli, Valerio Basile, Jelena Mitrovic, and M. Granitzer. 2021. Hatebert: Retraining bert for abusive language detection in english. ArXiv, abs/2010.12472. find
  9. S Cheryan and G V Bodenhausen. 2000. When positive stereotypes threaten intellectual performance: the psychological hazards of “model minority” status. Psychological science, 11(5):399–402. find
  10. Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A. Smith. 2021. All that’s ‘human’ is not gold: Evaluating human evaluation of generated text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International… find
  11. Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. 2022. Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations. Transactions of the Association for Computational Linguistics, 10:92–110. find our card
  12. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Proceedings of the International AAAI Conference on Web and Social Media, volume 11. find our card
  13. Thiago Dias Oliva, Dennys Marcelo Antonialli, and Alessandra Gomes. 2020. Fighting hate speech, silencing drag queens? artificial intelligence in content moderation and risks to LGBTQ voices online. Sexuality & culture. find
  14. Ángel Díaz and Laura Hecht-Felella. 2021. Double standards in social media content moderation. Technical report, Brennan Center for Justice at New York University School of Law. find
  15. Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019. Build it break it fix it for dialogue safety: Robustness from adversarial human attack. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Nat… find our card
  16. Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 67–73. find
  17. Yao Dou, Maxwell Forbes, Rik Koncel-Kedziorski, Noah A Smith, and Yejin Choi. 2021. Scarecrow: A framework for scrutinizing machine text. arXiv preprint arXiv:2107.01294. — [arXiv:2107.01294](https://arxiv.org/abs/2107.01294) find
  18. Mai ElSherief, Caleb Ziems, David Muchlinski, Vaishnavi Anupindi, Jordyn Seybolt, Munmun De Choudhury, and Diyi Yang. 2021. Latent hatred: A benchmark for understanding implicit hate speech. arXiv preprint arXiv:2109.05322. find our card
  19. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898. find
  20. Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76:378–382. beam search. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1535–1546, Vancouver, Canada. Association for Comput… find
  21. Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. 2018. Learning to write with cooperative discriminators. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1638–1649, Melbourne, Australia. A… find our card
  22. Joseph Hoover, Mohammad Atari, Aida M Davani, Brendan Kennedy, Gwenyth Portillo-Wightman, Leigh Yeh, Drew Kogon, and Morteza Dehghani. 2019. Bound in hatred: The role of group-based morality in acts of hate. find
  23. David L Hudson, Jr. 2017. Hate speech online. https: find
  24. Antigoni Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large scale crowdsourcing and characterization of twitter abusive behavior. In Twelfth International AAAI Confere… find
  25. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In ACL. find our card
  26. Sam Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. In Findings of EMNLP. find our card
  27. Ona de Gibert, Naiara Perez, Aitor García-Pablos, and Montse Cuadros. 2018. Hate speech dataset from a white supremacy forum. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2), pages 11–20. find our card
  28. Tarleton Gillespie, Patricia Aufderheide, Elinor Carmi, Ysabel Gerrard, Robert Gorwa, Ariadna Matamoros-Fernandez, Sarah T Roberts, Aram Sinnreich, and Sarah Myers West. 2020. Expanding the debate about content moderation: Scholarly research agendas for the coming policy debates. Internet Policy Rev… find
  29. Peter Glick and Susan T Fiske. 1996. The ambivalent sexism inventory: Differentiating hostile and benevolent sexism. Journal of personality and social psychology, 70(3):491. find
  30. Xiaochuang Han and Yulia Tsvetkov. 2020. Fortifying toxic speech detectors against veiled toxicity. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7732–7739, Online. Association for Computational Linguistics. find
  31. Chris Hokamp and Qun Liu. 2017. Lexically constrained decoding for sequence generation using grid //web.archive.org/web/20211115012316/ https://www.freedomforuminstitute.org/ first-amendment-center/topics/freedom-of-speech-2/internet-first-amendment/ hate-speech-online/ . Accessed: 2021-11-14. ↗
  32. Jonathan W Kanter, Monnica T Williams, Adam M Kuczynski, Katherine E Manbeck, Marlena De-breaux, and Daniel C Rosen. 2017. A preliminary report on the relationship between microaggressions against black people and racism among white college students. Race and social problems, 9(4):291– 299. find
  33. Brendan Kennedy, Mohammad Atari, Aida Mostafazadeh Davani, Leigh Yeh, Ali Omrani, Yehsong Kim, Kris Coombs, Shreya Havaldar, Gwenyth Portillo-Wightman, Elaine Gonzalez, et al. 2018. The gab hate corpus: A collection of 27k posts annotated for hate speech. find
  34. Ben Krause, Akhilesh Deepak Gotmare, Bryan Mc-Cann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. 2020. Gedi: Generative discriminator guided sequence generation. arXiv preprint arXiv:2009.06367. find our card
  35. Klaus Krippendorff. 1980. Content analysis: an introduction to its methodology. find
  36. Sumit Kumar and Raj Ratn Pranesh. 2021. Tweetblm: A hate speech dataset and analysis of black lives matter-related microblogs on twitter. arXiv preprint arXiv:2108.12521. — [arXiv:2108.12521](https://arxiv.org/abs/2108.12521) find
  37. Elisa Leonardelli, Stefano Menini, Alessio Palmero Aprosio, Marco Guerini, and Sara Tonelli. 2021. Agreeing to disagree: Annotating offensive language datasets with annotators’ disagreement. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10528–10539,… find
  38. Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. 2021a. Dexperts: Decoding-time controlled text generation with experts and anti-experts. In ACL. find our card
  39. Björn Ross, Michael Rist, Guillermo Carbonell, Benjamin Cabrera, Nils Kurowsky, and Michael Wojatzki. 2017. Measuring the reliability of hate speech annotations: The case of the european refugee crisis. ArXiv, abs/1701.08118. — [arXiv:1701.08118](https://arxiv.org/abs/1701.08118) find
  40. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021b. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06804. find our card
  41. Paul Röttger, Bertie Vidgen, Dong Nguyen, Zeerak Waseem, Helen Margetts, and Janet Pierrehumbert. 2021. HateCheck: Functional tests for hate speech detection models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference… find our card
  42. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021c. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586. find our card
  43. Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Neuro-Logic decoding: (un)supervised neural text generation with predicate logic constraints. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Lingu… find
  44. Sean MacAvaney, Hao-Ren Yao, Eugene Yang, Katina Russell, Nazli Goharian, and Ophir Frieder. 2019. Hate speech detection: Challenges and solutions. PloS one, 14(8):e0221152. find
  45. RWJF. ica: 2017. Discrimination experiences and views. in amer-https: //www.rwjf.org/en/library/research/ 2017/10/discrimination-in-america-- experiences-and-views.html . Accessed: 2019-11-5. find
  46. Muniba Saleem and Craig A Anderson. 2013. Arabs as terrorists: Effects of stereotypes within violent contexts on attitudes, perceptions, and affect. Psychology of violence, 3(1):84–99. find
  47. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A Smith. 2019. The risk of racial bias in hate speech detection. In ACL. find our card
  48. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2021. Natural instructions: Benchmarking generalization to new tasks from natural language instructions. arXiv preprint arXiv:2104.08773. find
  49. Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power implications of language. In ACL. find our card
  50. Kevin L Nadal. 2018. Microaggressions and traumatic stress: Theory, research, and clinical treatment. American Psychological Association. find
  51. Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2021. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. find our card
  52. Kevin L Nadal, Katie E Griffin, Yinglee Wong, Sahran Hamit, and Morgan Rasmus. 2014. The impact of racial microaggressions on mental health: Counseling implications for clients of color. Journal of counseling and development: JCD, 92(1):57–66. find
  53. Alexandra Olteanu, Carlos Castillo, Jeremy Boy, and Kush R. Varshney. 2018. The effect of extremist violence on hateful speech online. In ICWSM. find
  54. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505– 3506. find
  55. Rob Reich, Mehran Sahami, and Jeremy M Weinstein. 2021. System error: Where big tech went wrong and how we can reboot. Hodder & Stoughton. find
  56. Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. The woman worked as a babysitter: On biases in language generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process… find our card
  57. Wiktor Soral, Michał Bilewicz, and Mikołaj Winiewski. 2018. Exposure to hate speech increases prejudice through desensitization. Aggressive behavior, 44(2):136–146. find
  58. Derald Wing Sue, Christina M Capodilupo, Gina C Torino, Jennifer M Bucceri, Aisha M B Holder, Kevin L Nadal, and Marta Esquilin. 2007. Racial microaggressions in everyday life: implications for clinical practice. The American psychologist, 62(4):271–286. find
  59. Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. 2021. Learning from the worst: Dynamically generated datasets to improve online hate detection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Nat… find our card
  60. Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/ mesh-transformer-jax . ↗
  61. Zeerak Waseem. 2016. Are you a racist or am i seeing things? annotator influence on hate speech detection on twitter. In Proceedings of the first workshop on NLP and computational social science, pages 138– 142. find
  62. Michael Wiegand, Josef Ruppenhofer, and Elisabeth Eder. 2021. Implicitly abusive language–what does it actually look like and why are we not getting there? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologie… find
  63. Thomas Wischmeyer and Timo Rademacher, editors. 2020. Regulating Artificial Intelligence. Springer, Cham. find
  64. Albert Xu, Eshaan Pathak, Eric Wallace, Suchin Gururangan, Maarten Sap, and Dan Klein. 2021. Detoxifying language models risks marginalizing minority voices. In NAACL. find
  65. Kevin Yang and Dan Klein. 2021. FUDGE: Controlled text generation with future discriminators. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3511–3535, Online. Association for Computational Ling… find our card
  66. Danyaal Yasin. 2018. Black and banned: Who is free speech for? https://www.indexoncensorship. org/2018/09/black-and-banned-who-is-free-speech-for/ . Accessed: 2018-12-6. ↗
  67. Marcos Zampieri, S. Malmasi, Preslav Nakov, Sara Rosenthal, N. Farra, and Ritesh Kumar. 2019. Predicting the type and target of offensive posts in social media. In NAACL. find our card
  68. Xuhui Zhou, Maarten Sap, Swabha Swayamdipta, Yejin Choi, and Noah A. Smith. 2021. Challenges in automated debiasing for toxic language detection. In EACL. find