Elamur Research
☰
home / papers / 2021 / Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection

Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2021.acl-long.132.pdf

Summary

Hate speech classifiers trained on static benchmarks often lack robustness. They overfit on slurs and identity terms, miss implicit or obfuscated abuse, and generalise poorly to new settings. Vidgen et al. address this with a human-and-model-in-the-loop pipeline. Over four rounds, trained annotators wrote entries designed to fool the current target model. A new model was then trained on the accumulated data and placed back in the loop.

From the second round on, half of the entries were "contrast set" perturbations: small edits to an original text that flip its label. In later rounds, annotators also drew on real hate sites and focused on specific kinds of hate. The resulting synthetic dataset has about 40,000 entries. Each hateful item is labelled for the type and target of hate.

As rounds progressed, models became harder to trick, scored better on the test sets and improved markedly on the HateCheck functional tests. The work matters because it shows that adaptive, adversarial data collection produces sturdier detectors than fixed corpora. The data, code and guidelines are publicly released.

Key points

  • The dataset has about 40,000 entries, roughly 15,000 of them perturbations. About 54% are hateful, a much higher share than in comparable corpora.
  • On HateCheck, the target models' accuracy rose from 60% (Round 1) to 95% (Round 4). The annotators' success rate at fooling the models dropped as rounds advanced.
  • Hateful entries carry secondary labels for type: derogation, animosity, threatening language, support for hateful entities and dehumanization. They also carry target labels, based on a non-exhaustive list of 29 identities plus some intersectional groups. Men, white people and heterosexuals were treated as out of scope.
  • RoBERTa served as the in-the-loop model. Twenty annotators worked over four months and were paid £16 per hour.
  • Resources are released at github.com/bvidgen/Dynamically-Generated-Hate-Speech-Dataset.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

154 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 146 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.acl-long.132

@inproceedings{vidgen2021learning,
  title = {{Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection}},
  author = {Vidgen, Bertie and Thrush, Tristan and Waseem, Zeerak and Kiela, Douwe},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.132},
  url = {https://doi.org/10.18653/v1/2021.acl-long.132}
}

References

60 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020. Evaluating NLP Models’ Local Decision Boundaries via Contrast Sets. In Findings of the Assocation for Computational Linguistics: EMNLP 2020, pages 1307–1323. find
  2. Maria Anzovino, Elisabetta Fersini, and Paolo Rosso. 2018. Automatic identification and classification of misogynistic language on Twitter. In NLDB, pages 57–64. find
  3. Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. Sem Eval-2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter. In Proceedings of the 13th International Wo… find
  4. Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604. find
  5. Tommaso Caselli, Valerio Basile, Jelena Mitrović, Inga Kartoziya, and Michael Granitzer. 2020. I Feel Offended, Don’t Be Abusive! Implicit/Explicit Messages in Offensive and Abusive Language. In Proceedings of the 12th Conference on Language Resources and Evaluation (LREC), pages 6193–6202. find
  6. Yi-Ling Chung, Elizaveta Kuzmenko, Serra Sinem Tekiroglu, and Marco Guerini. 2019. CONAN -COunter NArratives through Nichesourcing: a Multilingual Dataset of Responses to Fight Online Hate Speech. In Proceedings of the 57th Annual Meeting of the ACL, pages 2819–2829. find our card
  7. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated Hate Speech Detection and the Problem of Offensive Language. In Proceedings of the 11th ICWSM, pages 1–4. find our card
  8. Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019. Build it break it fix it for dialogue safety: Robustness from adversarial human attack. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, pages 4537–4546. find our card
  9. Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and Mitigating Unintended Bias in Text Classification. In AAAI/ACM Conference on AI, Ethics, and Society, pages 67–73. find
  10. Antigoni-maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior. In ICWSM, pages 1–11. find
  11. Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hanna Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Sahaj Garg… find
  12. Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conf… find
  13. Jennifer Golbeck, Alicia A. Geller, Jayanee Thanki, Shalmali Naik, Kelly M. Hoffman, Derek Michael Wu, Alexandra Berlinger, Priyanka Vengataraman, Shivika Khare, Zahra Ashktorab, Marianna J. Martindale, Gaurav Shahane, Paul Cheakalos, Jenny Hottle, Siddharth Bhagwan, Raja Rajan Gunasekaran, Rajesh K… find
  14. Tommi Gröndahl, Luca Pajola, Mika Juuti, Mauro Conti, and N. Asokan. 2018. All You Need is ”Love”: Evading Hate-speech Detection. In Proceedings of the 11th ACM Workshop on Artificial Intelligence and Security, pages 2–12. find
  15. Kevin A Hallgren. 2012. Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutorials in quantitative methods for psychology, 8(1):23–34. find
  16. Hugo Lewi Hammer. 2014. Detecting threats of violence in online discussions using bigrams of important words. In Proceedings - 2014 IEEE Joint Intelligence and Security Informatics Conference, JISIC 2014, page 319. find
  17. N. Haslam and M. Stratemeyer. 2016. Recent research on dehumanization. Current Opinion in Psychology, 11:25–29. find
  18. Hossein Hosseini, Sreeram Kannan, Baosen Zhang, and Radha Poovendran. 2017. Deceiving Google’s Perspective API Built for Detecting Toxic Comments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 1305–1309. find
  19. Mladen Karan and Jan Šnajder. 2018. Cross-Domain Detection of Abusive Language Online. In 2nd Workshop on Abusive Language Online, pages 132– 137. find
  20. Divyansh Kaushik, Eduard Hovy, and Zachary C Lipton. 2019. Learning the difference that makes a difference with counterfactually-augmented data. arXiv preprint arXiv:1909.12434. find
  21. Brendan Kennedy, Xisen Jin, Aida Mostafazadeh Davani, Morteza Dehghani, and Xiang Ren. 2020. Contextualizing Hate Speech Classifiers with Post-hoc Explanation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5435–5442. find
  22. Ritesh Kumar, Atul Kr Ojha, Shervin Malmasi, and Marcos Zampieri. 2018. Benchmarking Aggression Identification in Social Media. In Proceedings ofthe First Workshop on Trolling, Aggression and Cyberbullying,, 1, pages 1–11. find
  23. Jana Kurrek, Haji Mohammad Saleem, and Derek Ruths. 2020. Towards a Comprehensive Taxonomy and Large-Scale Annotated Corpus for Online Slur Usage. In Proceedings of the Fourth Workshop on Online Abuse and Harms, pages 138–149. find
  24. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv:1907.11692. find
  25. Thomas Mandl, Sandip Modha, Prasenjit Majumder, Daksh Patel, Mohana Dave, Chintak Mandlia, and Aditya Patel. 2019. Overview of the HASOC track at FIRE 2019: Hate speech and offensive content identification in Indo-European languages. In FIRE ’19: Proceedings of the 11th Forum for Information Retriev… find
  26. Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. 2020. Hatexplain: A benchmark dataset for explainable hate speech detection. find
  27. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885–4901, Online. Association for Computa… find
  28. Chikashi Nobata, Achint Thomas, Yashar Mehdad, Yi Chang, and Joel Tetreault. 2016. Abusive Language Detection in Online User Content. In World Wide Web Conference, pages 145–153. find
  29. Alexis Palmer, Christine Carr, Melissa Robinson, and Jordan Sanders. 2020. COLD: Annotation scheme and evaluation data set for complex offensive language in English. The Journal for Language Technology and Computational Linguistics, 34(1):1–28. find
  30. Endang Pamungkas, Valerio Basile, and Viviana Patti. 2020. Misogyny Detection in Twitter: a Multilingual and Cross-Domain Study. Information Processing and Management, 57(6). find
  31. Georgios K. Pitsilis, Heri Ramampiaro, and Helge Langseth. 2018. Detecting Offensive Language in Tweets Using Deep Learning. arXiv:1801.04433v1, pages 1–17. — [arXiv:1801.04433](https://arxiv.org/abs/1801.04433) find
  32. Fabio Poletto, Valerio Basile, Manuela Sanguinetti, Cristina Bosco, and Viviana Patti. 2020. Resources and benchmark corpora for hate speech detection: a systematic review. Language Resources and Evaluation, pages 1–47. find
  33. Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902– 4912, Online. Association for Computational Linguisti… find
  34. Paul Röttger, Bertram Vidgen, Dong Nguyen, Zeerak Waseem, Helen Margetts, and Janet Pierrehumbert. 2020. Hatecheck: Functional tests for hate speech detection models. find our card
  35. Julia Mendelsohn, Yulia Tsvetkov, and Dan Jurafsky. 2020. A Framework for the Computational Linguistic Analysis of Dehumanization. Frontiers in Artificial Intelligence, 3(August):1–24. find
  36. Joni Salminen, Maximilian Hopf, Shammur A. Chowdhury, Soon gyo Jung, Hind Almerekhi, and Bernard J. Jansen. 2020. Developing an online hate classifier for multiple social media platforms. Human-centric Computing and Information Sciences, 10(1):1–34. find
  37. Pushkar Mishra, Helen Yannakoudakis, and Ekaterina Shutova. 2019. Tackling online abuse: A survey of automated abuse detection methods. arXiv:1908.06024v2, pages 1–17. — [arXiv:1908.06024](https://arxiv.org/abs/1908.06024) find
  38. Mattia Samory, Indira Sen, Julian Kohne, Fabian Flock, and Claudia Wagner. 2020. Unsex me here: Revisiting sexism detection using psychological scales and adversarial samples. arXiv preprint:2004.12764v1, pages 1–11. — [arXiv:2004.12764](https://arxiv.org/abs/2004.12764) find
  39. Robert C Nickerson, Upkar Varshney, and Jan Muntermann. 2013. A method for taxonomy development and its application in information systems. European Journal of Information Systems, 22(3):336– 359. find
  40. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, Noah A Smith, and Paul G Allen. 2019. The Risk of Racial Bias in Hate Speech Detection. In Proceedings of the 57th Annual Meeting of the ACL, pages 1668–1678. find our card
  41. Anna Schmidt and Michael Wiegand. 2017. A Survey on Hate Speech Detection using Natural Language Processing. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, pages 1–10. Association for Computational Linguistics. find our card
  42. Steve Durairaj Swamy, Anupam Jamatia, and Björn Gambäck. 2019. Studying generalisability across abusive language detection datasets. In CoNLL 2019 - 23rd Conference on Computational Natural Language Learning, Proceedings of the Conference, pages 940–950. find
  43. Bertie Vidgen, Austin Botelho, David Broniatowski, Ella Guest, Matthew Hall, Helen Margetts, Rebekah Tromble, Zeerak Waseem, and Scott Hale. 2020. Detecting East Asian Prejudice on Social Media. arXiv:2005.03909v1, pages 1–12. find
  44. Bertie Vidgen and Leon Derczynski. 2020. Directions in Abusive Language Training Data: Garbage In, Garbage Out. Plos ONE, pages 1–26. find
  45. Zeerak Waseem and Dirk Hovy. 2016. Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter. In NAACL-HLT, pages 88–93. find
  46. Zeerak Waseem, James Thorne, and Joachim Bingel. 2018. Bridging the gaps: Multi task learning for domain transfer of hate speech detection. In Jennifer Golbeck, editor, Online Harassment, pages 29–55. Springer International Publishing, Cham. find
  47. Michael Wiegand, Josef Ruppenhofer, and Thomas Kleinbauer. 2019. Detection of Abusive Language: the Problem of Biased Datasets. In NAACL-HLT, pages 602–608, Minneapolis. ACL. find our card
  48. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. find our card
  49. Bertie Vidgen, Alex Harris, Dong Nguyen, Rebekah Tromble, Scott Hale, and Helen Margetts. 2019a. Challenges and frontiers in abusive content detection. In Proceedings of the Third Workshop on Abusive Language Online (ACL), pages 80–93. find
  50. Lucas Wright, Derek Ruths, Kelly P Dillon, Haji Mohammad Saleem, and Susan Benesch. 2017. Vectors for counterspeech on Twitter. In Proceedings of the First Workshop on Abusive Language Online, pages 57–62, Vancouver, BC, Canada. Association for Computational Linguistics. find
  51. Bertie Vidgen, Helen Margetts, and Alex Harris. 2019b. How much online abuse is there? A systematic review of evidence for the UK. The Alan Turing Institute, London. find
  52. Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017a. Ex Machina: Personal Attacks Seen at Scale. In Proceedings of the International World Wide Web Conference, pages 1391–1399. find
  53. Bertie Vidgen, Dong Nguyen, Helen Margetts, Patricia Rossini, and Rebekah Tromble. 2021. Introducing CAD: the contextual abuse dataset. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2289–2303,… find
  54. Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017b. Ex Machina: Personal Attacks Seen at Scale. In Proceedings of the International World Wide Web Conference, pages 1391–1399. find
  55. Bertie Vidgen and Taha Yasseri. 2019. Detecting weak and strong Islamophobic hate speech on social media. Journal of Information Technology & Politics, 17(1):66–78. find
  56. William Warner and Julia Hirschberg. 2012. Detecting hate speech on the world wide web. In Proceedings of the Second Workshop on Language in Social Media, pages 19–26. find
  57. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. Predicting the type and target of offensive posts in social media. In Proceedings of NAACL HLT 2019, volume 1, pages 1415–1420. find our card
  58. Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Çağrı Çöltekin. 2020. SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020). arXiv preprint, pages 1–23. find
  59. Zeerak Waseem. 2016. Are you a racist or am I seeing things? annotator influence on hate speech detection on twitter. In Proceedings of the First Workshop on NLP and Computational Social Science, pages 138– 142, Austin, Texas. Association for Computational Linguistics. find
  60. Zeerak Waseem, Thomas Davidson, Dana Warmsley, and Ingmar Weber. 2017. Understanding Abuse: A Typology of Abusive Language Detection Subtasks. In Proceedings of the First Workshop on Abusive Language Online, pages 78–84. find our card