Elamur Research⌕
☰
home / papers / 2018 / Challenges for Toxic Comment Classification: An In-Depth Error Analysis

Challenges for Toxic Comment Classification: An In-Depth Error Analysis

Paper · 2018 · Betty van Aken, Julian Risch, Ralf Krestel et al.

Summary

Automatic detection of toxic online comments still fails on many real-world cases. The authors ask which difficulties current classifiers already handle and which remain open. They train a range of shallow and neural models on two corpora: the multi-label Wikipedia talk page data that Google Jigsaw released for a Kaggle challenge, and the multi-class Twitter hate speech set of Davidson et al. (2017). The models are Logistic Regression over word and character n-grams, a CNN, bidirectional LSTM and GRU networks, and a GRU with attention, with the neural models using pretrained word embeddings. The models' predictions overlap only partly, so combining them into an ensemble raises macro-averaged F1. The gain is largest on rare classes and on heterogeneous data. The central contribution is a manual analysis of the ensemble's false negatives and false positives, sorted into error classes. It shows that all approaches share the same weaknesses:

  • inconsistent annotation,
  • rare or obfuscated vocabulary,
  • harmless use of swear words,
  • quoted abuse,
  • sarcasm.

This points future work toward cleaner labels and representations that capture context and world knowledge.

Key points

  • The Wikipedia (Jigsaw/Kaggle) corpus has 223,549 comments with six overlapping labels: toxic, severe toxic, obscene, threat, insult and identity hate. Of these, 201,081 comments carry no label. The toxic class covers 9.6% of the data, and threat covers only 0.3%.
  • The Twitter dataset of Davidson et al. (2017) has 24,783 tweets in three mutually exclusive classes: hate, offensive and neither. Offensive tweets dominate at 77.4% because the tweets were collected with Hatebase.org seed keywords.
  • Pearson correlation between the models' predictions, together with their F1 scores, shows that each model tends to get different examples wrong. Mixing shallow learners with deep networks in the ensemble proved especially effective.
  • Results for false negatives (Wikipedia / Twitter):
    • questionable labels: 23% / 9%
    • rare or unknown words: 30% / 43%
    • quotations of or references to toxic language: 17% / 8%
    • sarcasm or irony: 11% / none
  • Results for false positives (Wikipedia / Twitter):
    • doubtful labels: 53% on Wikipedia (comments marked clean that are arguably toxic) / 10% on Twitter
    • swear words in non-toxic text: 60% / 77%
    • rare words: 8% / 17%

Interesting ideas

  • I1. Diverse ensembles: Build the ensemble from architectures that fail in different ways, and check this by correlating their predictions. Picking members for complementary errors rather than raw accuracy suits any imbalanced classification task.
  • I2. Error classes for false positives and negatives: Sample misclassified items from both directions and tag them with error classes, which makes vague "model weaknesses" measurable. The same protocol could be reused to audit Ukrainian NLP classifiers.
  • I3. Auditing gold labels: Include "doubtful label" as an explicit error class. This shows how much apparent model error is really annotation noise, which argues for tracking annotator agreement and writing clear class definitions when building datasets.
  • I4. Paradigmatic context: The concept explains failures such as moderator remarks that quote abuse or self-deprecating swearing. It is a useful lens for any task where trigger words appear in harmless frames.
  • I5. Cross-domain validation: Test the same pipeline on a second dataset with a different domain and task format (multi-label vs. multi-class). This cheaply checks whether conclusions generalise beyond one corpus.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of PotsdamPotsdam · Germany36 works · 5,236 citations🔬Hasso Plattner InstitutePotsdam · Germany7 works · 4,985 citations🎓Berliner Hochschule für TechnikBerlin · Germany5 works · 670 citations

📚 Shelves 2 Computer Science · 9,086 works Artificial Intelligence · 8,889 works

🧭 Topics 1 Bias, fairness & safety · 209 works

Cite

Cite as Betty van Aken, Julian Risch, Ralf Krestel and Alexander Löser. 2018. Challenges for Toxic Comment Classification: An In-Depth Error Analysis. Proceedings of the 2nd Workshop on Abusive Language Online (ALW2).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-5105

@inproceedings{aken2018challenges,
  title = {{Challenges for Toxic Comment Classification: An In-Depth Error Analysis}},
  author = {Betty {van Aken}, Julian Risch, Ralf Krestel and Alexander Löser},
  booktitle = {Proceedings of the 2nd Workshop on Abusive Language Online (ALW2)},
  year = {2018},
  doi = {10.18653/v1/w18-5105},
  url = {https://doi.org/10.18653/v1/w18-5105}
}

References

43 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pinkesh Badjatiya, Shashank Gupta, Manish Gupta, and Vasudeva Varma. 2017. Deep learning for hate speech detection in tweets. In WWW. find
  2. Cynthia Van Hee, Els Lefever, Ben Verhoeven, Julie Mennes, Bart Desmet, Guy De Pauw, Walter Daelemans, and Véronique Hoste. 2015. Detection and fine-grained classification of cyberbullying events. In RANLP. find
  3. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. TACL, 5:135–146. our card
  4. Akshita Jha and Radhika Mamidi. 2017. When does a compliment become sexist? analysis and classification of ambivalent sexism using twitter data. In NLP+CSS@ACL. find
  5. Pete Burnap and Matthew L. Williams. 2015. Cyber hate speech on twitter : An application of machine classification and statistical modeling for policy and decision making. volume 7, pages 223–242. find
  6. George W. Kennedy, Andrew W. McCollough, Edward Dixon, A. M. Parra Bastidas, J. Mark Ryan, and Chris Loo. 2017. Hack harassment : Technology solutions to combat online harassment. In ALW1@ACL. find
  7. Pete Burnap and Matthew L. Williams. 2016. Us and them: identifying cyber hate on twitter across multiple protected characteristics. EPJ Data Science, 5:1–15. find
  8. Ying Chen, Yilu Zhou, Sencun Zhu, and Heng Xu. 2012. Detecting offensive language in social media to protect adolescent online safety. SOCIALCOM-PASSAT, pages 71–80. find
  9. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In EMNLP. our card
  10. Irene Kwok and Yuzhou Wang. 2013. Locate the hate: Detecting tweets against blacks. In AAAI. find
  11. Maral Dadvar, Dolf Trieschnigg, Roeland Ordelman, and Franciska de Jong. 2013. Improving cyberbullying detection with user context. In ECIR. find
  12. Yashar Mehdad and Joel R. Tetreault. 2016. Do characters abuse more than words? In SIGDIAL. find
  13. Thomas Davidson, Dana Warmsley, Michael W. Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In ICWSM. our card
  14. Dennis Njagi, Z Zuping, Damien Hanyurwimfura, and Jun Long. 2015. A lexicon-based approach for hate speech detection. In International Journal of Multimedia and Ubiquitous Engineering, volume 10, pages 215–230. find
  15. Karthik Dinakar, Birago Jones, Catherine Havasi, Henry Lieberman, and Rosalind W. Picard. 2012. Common sense reasoning for detection, prevention, and mitigation of cyberbullying. TiiS, 2:18:1–18:30. find
  16. Chikashi Nobata, Joel R. Tetreault, Achint Oommen Thomas, Yashar Mehdad, and Yi Chang. 2016. Abusive language detection in online user content. In WWW. find
  17. Björn Gambäck and Utpal Kumar Sikdar. 2017. Using convolutional neural networks to classify hate-speech. In ALW1@ACL. our card
  18. Ji Ho Park and Pascale Fung. 2017. One-step and two-step classification for abusive language detection on twitter. CoRR, abs/1706.01206. our card
  19. Lei Gao and Ruihong Huang. 2017. Detecting online hate speech using context aware models. In RANLP. find
  20. John Pavlopoulos, Prodromos Malakasiotis, and Ion Androutsopoulos. 2017. Deeper attention to abusive user content moderation. In EMNLP. find
  21. Spiros V. Georgakopoulos, Sotiris K. Tasoulis, Aristidis G. Vrahatis, and Vassilis P. Plagianakos. 2018. Convolutional neural networks for toxic comment classification. In SETN. find
  22. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP. our card
  23. Michal Ptaszynski, Juuso Kalevi Kristian Eronen, and Fumito Masui. 2017. Learning deep on cyberbullying is always better than brute force. In IJCAI. find
  24. Jennifer Golbeck, Zahra Ashktorab, Rashad O. Banjo, Alexandra Berlinger, Siddharth Bhagwan, Cody Buntain, Paul Cheakalos, Alicia A. Geller, Quint Dawei Yin and Brian D. Davison. 2009. Detection of harassment on web 2 . 0. In CAW2.0@WWW. find
  25. Jing Qian, Mai ElSherief, Elizabeth M. Belding-Royer, and William Yang Wang. 2018. Leveraging intra-user and inter-user representation learning for automated hate speech detection. In NAACL-HLT. find
  26. Ziqi Zhang and Lei Luo. 2018. Hate speech detection: A solved problem? the challenging case of long tail on twitter. CoRR, abs/1803.03662. find
  27. Julian Risch and Ralf Krestel. 2018. Aggression identification using deep learning and data augmentation. In TRAC-1@COLING, pages 150–158. find
  28. Ziqi Zhang, David Robinson, and Jonathan A. Tepper. 2018. Detecting hate speech on twitter using a convolution-gru based deep neural network. In ESWC. find
  29. David Robinson, Ziqi Zhang, and Jonathan Tepper. 2018. Hate speech detection on twitter: Feature engineering v.s. feature selection. In ESWC. find
  30. Haoti Zhong, Hao Li, Anna Cinzia Squicciarini, Sarah Michele Rajtmajer, Christopher Griffin, David J. Miller, and Cornelia Caragea. 2016. Content-driven detection of cyberbullying on the instagram social network. In IJCAI. find
  31. Björn Ross, Michael Rist, Guillermo Carbonell, Benjamin Cabrera, Nils Kurowsky, and Michael Wojatzki. 2016. Measuring the reliability of hate speech annotations: The case of the european refugee crisis. CoRR, abs/1701.08118. find
  32. Steven Zimmerman, Udo Kruschwitz, and Chris Fox. 2018. Improving hate speech detection with deep learning ensembles. In LREC. find
  33. Niloofar Safi Samghabadi, Suraj Maharjan, Alan Sprague, Raquel Diaz-Sprague, and Thamar Solorio. 2017. Detecting nastiness in social media. In ALW1@ACL. find
  34. Anna Schmidt and Michael Wiegand. 2017. A survey on hate speech detection using natural language processing. In SocialNLP@EACL. our card
  35. Sanjana Sharma, Saksham Agrawal, and Manish Shrivastava. 2018. Degree based classification of harmful speech using twitter data. CoRR, abs/1806.04197. find
  36. Fabio Del Vigna, Andrea Cimino, Felice Dell’Orletta, Marinella Petrocchi, and Maurizio Tesconi. 2017. Hate me, hate me not: Hate speech detection on facebook. In ITASEC. find
  37. W. Lloyd Warner and Julia Hirschberg. 2012. Detecting hate speech on the world wide web. In LSM@ACL. our card
  38. Zeerak Waseem. 2016. Are you a racist or am i seeing things? annotator influence on hate speech detection on twitter. In NLP+CSS@EMNLP. our card
  39. Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In SRW@NAACL-HLT. our card
  40. Zeerak Waseem, James Thorne, and Joachim Bingel. 2018. Bridging the gaps: Multi task learning for domain transfer of hate speech detection. Online Harassment, pages 29–55. find
  41. Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex machina: Personal attacks seen at scale. In WWW. find
  42. Guang Xiang, Bin Fan, Ling Wang, Jason I. Hong, and Carolyn Penstein Rosé. 2012. Detecting offensive tweets via topical feature discovery over a large scale twitter corpus. In CIKM. find
  43. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In NAACL-HLT. our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/aken-2018-challenges-toxic-comment-classification-depth/