Elamur Research⌕
☰
home / papers / 2017 / One-step and Two-step Classification for Abusive Language Detection on Twitter

One-step and Two-step Classification for Abusive Language Detection on Twitter

Summary

Park and Fung address automatic detection of sexist and racist tweets. They compare two strategies. The one-step strategy is a single three-way classifier (none / sexism / racism). The two-step strategy first flags a tweet as abusive or not, then assigns an abusive tweet to sexism or racism. They merged the two expert-annotated English Twitter corpora by Waseem and Hovy (2016) and Waseem (2016), released for ALW1, into roughly 18k labelled tweets. They tested three CNN variants: CharCNN on characters, WordCNN on static word2vec embeddings, and a new HybridCNN that reads both. Baselines were character n-gram logistic regression, SVM and FastText.

The best one-step system was HybridCNN, with a weighted F1 of 0.827. Chaining two logistic regression classifiers reached a nearly equal 0.824. The second step (sexism vs. racism) proved easy for every model, at about 0.95 F1. This suggests the hard part is spotting abuse in the first place. The two-step design therefore lets cheap models compete with neural ones and makes it possible to mix classifier types.

Key points

  • Data: the merged corpus has 12,427 "none", 2,059 racism and 3,864 sexism tweets. For the first step, racism and sexism were pooled into 5,923 "abusive" tweets.
  • HybridCNN design: a character channel with filters of width 3, 4 and 5 and a word channel with widths 1, 2 and 3. Each uses 50 feature maps, and the outputs are 1-max-pooled, concatenated and fed to softmax.
  • WordCNN: follows Kim's CNN-static setup with frozen 300-d Google News word2vec. Hashtags such as #womenagainstfeminism are split into words using the wordsegment library.
  • CharCNN: the deep Zhang et al. (2015) architecture was cut down to 2 convolution layers because the dataset is small. It scored below WordCNN.
  • Evaluation: 10-fold cross-validation with label-weighted precision, recall and F1. On the binary abusive-detection step, HybridCNN led with F1 0.734, ahead of WordCNN (0.726) and LR (0.711).

Interesting ideas

  • I1. Split coarse detection from fine-grained typing. If the subtype distinction is easy, all the difficulty sits in the binary step. A large dataset labelled only abusive/not could train step one, and a small finely labelled set could train step two. This fits low-resource settings, such as Ukrainian moderation data.
  • I2. Mix models per stage. Each step can use whichever classifier works best for it. For example, HybridCNN followed by LR beat HybridCNN used for both steps. Any cascaded pipeline can choose components stage by stage instead of using one model throughout.
  • I3. Combine character and word channels. Running both in parallel within one CNN helps with misspellings and invented words like #feminazi. This is relevant for noisy social-media text and morphologically rich languages.
  • I4. Segment hashtags before embedding lookup. Splitting hashtags into ordinary words is a cheap way to recover out-of-vocabulary tokens when using pretrained embeddings.
  • I5. Use class-balanced mini-batches. Sampling each batch with equal label proportions is a simple alternative to loss reweighting for skewed datasets.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

80 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 74 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w17-3006

@inproceedings{park2017one,
  title = {{One-step and Two-step Classification for Abusive Language Detection on Twitter}},
  author = {Park, Ji Ho and Fung, Pascale},
  booktitle = {Proceedings of the First Workshop on Abusive Language Online},
  year = {2017},
  doi = {10.18653/v1/w17-3006},
  url = {https://doi.org/10.18653/v1/w17-3006}
}

References

14 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Badjatiya, P., Gupta, S., Gupta, M., & Varma, V. (2017). Deep learning for hate speech detection in tweets. Proceedings of the 26th International Conference on World Wide Web Companion, 759-760. find
  2. Schmidt, A., & Wiegand, M. (2017). A survey on hate speech detection using natural language processing. find our card
  3. Djuric, N., Zhou, J., Morris, R., Grbovic, M., Radosavljevic, V., & Bhamidipati, N. (2015). Hate speech detection with comment embeddings. Proceedings of the 24th International Conference on World Wide Web, 29-30. find
  4. Segaran, T., & Hammerbacher, J. (2009). Beautiful data: The stories behind elegant data solutions " O'Reilly Media, Inc.". find
  5. Wang, W., Chen, L., Thirunarayan, K., & Sheth, A. P. (2012). Harnessing twitter" big data" for automatic emotion identification. Privacy, Security, Risk and Trust (PASSAT), 2012 International Conference on and 2012 International Confernece on Social Computing (SocialCom), 587-592. find
  6. Joulin, A., Grave, E., Bojanowski, P., & Mikolov, T. (2016). Bag of tricks for efficient text classification. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, Kim, Y. (2014). Convolutional neural networks for sentenc… find
  7. Kingma, D., & Ba, J. (2014). Adam: A method for stochastic optimization. Proceedings of the 3rd International Conference on Learning Representations (ICLR) find
  8. Waseem, Z. (2016). Are you a racist or am I seeing things? annotator influence on hate speech detection on twitter. Proceedings of the 1st Workshop on Natural Language Processing and Computational Social Science, 138-142. find our card
  9. LeCun, Y., Kavukcuoglu, K., & Farabet, C. (2010). Convolutional networks and applications in vision. Circuits and Systems (ISCAS), Proceedings of 2010 IEEE International Symposium on, 253-256. find
  10. Waseem, Z., & Hovy, D. (2016). Hateful symbols or hateful people? predictive features for hate speech detection on twitter. Proceedings of NAACL-HLT, 88-93. find our card
  11. Meyer, R. (2016, 07/21). Twitter's famous racist problem. find
  12. The Atlantic, Zhang, X., Zhao, J., & LeCun, Y. (2015). Character-level convolutional networks for text classification. Advances in Neural Information Processing Systems, 649-657. find our card
  13. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems, 3111-3119. find
  14. Nobata, C., Tetreault, J., Thomas, A., Mehdad, Y., & Chang, Y. (2016). Abusive language detection in online user content. Proceedings of the 25th International Conference on World Wide Web, 145-153. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/park-2017-one-step-two-step-classification/