Elamur Research⌕
☰
home / papers / 2020 / Effect of Word Embedding Models on Hate and Offensive Speech Detection

Effect of Word Embedding Models on Hate and Offensive Speech Detection

Paper · 2020 · Safa Alsafari, Samira Sadaoui and Malek Mouhoub

Summary

The paper asks how much the choice of word-embedding model matters for detecting hate and offensive speech in Arabic tweets, a language with little prior deep learning work on this problem. Earlier comparisons were muddied because public Arabic embeddings were trained on different corpora. To remove that confound, the authors built a single 10 GB Arabic corpus from Wikipedia, United Nations monolingual text and Twitter. On it they trained five embedding variants: GloVe, Word2vec CBOW and skip-gram, and FastText CBOW and skip-gram. They added randomly initialized embeddings as a baseline. Each embedding was paired with four networks (CNN, BiLSTM, GRU, CNN+BiLSTM) on a previously published Arabic hate speech dataset. They tested three label granularities: binary, three-way, and six-way, where the six-way task splits hate by target group. Skip-gram embeddings combined with a simple CNN won in all three tasks. Pretrained vectors beat random initialization by several F-macro points. The broader finding is that the embedding choice matters about as much as the network architecture. The authors released the embeddings on GitHub.

Key points

  • Six embedding settings (five pretrained plus random) were crossed with four architectures, giving 24 pairs per task. Each pair was trained 15 times with different random weight initializations, and scores were averaged. The text reports the total classifier count inconsistently, as both 144 and 114.
  • Data was split 60/10/30 into train, validation and test. Evaluation used macro-averaged Precision, Recall and F-macro because the classes are imbalanced. Preprocessing cut letter elongations longer than two repeats.
  • The CNN follows the Kim (2014) design: 250 filters of width 2, dropout 0.5, max pooling, and a linear output layer.
  • Best results by task:
    • Binary: CNN with Word2vec skip-gram, about 3.7% F-macro above the random baseline and 0.44% above CNN with FastText skip-gram.
    • Three-class and six-class: CNN with FastText skip-gram, about 5.4% and 4.7% above baseline.
  • FastText skip-gram always beat FastText CBOW under the same network. FastText CBOW was the only embedding that worked best with recurrent networks (GRU or BiLSTM).

Interesting ideas

  • I1. Hold the pretraining corpus fixed across all embedding methods so that any score difference can be attributed to the learning method rather than to the data.
  • I2. Rerun every configuration many times with only the random seed changed, and report the average to separate real effects from training noise.
  • I3. Keep a deliberately uninformed variant (random embeddings) in every comparison as a floor that each method must clearly beat.
  • I4. Evaluate the same data at nested label granularities (2, 3, then 6 classes) to see whether a method's advantage survives finer distinctions.
  • I5. Shorten letter elongations typical of social media text before lookup, which reduces vocabulary sparsity in noisy informal writing.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when comparing witness parsers or tagger variants, train them on the identical silver set so disagreements reflect the model, not the data.
  • I2 (learn): require multi-seed runs before accepting a new parser setting or self-induced repair rule, since gaps under one point may be seed noise.
  • I3 (learn): add a trivial or randomized baseline as a negative control in each evaluation, so gains from compiled knowledge are measured against a meaningful floor.
  • I4 (frames): score speech-act and irony detection both coarsely and at fine subtypes to find where the explainable rules start to break down.
  • I5 (tokens): add an elongation-normalizing step to the en tokenizer, followed by a dictionary check, so forms like "sooo" reach the right lemma and morphology.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 1,484 works

Cite

Cite as Safa Alsafari, Samira Sadaoui and Malek Mouhoub. 2020. Effect of Word Embedding Models on Hate and Offensive Speech Detection. arXiv:2012.07534.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2012.07534 · arXiv 2012.07534

@misc{alsafari2020effect,
  title = {{Effect of Word Embedding Models on Hate and Offensive Speech Detection}},
  author = {Safa Alsafari, Samira Sadaoui and Malek Mouhoub},
  year = {2020},
  doi = {10.48550/arxiv.2012.07534},
  eprint = {2012.07534},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2012.07534}
}

References

29 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alsafari, S., Sadaoui, S., and Mouhoub, M. (2020). Hate and offensive speech detection on arabic social media. Online Social Networks and Media, 19. ↗
  2. Anowar, F. and Sadaoui, S. (2020a). Chunk-based incremental classification of fraud data. In The Thirty-Third International FLAIRS Conference (FLAIRS-33), AAAI, pages 176–179. find
  3. Anowar, F. and Sadaoui, S. (2020b). Incremental Neural-Network Learning for Big Fraud Data. In International Conference on Systems, Man, and Cybernetics (IEEE SMC 2020). find
  4. Badjatiya, P., Gupta, S., Gupta, M., and Varma, V. (2017). Deep Learning for Hate Speech Detection in Tweets. find
  5. Burnap, P. and Williams, M. L. (2015). Cyber hate speech on twitter: An application of machine classification and statistical modeling for policy and decision making. Policy and Internet, 7(2):223–242. ↗
  6. Cho, K., van Merriënboer, B., Bahdanau, D., and Bengio, Y. (2014). On the Properties of Neural Machine Translation: Encoder-Decoder Approaches.
  7. Davidson, T., Warmsley, D., Macy, M., and Weber, I. (2017). Automated hate speech detection and the problem of offensive language. Proceedings of the 11th International Conference on Web and Social Media, ICWSM 2017, (Icwsm):512–515. ↗
  8. Del Vigna, F., Cimino, A., Dell’Orletta, F., Petrocchi, M., and Tesconi, M. (2017). Hate me, hate me not: Hate speech detection on Facebook. find
  9. Elrazzaz, M., Elbassuoni, S., Shaban, K., and Helwe, C. (2017). Methodical Evaluation of Arabic Word Embeddings. pages 454–458. find
  10. Elshaar, S. and Sadaoui, S. (2020). Semi-supervised classification of fraud data in commercial auctions. Applied Artificial Intelligence, 34(1):47–63. ↗
  11. Gambäck, B. and Sikdar, U. (2017). Using Convolutional Neural Networks to Classify Hate-Speech. pages 85– 90.
  12. Ghannay, S., Favre, B., Estève, Y., and Camelin, N. (2016). Word Embeddings Evaluation and Combination. Language Resources and Evaluation. find
  13. Grave, E., Bojanowski, P., Gupta, P., Joulin, A., and Mikolov, T. (2018). Learning Word Vectors for 157 Languages. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018).
  14. Jozefowicz, R., Zaremba, W., and Sutskever, I. (2015). An Empirical Exploration of Recurrent Network Architectures. In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, ICML’15, pages 2342–2350. find
  15. JMLR.org. Kim, Y. (2014). Convolutional Neural Networks for Sentence Classification. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing.
  16. Malmasi, S. and Zampieri, M. (2018). Challenges in discriminating profanity from hate speech. Journal of Experimental and Theoretical Artificial Intelligence, 30(2):187–202. ↗
  17. Mehdad, Y. and Tetreault, J. (2016). Do Characters Abuse More Than Words? pages 299–303. find
  18. Mikolov, T., Sutskever, I., Chen, K., Corrado, G., and Dean, J. (2013). Distributed Representations of Words and Phrases and Their Compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2, NIPS’13, pages 3111–3119, Red Hook, NY, USA. Curran Associates Inc. ↗
  19. Mohammad, A. B., Eissa, K., and El-Beltagy, S. (2017). AraVec: A set of Arabic Word Embedding Models for use in Arabic NLP. Procedia Computer Science, 117:256–265. find
  20. Park, J. and Fung, P. (2017). One-step and Two-step Classification for Abusive Language Detection on Twitter.
  21. Pennington, J., Socher, R., and Manning, C. (2014). Glove: Global Vectors for Word Representation. In EMNLP, volume 14, pages 1532–1543.
  22. Pitsilis, G. K., Ramampiaro, H., and Langseth, H. (2018). Effective hate-speech detection in Twitter data using recurrent neural networks. Applied Intelligence, 48(12):4730–4742. ↗
  23. Stein, R. A., Jaques, P. A., and Valiati, J. F. (2019). An analysis of hierarchical text classification using word embeddings. Information Sciences, 471:216–232. ↗
  24. Viera, A. and Garrett, J. (2005). Understanding Interobserver Agreement: The Kappa Statistic. Family medicine, 37:360–363. find
  25. Wang, Y., Liu, S., Afzal, N., Rastegar-Mojarad, M., Wang, L., Shen, F., Kingsbury, P., and Liu, H. (2018). A Comparison of Word Embeddings for the Biomedical Natural Language Processing. Journal of Biomedical Informatics, 87. ↗
  26. Waseem, Z. and Hovy, D. (2016). Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter. pages 88–93.
  27. Zampieri, M., Malmasi, S., Nakov, P., Rosenthal, S., Farra, N., and Kumar, R. (2019). Predicting the Type and Target of Offensive Posts in Social Media. pages 1415–1420.
  28. Zhang, Z. and Luo, L. (2018). Hate Speech Detection: A Solved Problem? The Challenging Case of Long Tail on Twitter. Semantic Web, Accepted. find
  29. Zhang, Z., Robinson, D., and Tepper, J. (2018). Detecting hate speech on Twitter using a convolution-GRU based deep neural network. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alsafari-2020-effect-word-embedding-models-hate/