Elamur Research⌕
☰
home / papers / 2020 / Multitask Learning for Arabic Offensive Language and Hate-Speech Detection

Multitask Learning for Arabic Offensive Language and Hate-Speech Detection

Paper · 2020 · Ibrahim Abu Farha and Walid Magdy

Summary

This shared-task paper from the SMASH team at OSACT4 tackles automatic detection of offensive language and hate-speech in Arabic tweets. Social media content mixes Modern Standard Arabic with dialects. The authors used the 10,000-tweet dataset that is shared with the SemEval 2020 Arabic offensive language task. They compared several systems:

  • two simple baselines;
  • a BiLSTM and a CNN-BiLSTM, both built on Mazajak word embeddings;
  • fine-tuned multilingual BERT;
  • multitask variants of the CNN-BiLSTM.

The multitask models exploit the fact that hate-speech is a subset of offensive content, so both labels are predicted together. One variant adds sentiment as a third output, using labels produced automatically by the Mazajak sentiment analyser. A further version corrects those automatic labels by forcing them to negative whenever the gold annotation marks the tweet as offensive or hateful. The multitask systems gave the best development scores. On the hidden test set they reached macro F1 of 0.76 for hate-speech and 0.877 for offensive language. The work shows that cheap related signals can help a rare, imbalanced target class.

Key points

  • Data: 10,000 tweets, split into 7,000 for training, 1,000 for development and 2,000 for a hidden test set. Hate-speech is rare, with only 361 positive training tweets and 44 positive development tweets. Offensive tweets number 1,410 in training and 179 in development.
  • Preprocessing and embeddings: Preprocessing follows Mazajak's pipeline of letter normalisation, elongation removal and cleaning. The embeddings are skip-gram word2vec vectors trained on roughly 250M tweets.
  • Model variants:
    • MTL predicts offensive and hate-speech jointly.
    • MTL-S adds raw Mazajak sentiment as a third output.
    • MTL-S-N is MTL-S with sentiment set to negative for gold-positive tweets.
    • All three share a CNN-BiLSTM trunk that forks before the output layer.
  • Development macro F1:
    • MTL-S-N scored best on offensive language (0.904).
    • MTL scored best on hate-speech (0.737).
    • Multilingual BERT trailed on offensive language (0.857). The authors attribute this to its limited vocabulary and Wikipedia/MSA training, which does not match dialectal tweets.
  • Test results: MTL scored 0.76 on hate-speech, ranking 6th of 13 teams. MTL-S-N scored 0.877 on offensive language, ranking 5th of 14.

Interesting ideas

  • I1. When one label logically implies another (hateful ⇒ offensive), training a shared model with one output head per label lets the rare label borrow evidence from the more frequent one.
  • I2. Another system's predictions can serve as an extra training objective, adding a related signal without new manual annotation.
  • I3. Noisy auxiliary labels can be repaired with a simple rule drawn from gold annotations (offensive ⇒ negative sentiment) before training, which reduces error propagation from the upstream tool.
  • I4. A strong pretrained model can lose to a smaller in-domain one when its vocabulary and training register (MSA Wikipedia) differ from the target text (dialectal tweets). Checking domain coverage is therefore worth doing before choosing a model.
  • I5. Reporting macro F1 next to a majority-class baseline and a Naive Bayes baseline exposes how little a skewed task's accuracy really says.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Speech-act or irony detectors could be trained jointly with broader, better-populated pragmatic labels they entail, giving sparse categories more support.
  • I2 (learn): Outputs from witness parsers or LLM teacher tiers could become auxiliary heads rather than only gold substitutes, so their errors shape the representation less directly.
  • I3 (learn): Silver data could be cleaned by explicit implication rules from trusted annotations, a form of self-induced repair that stays explainable.
  • I4 (tokens): Before relying on a pretrained component for informal English, measure lexicon and lemma coverage on the target genre and prefer in-domain resources where coverage is poor.
  • I5 (learn): Evaluation of rare phenomena could always include trivial-predictor scores as negative controls, so gains are judged against a meaningful floor.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬The Alan Turing InstituteLondon · United Kingdom29 works · 4,368 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 worksBias, fairness & safety · 517 works

Cite

Cite as Ibrahim Abu Farha and Walid Magdy. 2020. Multitask Learning for Arabic Offensive Language and Hate-Speech Detection. Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{farha2020multitask,
  title = {{Multitask Learning for Arabic Offensive Language and Hate-Speech Detection}},
  author = {Ibrahim Abu Farha and Walid Magdy},
  booktitle = {Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection},
  year = {2020},
  url = {https://aclanthology.org/2020.osact-1.14/}
}

References

25 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., and Zheng, X. (2015). TensorFlow: Large-scale machine learning on heterogeneous systems. Software available from tensorflow.org. find
  2. Abu Farha, I. and Magdy, W. (2019). Mazajak: An online Arabic sentiment analyser. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 192–198, Florence, Italy, August. Association for Computational Linguistics.
  3. Alakrot, A., Murray, L., and Nikolov, N. S. (2018). Towards accurate detection of offensive language in online communication in arabic. Procedia computer science, 142:315–320. find
  4. Albadi, N., Kurdi, M., and Mishra, S. (2018). Are they our brothers? analysis and detection of religious hate speech in the arabic twittersphere. In 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 69–76. IEEE. find
  5. Caruana, R. (1997). Multitask learning. Machine learning, 28(1):41–75. find
  6. Chollet, F. et al. (2015). Keras. https://keras.io. ↗
  7. Darwish, K., Magdy, W., and Mourad, A. (2012). Language processing for arabic microblog retrieval. In Proceedings of the 21st ACM international conference on Information and knowledge management, pages 2427– 2430. ACM. find
  8. Darwish, K., Magdy, W., et al. (2014). Arabic information retrieval. Foundations and Trends R in Information Retrieval, 7(4):239–342. ↗
  9. Davidson, T., Warmsley, D., Macy, M., and Weber, I. (2017). Automated hate speech detection and the problem of offensive language. In Eleventh international aaai conference on web and social media. ↗
  10. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June. Association for Computational Linguistics. ↗
  11. Haidar, B., Chamoun, M., and Serhrouchni, A. (2017). A multilingual system for cyberbullying detection: Arabic content detection using machine learning. Advances in Science, Technology and Engineering Systems Journal, 2(6):275–284. ↗
  12. Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8):1735–1780. ↗
  13. Howard, J. and Ruder, S. (2018). Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339, Melbourne, Australia, July. Association for Computational Linguistics. find
  14. Malmasi, S. and Zampieri, M. (2017). Detecting hate speech in social media. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, pages 467–472, Varna, Bulgaria, September. INCOMA Ltd. ↗
  15. Mubarak, H. and Darwish, K. (2019). Arabic offensive language classification on twitter. In International Conference on Social Informatics, pages 269–276. Springer. find
  16. Mubarak, H., Darwish, K., and Magdy, W. (2017). Abusive language detection on Arabic social media. In Proceedings of the First Workshop on Abusive Language Online, pages 52–56, Vancouver, BC, Canada, August. Association for Computational Linguistics.
  17. Mubarak, H., Darwish, K., Magdy, W., Elsayed, T., and Al-Khalifa, H. (2020). Overview of osact4 arabic offensive language detection shared task. 4.
  18. Nobata, C., Tetreault, J., Thomas, A., Mehdad, Y., and Chang, Y. (2016). Abusive language detection in online user content. In Proceedings of the 25th International Conference on World Wide Web, WWW ’16, page 145–153, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee. find
  19. Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018). Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227– 2237, New Orleans, Louisiana, June. Association for Computational Linguistics.
  20. Razavi, A. H., Inkpen, D., Uritsky, S., and Matwin, S. (2010). Offensive language detection using multi-level classification. In Canadian Conference on Artificial Intelligence, pages 16–27. Springer. find
  21. Søgaard, A. and Goldberg, Y. (2016). Deep multi-task learning with low level tasks supervised at lower layers. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 231–235, Berlin, Germany, August. Association for Computational Linguistics.
  22. Waseem, Z. and Hovy, D. (2016). Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In Proceedings of the NAACL Student Research Workshop, pages 88–93, San Diego, California, June. Association for Computational Linguistics.
  23. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., and Brew, J. (2019). Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. ↗
  24. Yin, D., Xue, Z., Hong, L., Davison, B. D., Kontostathis, A., and Edwards, L. (2009). Detection of harassment on web 2.0. Proceedings of the Content Analysis in the WEB, 2:1–7. find
  25. Zampieri, M., Malmasi, S., Nakov, P., Rosenthal, S., Farra, N., and Kumar, R. (2019). SemEval-2019 task 6: Identifying and categorizing offensive language in social media (OffensEval). In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 75–86, Minneapolis, Minnesota, USA, June. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/farha-2020-multitask-learning-arabic-offensive-language/