Elamur Research⌕
☰
home / papers / 2022 / GOF at Arabic Hate Speech 2022: Breaking The Loss Function Convention For Data-Imbalanced Arabic Offensive Text Detection

GOF at Arabic Hate Speech 2022: Breaking The Loss Function Convention For Data-Imbalanced Arabic Offensive Text Detection

Paper · 2022 · Ali Mostafa, Omar Mohamed and Ali Ashraf

Summary

The paper addresses offensive-language detection in Arabic tweets for the OSACT5 shared task. The core difficulty is a long-tailed label distribution. In the roughly 13k-tweet dataset of Mubarak et al. (2022), only about 35% of tweets are offensive and 11% are hate speech, and some fine-grained categories are far rarer.

The common remedy is cross-entropy with class weights set inversely to class frequency. Instead of relying on it, the authors test a set of imbalance-oriented objectives while fine-tuning several Arabic pretrained encoders, including MARBERT, MARBERTV2, QARiB and the AraBERT Twitter models. The objectives are focal, Dice, Tversky, focal Tversky and VS loss, plus label-smoothing variants. Tweets are first cleaned: URLs, mentions and emails become Arabic placeholder words, and diacritics, digits, punctuation and non-Arabic characters are removed before normalization.

Changing the loss raised macro F1 by about 0.5–2 points over weighted cross-entropy. The final ensemble of MARBERT, MARBERTV2 and QARiB reaches about 87% macro F1 on development and 85.17% on test. The study shows that the training objective is a cheap, data-free lever for skewed classification problems.

Key points

  • Data: the OSACT-2022 corpus of about 13k emoji-bearing Arabic tweets, split 70/10/20 into train, dev and test, with labels for offensive, hate speech and fine-grained hate speech.
  • Losses compared:
    • Weighted CE, alone and with label smoothing
    • Focal loss, alone and with label smoothing
    • Dice, Tversky, focal Tversky and VS loss
  • The best loss differed by model: VS loss for MARBERT (85.6), focal loss with label smoothing for MARBERTV2 (85.2), and Dice loss for QARiB (85.7).
  • The best single system on dev was LightGBM trained on QARiB embeddings (85.798 macro F1). The MARBERTV2+MARBERT+QARiB ensemble scored 87.044 on dev (the abstract reports 86.97%) and 85.17% on test.
  • Evaluation used macro precision, recall and F1, since accuracy is misleading under heavy imbalance. Captum's Integrated Gradients was used to show which tokens and emojis drove each prediction.

Interesting ideas

  • I1. Treat the loss function as a tunable knob for rare classes, trying focal, Dice or Tversky objectives instead of defaulting to frequency-weighted cross-entropy.
  • I2. Run the same set of losses across several backbones, because the winning objective changed from model to model rather than being universal.
  • I3. Report macro-averaged F1 rather than accuracy whenever classes are skewed, so poor minority-class performance cannot hide behind majority-class hits.
  • I4. Replace URLs, mentions and emails with fixed placeholder words instead of deleting them, so the model keeps the signal that such an element was present.
  • I5. Use Integrated Gradients attributions to show which tokens pushed a prediction toward each class, making a neural decision inspectable per example.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): gparse could train with focal or Tversky loss so rare UD relations such as vocative, reparandum or dislocated are not swamped by frequent ones like nsubj or det.
  • I2 (learn): when retraining the perceptron and transformer parsers, sweep losses per architecture instead of copying one parser's best setting to the other.
  • I3 (frames): speech-act and irony detectors, where ironic or directive utterances are rare, should be scored with per-class and macro metrics in evaluation.
  • I4 (tokens): the en tokenizer could map web artifacts to typed placeholder tokens with a stable lemma and POS, keeping UD trees well-formed around them.
  • I5 (frames): any neural component in prag, such as an irony classifier, could attach token attributions to its output, supporting the requirement that every step be explainable.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 3Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 worksBias, fairness & safety · 517 works

Cite

Cite as Ali Mostafa, Omar Mohamed and Ali Ashraf. 2022. GOF at Arabic Hate Speech 2022: Breaking The Loss Function Convention For Data-Imbalanced Arabic Offensive Text Detection. Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{mostafa2022gof,
  title = {{GOF at Arabic Hate Speech 2022: Breaking The Loss Function Convention For Data-Imbalanced Arabic Offensive Text Detection}},
  author = {Ali Mostafa, Omar Mohamed and Ali Ashraf},
  booktitle = {Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection},
  year = {2022},
  url = {https://aclanthology.org/2022.osact-1.21/}
}

References

55 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdelali, A., Hassan, S., Mubarak, H., Darwish, K., and Samih, Y. (2021a). Pre-training bert on arabic tweets: Practical considerations. ↗
  2. Abdelali, A., Hassan, S., Mubarak, H., Darwish, K., and Samih, Y. (2021b). Pre-training bert on arabic tweets: Practical considerations. arXiv preprint arXiv:2102.10684. ↗
  3. Abdellatif, M. and Elgammal, A. (2020). Offensive language detection in arabic using ulmfit. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 82–85.
  4. Abdul-Mageed, M., Elmadany, A., and Nagoudi, E. M. B. (2021). ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online, August. Association for Computational Linguistics. ↗
  5. Agrawal, S. and Awekar, A. (2018). Deep learning for detecting cyberbullying across multiple social media platforms. In European conference on information retrieval, pages 141–153. Springer. find
  6. Alakrot, A., Murray, L., and Nikolov, N. S. (2018). Dataset construction for the detection of anti-social behaviour in online communication in arabic. Procedia Computer Science, 142:174–181. find
  7. Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T. (2017). Enriching word vectors with subword information. Transactions of the association for computational linguistics, 5:135–146. ↗
  8. Cao, K., Wei, C., Gaidon, A., Arechiga, N., and Ma, T. (2019). Learning imbalanced datasets with label-distribution-aware margin loss. Advances in neural information processing systems, 32. ↗
  9. Chen, D., Fisch, A., Weston, J., and Bordes, A. (2017). Reading wikipedia to answer open-domain questions. arXiv preprint arXiv:1704.00051. ↗
  10. Davidson, T., Warmsley, D., Macy, M., and Weber, I. (2017). Automated hate speech detection and the problem of offensive language. In Proceedings of the International AAAI Conference on Web and Social Media, volume 11, pages 512–515. ↗
  11. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  12. Dice, L. R. (1945). Measures of the amount of ecologic association between species. Ecology, 26(3):297– 302. ↗
  13. Djandji, M., Baly, F., Antoun, W., and Hajj, H. (2020). Multi-task learning using arabert for offensive language detection. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 97–101.
  14. El-Khair, I. A. (2016). 1.5 billion words arabic corpus. arXiv preprint arXiv:1611.04033. ↗
  15. Elmadany, A., Zhang, C., Abdul-Mageed, M., and Hashemi, A. (2020). Leveraging affective bidirectional transformers for offensive language detection. arXiv preprint arXiv:2006.01266. ↗
  16. Farha, I. A. and Magdy, W. (2020). Multitask learning for arabic offensive language and hate-speech detection. In Proceedings of the 4th workshop on open-source Arabic corpora and processing tools, with a shared task on offensive language detection, pages 86–90.
  17. Gupta, A., Pal, A., Khurana, B., Tyagi, L., and Modi, A. (2021). Humor@IITK at SemEval-2021 task 7: Large language models for quantifying humor and offensiveness. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 290–296, Online, August. Association for Computational Linguistics. find
  18. Hada, R., Sudhir, S., Mishra, P., Yannakoudakis, H., Mohammad, S. M., and Shutova, E. (2021). Ruddit: Norms of offensiveness for English Reddit comments. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2700– 2717, Online, August. Association for Computational Linguistics.
  19. Haddad, B., Orabe, Z., Al-Abood, A., and Ghneim, N. (2020). Arabic offensive language detection with attention-based deep neural networks. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 76–81. ↗
  20. Hashemi, S. R., Salehi, S. S. M., Erdogmus, D., Prabhu, S. P., Warfield, S. K., and Gholipour, A. (2018). Tversky as a loss function for highly unbalanced image segmentation using 3d fully convolutional deep networks. arXiv preprint arXiv:1803.11078. ↗
  21. Hassan, S., Samih, Y., Mubarak, H., Abdelali, A., Rashed, A., and Chowdhury, S. A. (2020). Alt submission for osact shared task on offensive language detection. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 61–65.
  22. Hegazi, M. O., Al-Dossari, Y., Al-Yahy, A., Al-Sumari, A., and Hilal, A. (2021). Preprocessing arabic text on social media. Heliyon, 7(2):e06191. ↗
  23. Howard, J. and Ruder, S. (2018). Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146. ↗
  24. Huang, C., Li, Y., Loy, C. C., and Tang, X. (2016). Learning deep representation for imbalanced classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5375–5384. find
  25. Husain, F. (2020). Osact4 shared task on offensive language detection: Intensive preprocessing-based approach. arXiv preprint arXiv:2005.07297. ↗
  26. Keleg, A., El-Beltagy, S. R., and Khalil, M. (2020). Asu opto at osact4-offensive language detection for arabic text. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 66–70.
  27. Kini, G. R., Paraskevas, O., Oymak, S., and Thrampoulidis, C. (2021). Label-imbalanced and group-sensitive classification under overparameterization. Advances in Neural Information Processing Systems, 34. ↗
  28. Kokhlikyan, N., Miglani, V., Martin, M., Wang, E., Alsallakh, B., Reynolds, J., Melnikov, A., Kliushkina, N., Araya, C., Yan, S., et al. (2020). Captum: A unified and generic model interpretability library for pytorch. arXiv preprint arXiv:2009.07896. ↗
  29. Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., and Dyer, C. (2016). Neural architectures for named entity recognition. arXiv preprint arXiv:1603.01360. ↗
  30. Li, X., Sun, X., Meng, Y., Liang, J., Wu, F., and Li, J. (2019). Dice loss for data-imbalanced nlp tasks. arXiv preprint arXiv:1911.02855. ↗
  31. Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollár, P. (2017). Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980–2988. ↗
  32. Lison, P. and Tiedemann, J. (2016). OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 923–929, Portorož, Slovenia, May. European Language Resources Association (ELRA).
  33. Ma, X. and Hovy, E. (2016). End-to-end sequence labeling via bi-directional lstm-cnns-crf. arXiv preprint arXiv:1603.01354. ↗
  34. Mahajan, D., Girshick, R., Ramanathan, V., He, K., Paluri, M., Li, Y., Bharambe, A., and Van Der Maaten, L. (2018). Exploring the limits of weakly supervised pretraining. In Proceedings of the European conference on computer vision (ECCV), pages 181–196. find
  35. McCann, B., Keskar, N. S., Xiong, C., and Socher, R. (2018). The natural language decathlon: Multi-task learning as question answering. arXiv preprint arXiv:1806.08730. ↗
  36. Meng, Y., Wu, W., Wang, F., Li, X., Nie, P., Yin, F., Li, M., Han, Q., Sun, X., and Li, J. (2019). Glyce: Glyph-vectors for chinese character representations. Advances in Neural Information Processing Systems, 32. ↗
  37. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. (2013). Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26. ↗
  38. Milletari, F., Navab, N., and Ahmadi, S.-A. (2016). V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), pages 565–571. IEEE. find
  39. Mubarak, H. and Darwish, K. (2019). Arabic offensive language classification on twitter. In International Conference on Social Informatics, pages 269–276. Springer. find
  40. Mubarak, H., Darwish, K., and Magdy, W. (2017). Abusive language detection on Arabic social media. In Proceedings of the First Workshop on Abusive Language Online, pages 52–56, Vancouver, BC, Canada, August. Association for Computational Linguistics.
  41. Mubarak, H., Darwish, K., Magdy, W., Elsayed, T., and Al-Khalifa, H. (2020). Overview of osact4 arabic offensive language detection shared task. In Proceedings of the 4th Workshop on open-source arabic corpora and processing tools, with a shared task on offensive language detection, pages 48–52.
  42. Mubarak, H., Hassan, S., and Chowdhury, S. A. (2022). Emojis as anchors to detect arabic offensive language and hate speech. arXiv preprint arXiv:2201.06723. ↗
  43. Müller, R., Kornblith, S., and Hinton, G. E. (2019). When does label smoothing help? Advances in neural information processing systems, 32. find
  44. Nagoudi, E. M. B., Elmadany, A., Abdul-Mageed, M., and Alhindi, T. (2020). Machine generation and detection of Arabic manipulated and fake news. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 69–84, Barcelona, Spain (Online), December. Association for Computational Linguistics. ↗
  45. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., De-Vito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. (2019). Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, et al., editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc. ↗
  46. Saeed, H. H., Calders, T., and Kamiran, F. (2020). Osact4 shared tasks: Ensembled stacked classification for offensive and hate speech in arabic tweets. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 71–75.
  47. Soliman, A. B., Eissa, K., and El-Beltagy, S. R. (2017). Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256–265. find
  48. Sorensen, T. A. (1948). A method of establishing groups of equal amplitude in plant sociology based on similarity of species content and its application to analyses of the vegetation on danish commons. Biol. Skar., 5:1–34. find
  49. Suárez, P. J. O., Sagot, B., and Romary, L. (2019). Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7). Leibniz-Institut für Deutsche Sprache. find
  50. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016). Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826. ↗
  51. Tversky, A. (1977). Features of similarity. Psychological review, 84(4):327. ↗
  52. Wang, Y.-X., Ramanan, D., and Hebert, M. (2017). Learning to model the tail. Advances in Neural Information Processing Systems, 30. find
  53. Wolpert, D. H. and Macready, W. G. (1997). No free lunch theorems for optimization. IEEE transactions on evolutionary computation, 1(1):67–82. ↗
  54. Yu, A. W., Dohan, D., Luong, M.-T., Zhao, R., Chen, K., Norouzi, M., and Le, Q. V. (2018). Qanet: Combining local convolution with global self-attention for reading comprehension. arXiv preprint arXiv:1804.09541. ↗
  55. Zeroual, I., Goldhahn, D., Eckart, T., and Lakhouaja, A. (2019). OSIAN: Open source international Arabic news corpus - preparation and integration into the CLARIN-infrastructure. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 175–182, Florence, Italy, August. Association for Computational Linguistics. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mostafa-2022-gof-arabic-hate-speech-2022/