Elamur Research⌕
☰
home / papers / 2022 / Meta AI at Arabic Hate Speech 2022: MultiTask Learning with Self-Correction for Hate Speech Classification

Meta AI at Arabic Hate Speech 2022: MultiTask Learning with Self-Correction for Hate Speech Classification

Paper · 2022 · Badr AlKhamissi and Mona Diab

Summary

The paper describes a submission to the OSACT 2022 shared task on fine-grained Arabic hate speech detection in tweets. The task has three linked levels: offensive or not, hate speech or not, and which of six hate speech types. The authors used the task's ~13k-tweet corpus, which covers MSA and dialects and is heavily skewed toward clean tweets. They fine-tuned one MARBERTv2 encoder that feeds three separate classification heads, so all three subtasks are learned together. On top of this they combined a grid of differently tuned models into an ensemble. They also added a post-hoc "self-consistency correction" that uses the coarser heads to repair contradictory fine-grained predictions. The system, called AraHS, beat the organisers' QARiB baselines on every reported metric, reaching 82.7% macro-F1 on hate speech detection. An ablation shows that joint training mainly helps the hate speech subtasks. The work matters because Arabic hate speech data is scarce and imbalanced, and it shows that the label hierarchy itself can serve as extra supervision and a check on errors.

Key points

  • The corpus has about 13k tweets, each judged by 3 crowd annotators, split 70/10/20. Hate speech makes up 10.54% of tweets, and the Disability class has only 3 examples, none of them in training.
  • The backbone is MARBERTv2, a BERT-base-sized model (~163M parameters) pretrained on 1B Arabic tweets. Each head is a 768-unit dense layer, then GELU, then LayerNorm, then a linear output layer. The training loss is the sum of the three negative log-likelihoods.
  • Test results against QARiB:
    • Offensive detection macro-F1: 84.5% vs 82.3%.
    • Hate speech detection macro-F1: 82.7% vs 80.0%, with accuracy 94.1% vs 93.0%.
    • Fine-grained classification macro-F1: 51.9%, for which no baseline exists.
  • Hyperparameter grid: 12 runs over batch sizes {2,4,8,16} and learning rates {1e-5, 5e-6, 1e-6}. The ensemble multiplies the class probabilities of the models element-wise and takes the argmax.
  • Validation ablation:
    • Multitask training raised hate speech detection F1 from 86.4% to 88.1%.
    • Offensive detection stayed flat (86.7% single-task vs 86.6% multitask).
    • Self-consistency correction lifted fine-grained F1 from 54.8% to 56.6%.

Interesting ideas

  • I1. When a fine-grained head contradicts the coarser heads' decisions, the system does not keep the contradiction; it picks the next most probable label that agrees with the hierarchy above. This is a cheap repair for any stack of nested labels.
  • I2. Training one shared encoder with separate heads for interdependent decisions helped most on the rare, fine-grained labels, while the easy coarse task neither gained nor lost. This suggests that related decisions can share data without hurting each other.
  • I3. The ensemble multiplies probabilities instead of averaging them, so a label survives only if no model gives it very low probability. This acts like a soft veto from every member.
  • I4. During training, a separate checkpoint is saved for each subtask whenever that subtask's own validation F1 improves. Each head can therefore peak at a different moment.
  • I5. The paper points out that a class missing from training (Disability) drags down macro-F1 on test, which shows how label coverage shapes the headline metric.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): A speech-act or irony classifier could be overruled the same way when its choice clashes with an already-decided coarser frame, and the system could log which rule fired so the correction stays explainable.
  • I2 (tree): gparse could predict UPOS, morphological features and deprels from one encoder with separate heads, with the expected benefit on rare deprels.
  • I3 (learn): Witness parsers' arc or label distributions could be combined by product when building silver data, so that any single strong objection pushes an arc out of the agreed set.
  • I4 (learn): For multi-output training, Mova could keep the best snapshot per output (tagger vs parser vs labeler) instead of one global best epoch.
  • I5 (learn): Evaluation reports could flag labels absent from training data and show macro scores with and without them, acting as a sanity check alongside negative controls.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MetaMenlo Park · United States183 works · 228,558 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 3Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 worksBias, fairness & safety · 517 works

Cite

Cite as Badr AlKhamissi and Mona Diab. 2022. Meta AI at Arabic Hate Speech 2022: MultiTask Learning with Self-Correction for Hate Speech Classification. Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{alkhamissi2022meta,
  title = {{Meta AI at Arabic Hate Speech 2022: MultiTask Learning with Self-Correction for Hate Speech Classification}},
  author = {Badr AlKhamissi and Mona Diab},
  booktitle = {Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection},
  year = {2022},
  url = {https://aclanthology.org/2022.osact-1.24/}
}

References

46 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdelali, A., Mubarak, H., Samih, Y., Hassan, S., and Darwish, K. (2021). QADI: Arabic dialect identification in the wild. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 1–10, Kyiv, Ukraine (Virtual), April. Association for Computational Linguistics.
  2. Abdul-Mageed, M., Elmadany, A., and Nagoudi, E. M. B. (2021). ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online, August. Association for Computational Linguistics. ↗
  3. Abu Farha, I. and Magdy, W. (2019). Mazajak: An online Arabic sentiment analyser. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 192–198, Florence, Italy, August. Association for Computational Linguistics.
  4. Abu Farha, I. and Magdy, W. (2020). Multitask learning for Arabic offensive language and hate-speech detection. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 86–90, Marseille, France, May. European Language Resource Association.
  5. Al-Khalifa, H., Magdy, W., Darwish, K., Elsayed, T., and Mubarak, H. (2020). Proceedings of the 4th workshop on open-source arabic corpora and processing tools, with a shared task on offensive language detection. Marseille, France, May. European Language Resource Association. find
  6. Albadi, N., Kurdi, M., and Mishra, S. (2018). Are they our brothers? analysis and detection of religious hate speech in the arabic twittersphere. 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 69– 76. find
  7. Aldjanabi, W., Dahou, A., Al-qaness, M. A. A., Elsayed Abd Elaziz, M., Helmi, A., and Damasevicius, R. (2021). Arabic offensive and hate speech detection using a cross-corpora multi-task learning model. Informatics, 8:69, 10. ↗
  8. Alshehri, A., Nagoudi, E. M. B., Alhuzali, H., and Abdul-Mageed, M. (2018). Think before your click: Data and models for adult content in arabic twitter. find
  9. Antoun, W., Baly, F., and Hajj, H. (2020). AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France, May. European Language Resource Association. ↗
  10. Ashi, M. M., Siddiqui, M. A., and Nadeem, F. (2018). Pre-trained word embeddings for arabic aspect-based sentiment analysis of airline tweets. In AISI. find
  11. Assimakopoulos, S., Muskat, R. V., van der Plas, L., and Gatt, A. (2020a). Annotating for hate speech: The maneco corpus and some input from critical discourse analysis. arXiv preprint arXiv:2008.06222. ↗
  12. Assimakopoulos, S., Vella Muskat, R., van der Plas, L., and Gatt, A. (2020b). Annotating for hate speech: The MaNeCo corpus and some input from critical discourse analysis. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 5088–5097, Marseille, France, May. European Language Resources Association. find
  13. Ba, J., Kiros, J., and Hinton, G. E. (2016). Layer normalization. ArXiv, abs/1607.06450. ↗
  14. Crawshaw, M. (2020). Multi-task learning with deep neural networks: A survey. ArXiv, abs/2009.09796. ↗
  15. Davidson, T., Warmsley, D., Macy, M. W., and Weber, I. (2017). Automated hate speech detection and the problem of offensive language. In ICWSM. ↗
  16. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171– 4186, Minneapolis, Minnesota, June. Association for Computational Linguistics. ↗
  17. Djandji, M., Baly, F., Antoun, W., and Hajj, H. (2020). Multi-task learning using AraBert for offensive language detection. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 97–101, Marseille, France, May. European Language Resource Association.
  18. Duwairi, R., Hayajneh, A., and Quwaider, M. (2021). A deep learning framework for automatic detection of hate speech embedded in arabic tweets. Arabian Journal for Science and Engineering, 46:1–14. ↗
  19. El Mahdaouy, A., El Mekki, A., Essefar, K., El Mamoun, N., Berrada, I., and Khoumsi, A. (2021). Deep multi-task model for sarcasm detection and sentiment analysis in Arabic language. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 334–339, Kyiv, Ukraine (Virtual), April. Association for Computational Linguistics. ↗
  20. Fortuna, P. and Nunes, S. (2018). A survey on automatic detection of hate speech in text. ACM Computing Surveys (CSUR), 51(4):1–30. ↗
  21. Founta, A. M., Djouvas, C., Chatzakou, D., Leontiadis, I., Blackburn, J., Stringhini, G., Vakali, A., Sirivianos, M., and Kourtellis, N. (2018). Large scale crowdsourcing and characterization of twitter abusive behavior. In Twelfth International AAAI Conference on Web and Social Media. ↗
  22. Gülaçti, F. (2010). The effect of perceived social support on subjective well-being. Procedia - Social and Behavioral Sciences, 2:3844–3849. find
  23. Haddad, H., Mulki, H., and Oueslati, A., (2019). T-HSAB: A Tunisian Hate Speech and Abusive Dataset, pages 251–263. 10. find
  24. Hassan, S., Samih, Y., Mubarak, H., Abdelali, A., Rashed, A., and Chowdhury, S. A. (2020). ALT submission for OSACT shared task on offensive language detection. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 61–65, Marseille, France, May. European Language Resource Association.
  25. Hendrycks, D. and Gimpel, K. (2016). Bridging nonlinearities and stochastic regularizers with gaussian error linear units. ArXiv, abs/1606.08415. ↗
  26. League, A.-D. (2020). Online hate and harassment. the american experience 2021. Center for Technology and Society. Retrieved from www. adl. org/media/14643/download. find
  27. Liu, Y., Gu, J., Goyal, N., Li, X., Edunov, S., Ghazvininejad, M., Lewis, M., and Zettlemoyer, L. (2020). Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742. ↗
  28. Loshchilov, I. and Hutter, F. (2019). Decoupled weight decay regularization. In International Conference on Learning Representations. find
  29. MacAvaney, S., Yao, H.-R., Yang, E., Russell, K., Goharian, N., and Frieder, O. (2019). Hate speech detection: Challenges and solutions. PloS one, 14(8):e0221152. ↗
  30. Mollas, I., Chrysopoulou, Z., Karlos, S., and Tsoumakas, G. (2020). Ethos: an online hate speech detection dataset. ArXiv, abs/2006.08328. ↗
  31. Mubarak, H., Darwish, K., and Magdy, W. (2017). Abusive language detection on Arabic social media. In Proceedings of the First Workshop on Abusive Language Online, pages 52–56, Vancouver, BC, Canada, August. Association for Computational Linguistics.
  32. Mubarak, H., Hassan, S., and Chowdhury, S. A. (2022). Emojis as anchors to detect arabic offensive language and hate speech. arXiv preprint arXiv:2201.06723. ↗
  33. Mulki, H., Haddad, H., Bechikh Ali, C., and Alshabani, H. (2019). L-HSAB: A Levantine Twitter dataset for hate speech and abusive language. In Proceedings of the Third Workshop on Abusive Language Online, pages 111–118, Florence, Italy, August. Association for Computational Linguistics.
  34. Nagoudi, E. M. B., Elmadany, A., and Abdul-Mageed, M. (2021). Arat5: Text-to-text transformers for arabic language generation.
  35. Ousidhoum, N., Lin, Z., Zhang, H., Song, Y., and Yeung, D.-Y. (2019). Multilingual and multi-aspect hate speech analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4675–4684, Hong Kong, China, November. Association for Computational Linguistics.
  36. Paz, M. A., Montero-Dı́az, J., and Moreno-Delgado, A. (2020). Hate speech: A systematized review. SAGE Open, 10(4):2158244020973022. find
  37. Sanguinetti, M., Poletto, F., Bosco, C., Patti, V., and Stranisci, M. (2018). An Italian Twitter corpus of hate speech against immigrants. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May. European Language Resources Association (ELRA).
  38. Sap, M., Gabriel, S., Qin, L., Jurafsky, D., Smith, N. A., and Choi, Y. (2020). Social bias frames: Reasoning about social and power implications of language. In ACL.
  39. Schmidt, A. and Wiegand, M. (2017). A survey on hate speech detection using natural language processing. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, pages 1–10, Valencia, Spain, April. Association for Computational Linguistics.
  40. Strubell, E., Ganesh, A., and McCallum, A. (2019). Energy and policy considerations for deep learning in nlp. ArXiv, abs/1906.02243. ↗
  41. Tinsley, T. and Board, K. (2013). Languages for the Future. British Council. find
  42. Vogels, E. A. (2021). The state of online harassment. Pew Research Center, 13. find
  43. Waldron, J. (2012). The Harm in Hate Speech. Harvard University Press. ↗
  44. Waseem, Z. and Hovy, D. (2016). Hateful symbols or hateful people? predictive features for hate speech detection on Twitter. In Proceedings of the NAACL Student Research Workshop, pages 88–93, San Diego, California, June. Association for Computational Linguistics.
  45. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., and Brew, J. (2019). Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. Wu, Y., Schuster, M., Chen, Z., Le, Q., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., Klingner, J., Shah, A., Johnson, M., Liu, X., Kaiser, u., Gouws, S., Kato, Y., Kudo, T., Kazawa, H., and Dean, J. (2016). Google’s neural machine translation system: Bridging the gap between human and machine translation. 09. ↗
  46. Zampieri, M., Malmasi, S., Nakov, P., Rosenthal, S., Farra, N., and Kumar, R. (2019). Predicting the type and target of offensive posts in social media. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1415– 1420, Minneapolis, Minnesota, June. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alkhamissi-2022-meta-ai-arabic-hate-speech/