Elamur Research⌕
☰
home / papers / 2022 / UPV at the Arabic Hate Speech 2022 Shared Task: Offensive Language and Hate Speech Detection using Transformers and Ensemble Models

UPV at the Arabic Hate Speech 2022 Shared Task: Offensive Language and Hate Speech Detection using Transformers and Ensemble Models

Paper · 2022 · Angel Felipe Magnossão de Paula, Paolo Rosso, Imene Bensalem et al.

Summary

The authors tackle automatic detection of abusive content in Arabic tweets, many of which are written in dialect, as part of an OSACT 2022 competition focused on identifying hate speech at a fine-grained level. The task has three subtasks of increasing difficulty: deciding whether a tweet is offensive, whether it contains hate speech, and which target type the hate addresses (race, religion, ideology, disability, social class, gender). They fine-tuned six pretrained transformers that differ along two axes: encoder versus decoder architecture, and Arabic-only versus multilingual pretraining. These were AraBert, AraElectra, Albert-Arabic, AraGPT2, mBert and XLM-Roberta. Hyperparameters were tuned with cross-validation on the training data. The models' predictions were then merged with two ensemble schemes, Majority vote and Highest sum. The submitted systems beat the official baseline on every subtask, on F1-macro as well as accuracy, precision and recall. Highest sum was consistently among the two best configurations on both development and test data. This suggests that pooling model outputs is a robust, low-cost way to combine heterogeneous transformers.

Key points

  • The dataset has about 13K tweets, each labelled by three crowd annotators with majority resolution. About 35% are offensive and only 11% are hate speech, while vulgar and violent tweets are roughly 1.5% and 0.7%. The split is 70/10/20 for train, dev and test.
  • Albert-Arabic was used in its large size. The other five models were base size because of compute limits. AraGPT2 is the only decoder-based model.
  • 5-fold cross-validation on the training set used F1-macro to select epochs, learning rate and dropout. All models ended up with dropout 0.3 and learning rate 0.00005. Max length was 64 tokens and batch size was 32.
  • Highest sum produced a top-two F1-macro in every experiment on both dev and test.
  • The authors attribute the weaker F1-macro on subtasks B and C to class imbalance, which pushed models toward the negative class and increased false negatives. Subtask C was further hurt by its multi-label target.

Interesting ideas

  • I1. Deliberately choosing ensemble members that differ in architecture (encoder vs decoder) and pretraining language (mono- vs multilingual) gives the vote more independent errors to cancel out.
  • I2. Pooling the members' outputs and taking the strongest overall class (Highest sum) was steadier than a plain majority vote, because a confident minority can still win.
  • I3. The authors set hyperparameters by k-fold cross-validation on training data alone, keeping dev and test untouched for honest comparison.
  • I4. High accuracy can hide many false negatives on a rare class, so they read F1-macro and recall alongside accuracy and explained the gap by imbalance.
  • I5. The task is layered as a coarse decision (offensive?), then a narrower one (hate?), then a fine target type, with each level harder than the last.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When choosing witness parsers for silver data, prefer models with genuinely different designs, so that agreement means something.
  • I2 (learn): Silver-tree agreement could sum per-arc or per-label scores across witnesses, rather than only counting votes, and keep the summed margin as an explainable confidence value.
  • I3 (learn): Tune gparse and perceptron settings inside training folds, so dev and test numbers stay clean controls.
  • I4 (learn): Evaluation reports should break scores down per rare label (e.g., uncommon deprels or speech acts) so that majority classes cannot mask failures.
  • I5 (frames): Irony and speech-act detection in prag could follow the same cascade, first flagging a non-literal utterance and then typing it, with each step logged for explanation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

🧭 Topics 3Research subjects and languages the work is about.Sentiment & emotion · 1,035 worksSemitic languages · 3,066 worksBias, fairness & safety · 587 works

Cite

Cite as Angel Felipe Magnossão de Paula, Paolo Rosso, Imene Bensalem and Wajdi Zaghouani. 2022. UPV at the Arabic Hate Speech 2022 Shared Task: Offensive Language and Hate Speech Detection using Transformers and Ensemble Models. Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{paula2022upv,
  title = {{UPV at the Arabic Hate Speech 2022 Shared Task: Offensive Language and Hate Speech Detection using Transformers and Ensemble Models}},
  author = {Angel Felipe Magnossão {de Paula}, Paolo Rosso, Imene Bensalem and Wajdi Zaghouani},
  booktitle = {Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection},
  year = {2022},
  url = {https://aclanthology.org/2022.osact-1.23/}
}

References

21 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Albadi, N., Kurdi, M., and Mishra, S. (2018). Are they our brothers? analysis and detection of religious hate speech in the Arabic twittersphere. In 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), Barcelona. IEEE. find
  2. Antoun, W., Baly, F., and Hajj, H. (2020). AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France, May. European Language Resource Association (ELRA). ↗
  3. Antoun, W., Baly, F., and Hajj, H. (2021a). Ara-ELECTRA: Pre-training text discriminators for Arabic language understanding. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 191–195, Kyiv, Ukraine (Virtual), April. Association for Computational Linguistics (ACL). find
  4. Antoun, W., Baly, F., and Hajj, H. (2021b). AraGPT2: Pre-trained transformer for Arabic language generation. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 196–207, Kyiv, Ukraine (Virtual), April. Association for Computational Linguistics (ACL). ↗
  5. Chowdhury, S. A., Mubarak, H., Abdelali, A., Jung, S.-G., Jansen, B. J., and Salminen, J. (2020). A multi-platform arabic news comment dataset for offensive language detection. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 6203–6212, Marseille. European Language Resources Association (ELRA).
  6. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, É., Ott, M., Zettlemoyer, L., and Stoyanov, V. (2020). Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the 0.751 0.711 0.500 0.369 0.294 0.143 Association for Computational Linguistics, pages 8440–8451, Online. Association for Computational Linguistics (ACL).
  7. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171– 4186, Minneapolis, Minnesota, June. Association for Computational Linguistics (ACL).
  8. Guellil, I., Adeel, A., Azouaou, F., Chennoufi, S., Maafi, H., and Hamitouche, T. (2020). Detecting hate speech against politicians in arabic community on social media. International Journal of Web Information Systems, 16(3):295–313. ↗
  9. Haddad, H., Mulki, H., and Oueslati, A. (2019). T-HSAB: A tunisian hate speech and abusive dataset. In International Conference on Arabic Language Processing, pages 251–263. Springer. find
  10. Husain, F. and Uzuner, O. (2021). A survey of offensive language detection for the Arabic language. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 20(1):1– 44. ↗
  11. Lin, T., Wang, Y., Liu, X., and Qiu, X. (2021). A survey of transformers. arXiv preprint arXiv:2106.04554. ↗
  12. Mohammed, A. H. and Ali, A. H. (2021). Survey of BERT (bidirectional encoder representation transformer) types. Journal of Physics: Conference Series, 1963(1):012173. find
  13. Mubarak, H., Darwish, K., and Magdy, W. (2017). Abusive language detection on Arabic social media. In Proceedings of the First Workshop on Abusive Language Online, pages 52–56, Vancouver. Association for Computational Linguistics (ACL).
  14. Mubarak, H., Darwish, K., Magdy, W., and Al-Khalifa, H. (2020). Overview of OSACT4 arabic offensive language detection shared task. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection.
  15. Language Resources and Evaluation Conference (LREC 2020), pages 48–52, Marseille. European Language Resources Association (ELRA). find
  16. Mubarak, H., Hassan, S., and Chowdhury, S. A. (2022). Emojis as anchors to detect arabic offensive language and hate speech. arXiv preprint arXiv:2201.06723. ↗
  17. Mulki, H. and Ghanem, B. (2021). Let-Mi: An arabic levantine twitter dataset for misogynistic language. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 154–163, Kyiv. ↗
  18. Mulki, H., Haddad, H., Bechikh Ali, C., and Alshabani, H. (2019). L-HSAB: A levantine twitter dataset for hate speech and abusive language. In Proceedings of the Third Workshop on Abusive Language Online, pages 111–118, Florence. Association for Computational Linguistics (ACL).
  19. Ousidhoum, N., Lin, Z., Zhang, H., Song, Y., and Yeung, D.-Y. (2019). Multilingual and multi-aspect hate speech analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4675–4684, Stroudsburg, PA, USA. Association for Computational Linguistics (ACL).
  20. Ravichandiran, S. (2021). Getting Started with Google BERT: Build and train state-of-the-art natural language processing models using BERT. Packt Publishing Ltd. find
  21. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/paula-2022-upv-arabic-hate-speech-2022/