Elamur Research⌕
☰
home / papers / 2022 / CHILLAX - at Arabic Hate Speech 2022: A Hybrid Machine Learning and Transformers based Model to Detect Arabic Offensive and Hate Speech

CHILLAX - at Arabic Hate Speech 2022: A Hybrid Machine Learning and Transformers based Model to Detect Arabic Offensive and Hate Speech

Paper · 2022 · Kirollos Makram, Kirollos George Nessim, Malak Emad Abd-Almalak et al.

Summary

The paper describes a team entry to the OSACT2022 shared task on Arabic tweets. It addresses two binary subtasks: spotting offensive tweets (subtask 1) and spotting hate speech (subtask 2). The authors do not fine-tune the transformer end to end. Instead, they use MARBERT, a BERT model pre-trained on Arabic Twitter text, as a frozen feature extractor. They sum its last four hidden layers and average over tokens to get one 768-dimensional vector per tweet. These vectors go to two scikit-learn classifiers, Logistic Regression and Random Forest. The data is skewed, especially for hate speech, so they balance the classes by generating extra minority-class tweets with contextual MARBERT-based word substitution. Cleaning is deliberately light: only URLs and user mentions are removed. On the official test set, Logistic Regression gives the best results: 78% macro F1 for offensive detection and 76% for hate speech. Both clearly beat the organisers' baselines. The work shows that a cheap, frozen-encoder pipeline with a simple linear classifier can be competitive on noisy, imbalanced dialectal text.

Key points

  • Data. The OSACT2022 dataset has about 13,000 Arabic tweets: roughly 35% offensive and 11% hate speech, with vulgar and violent tweets under 2% each. It is split 70/10/20 into train, development and test. The hate-speech training portion has 7,928 negative and 959 positive tweets.

  • Features. MARBERT has 12 layers and hidden size 768. The outputs of its top four layers are summed and then mean-pooled over the tweet length to form the feature vector.

  • Classifiers.

    • Logistic Regression uses the saga solver and a strong regularisation setting (reported as 1e-3).
    • Random Forest draws 40% of the samples for each tree (max_samples = 0.4).

  • Augmentation and cleaning.

    • Augmentation uses the nlpaug library with contextual embeddings. Generated [UNK] tokens are stripped from the synthetic tweets.
    • Stemming, lemmatising and punctuation removal hurt results, because MARBERT was trained on raw tweets.

  • Results.

    SubtaskModelAccuracyPrecisionRecallF1
    OffensiveLogistic Regression80%78%78%78%
    OffensiveRandom Forest74%––72%
    OffensiveOrganisers' baseline–––39%
    Hate speechLogistic Regression89%72%80%76%
    Hate speechRandom Forest–––73%
    Hate speechOrganisers' baseline–––47%

    Random Forest reached 97–98% on its own training data, a sign of heavy overfitting.

Interesting ideas

  • I1. Pooling several upper encoder layers (summing the top four, then averaging over tokens) gives richer sentence features than taking only the final layer, which is tuned for language modelling.
  • I2. A frozen pretrained encoder feeding a small, well-regularised linear model was enough to beat a Random Forest, and the linear model's decisions are easier to inspect.
  • I3. Preprocessing should match the pretraining domain: aggressive normalisation hurt because the encoder had only ever seen raw text, so only true noise (links, mentions) was removed.
  • I4. Minority classes were enlarged with contextual masked-LM word substitutions, and degenerate outputs (unknown tokens) were filtered out before training.
  • I5. Reporting scores on training data next to held-out sets, alongside a majority-class baseline, made the Random Forest's overfitting and the real gain over trivial prediction visible at once.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): gparse could build its token representations from a sum or mix of upper-middle layers rather than the last one, and test which layer combination best predicts heads and labels.
  • I2 (learn): small explainable probes trained on frozen encoder features could serve as cheap extra witnesses when building silver data, with their weights open to audit.
  • I3 (tokens): the tokenizer and tagger should keep surface forms that the downstream models were trained on, and strip only clearly non-linguistic material such as URLs and handles.
  • I4 (learn): rare constructions or frame types could be augmented by masked-LM substitution, with a filter that rejects synthetic sentences whose parse or tokens break.
  • I5 (learn): every evaluation report could routinely show train-vs-held-out gaps and a trivial-baseline row as negative controls.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

🧭 Topics 3Research subjects and languages the work is about.Sentiment & emotion · 1,035 worksSemitic languages · 3,066 worksBias, fairness & safety · 587 works

Cite

Cite as Kirollos Makram, Kirollos George Nessim, Malak Emad Abd-Almalak, Shady Zekry Roshdy, Seif Hesham Salem, Fady Fayek Thabet and Ensaf Hussien Mohamed. 2022. CHILLAX - at Arabic Hate Speech 2022: A Hybrid Machine Learning and Transformers based Model to Detect Arabic Offensive and Hate Speech. Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{makram2022chillax,
  title = {{CHILLAX - at Arabic Hate Speech 2022: A Hybrid Machine Learning and Transformers based Model to Detect Arabic Offensive and Hate Speech}},
  author = {Kirollos Makram, Kirollos George Nessim, Malak Emad Abd-Almalak, Shady Zekry Roshdy, Seif Hesham Salem, Fady Fayek Thabet and Ensaf Hussien Mohamed},
  booktitle = {Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection},
  year = {2022},
  url = {https://aclanthology.org/2022.osact-1.25/}
}

References

15 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. score Abdul-Mageed, M., Elmadany, A., and Nagoudi, E. M. B. (2020). Arbert & marbert: deep bidirectional transformers for arabic. arXiv preprint arXiv:2101.01785. 91% 91% ”Abdul-Mageed, M., Elmadany, A., and Nagoudi, E. M. B. (”2021”). ”ARBERT & MARBERT: Deep bidirectional transformers for Arabic”. In ”Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)”, pages ”7088–7105”, ”Online”, aug. ”Association for Computational Linguistics”. 74% 76% Abuzayed, A. and Elsayed, T. (2020). Quick and simple approach for detecting hate speech in arabic tweets. In Proceedings of the 4th workshop on open-source Arabic Corpora and processing tools, with a shared task on offensive language detection, pages 109–114. ↗
  2. Al-khalifa, S., Aljarah, I., and Abushariah, M. A. M. (2020). Hate speech classification in arabic tweets. Journal of Theoretical and Applied Information Technology, 98:1816–1831. find
  3. Aldjanabi, W., Dahou, A., Al-qaness, M. A., Abd Elaziz, M., Helmi, A. M., and Damaševičius, R. (2021). Arabic offensive and hate speech detection using a cross-corpora multi-task learning model. In Informatics, volume 8, page 69. Multidisciplinary Digital Publishing Institute. find
  4. Alshaalan, R. and Al-Khalifa, H. (2020). Hate speech detection in saudi twittersphere: A deep learning approach. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 12–23.
  5. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  6. Farha, I. A. and Magdy, W. (2021). Benchmarking transformer-based language models for arabic sentiment and sarcasm detection. In Proceedings of the sixth Arabic natural language processing workshop, pages 21–31.
  7. Faris, H., Aljarah, I., Habib, M., and Castillo, P. A. (2020). Hate speech detection using word embedding and deep learning in the arabic language context. In ICPRAM, pages 453–460. find
  8. Hany, K. (2022). Osact2022-source-code. https://github.com/kirollos-hany/OSACT2022-source-code. ↗
  9. Ma, E. (2019). Nlp augmentation. https://github.com/makcedward/nlpaug. Mubarak, H., Hassan, S., and Chowdhury, S. A. ↗
  10. (2022). Emojis as anchors to detect arabic offensive language and hate speech. arXiv preprint arXiv:2201.06723. ↗
  11. Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. (2018). Improving language understanding by generative pre-training. find
  12. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.
  13. Williams, M. L., B. P. J. A. L. H. and Ozalp. (2020). Hate in the machine: anti-black and anti-muslim social media posts as predictors of offline racially and religiously aggravated crime. The British Journal of Criminology 60(1): 93–117. find
  14. Yang, J. and Zhao, H. (2019). Deepening hidden representations from pre-trained language models. arXiv preprint arXiv:1911.01940. ↗
  15. Zhuang, L., Wayne, L., Ya, S., and Jun, Z. (2021). A robustly optimized bert pre-training approach with post-training. In Proceedings of the 20th Chinese National Conference on Computational Linguistics, pages 1218–1227.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/makram-2022-chillax-arabic-hate-speech-2022/