Elamur Research⌕
☰
home / papers / 2023 / Transformers and Ensemble methods: A solution for Hate Speech Detection in Arabic languages

Transformers and Ensemble methods: A solution for Hate Speech Detection in Arabic languages

Paper · 2023 · Angel Felipe Magnossao de Paula, Imene Bensalem, Paolo Rosso et al.

Summary

The paper describes a submission to the hate speech subtask (task 1.d) of the CERIST NLP Challenge 2022. The task asks a system to label Arabic tweets as Hateful or Not Hateful. Arabic social media text is largely dialectal and has relatively few resources, which makes the problem hard. The data are COVID-19 disinformation tweets, and about one in nine is hateful.

The authors fine-tuned six transformers with shared hyper-parameters:

  • four Arabic-only models: AraBERT, AraELECTRA, Albert-Arabic and the decoder-based AraGPT2;
  • two multilingual models: mBERT and XLM-RoBERTa.

They then combined the models with two ensembles. Majority Vote counts each model's predicted class. Highest Sum adds up the models' class scores. The number of training epochs for each model was chosen by five-fold cross-validation, scored by F1 on the Hateful class.

In cross-validation, Majority Vote was best (F1 0.76), ahead of the strongest single model, AraBERT. Several individual models collapsed into predicting almost everything as hateful. The single allowed test submission used Majority Vote and scored F1 0.60 and accuracy 0.86. The work shows that simple voting over diverse pretrained models gives a sturdier classifier for a low-resource language.

Key points

  • Dataset: 10,828 tweets from the CERIST challenge, split 80/20 into train and test, with roughly 11% labelled hate speech. The corpus is multi-label and also annotated for fake news.
  • Models: six transformers. Five are encoders and one (AraGPT2) is a decoder. Albert-Arabic was used in its Large size because its parameter reduction fits in GPU memory; all others were Base.
  • Fine-tuning settings: learning rate 1e-5, dropout 0.3, maximum length 128 tokens, batch size 18. Epochs were tuned per model by 5-fold CV.
  • Cross-validation results:
SystemF1AccuracyPrecisionRecall
Majority Vote0.760.950.880.69
AraBERT0.68———
AraELECTRA0.61———
Highest Sum0.62—0.450.96
AraGPT2, mBERT, XLM-RoBERTa0.200.11—1.00

The last three models effectively predicted only the positive class. The paper's prose ranks AraGPT2 second, which contradicts its own table.

  • Official test result (Majority Vote): F1 0.60, accuracy 0.86.

Interesting ideas

  • I1. Reporting accuracy next to positive-class F1 and recall exposes models that collapsed into an "always positive" classifier: recall reaches 1.0 while accuracy drops to the base rate.
  • I2. Hard majority voting across heterogeneous models, with random tie-breaking, beat every single member and gave much higher precision than any of them.
  • I3. Summing raw output scores let poorly calibrated, overconfident members dominate (recall 0.96, precision 0.45), whereas counting votes resisted this.
  • I4. The number of training epochs was selected separately for each model by k-fold cross-validation on the official target metric, rather than using one fixed schedule.
  • I5. When the target class is rare, evaluation should centre on F1 for that class, since overall accuracy hides failure on the class of interest.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Add a trivial constant-output baseline as a negative control in evaluation, and flag any trained component whose scores match it.
  • I2 (learn): This supports building silver data from agreement among several witness parsers, voting per arc or label rather than trusting the single best parser.
  • I3 (tree): When combining gparse and the perceptron parser, prefer arc-level voting or calibrated scores over adding raw scores from models with different confidence scales.
  • I4 (learn): Per-component early stopping, tuned on the metric that actually matters downstream, could replace global training schedules.
  • I5 (frames): Irony and offensive-speech-act detection should be scored by positive-class F1 alongside accuracy, because positives are scarce.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hamad bin Khalifa UniversityDoha · Qatar71 works · 12,942 citations🎓Universitat Politècnica de ValènciaValencia · Spain21 works · 3,275 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 3 Sentiment & emotion · 608 works Semitic languages · 1,484 works Bias, fairness & safety · 209 works

Cite

Cite as Angel Felipe Magnossao de Paula, Imene Bensalem, Paolo Rosso and Wajdi Zaghouani. 2023. Transformers and Ensemble methods: A solution for Hate Speech Detection in Arabic languages. arXiv:2303.09823 · Revue de l Information Scientifique et Technique, vol. 27, no. 2, pp. 7-12, November 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2303.09823 · arXiv 2303.09823

@misc{paula2023transformers,
  title = {{Transformers and Ensemble methods: A solution for Hate Speech Detection in Arabic languages}},
  author = {Angel Felipe Magnossao {de Paula}, Imene Bensalem, Paolo Rosso and Wajdi Zaghouani},
  year = {2023},
  doi = {10.48550/arxiv.2303.09823},
  eprint = {2303.09823},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.09823}
}

References

19 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alkomah, F., & Ma, X., 2022. A literature review of textual hate speech detection methods and datasets. Information, 13(6), 273.Antoun, W., Baly, F., & Hajj, H., 2020. AraBERT: Transformer-based Model for Arabic Language Understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection (pp. 9-15). ↗ our card
  2. Antoun, W., Baly, F., & Hajj, H., 2021a. AraELECTRA: Pre-Training Text Discriminators for Arabic Language Understanding. In Proceedings of the Sixth Arabic Natural Language Processing Workshop (pp. 191-195). ↗ our card
  3. Antoun, W., Baly, F., & Hajj, H. 2021b. AraGPT2: Pre-Trained Transformer for Arabic Language Generation. In Proceedings of the Sixth Arabic Natural Language Processing Workshop (pp. 196-207). ↗ our card
  4. Bennessir, M. A., Rhouma, M., Haddad, H., & Fourati, C. (2022, June). icompass at arabic hate speech 2022: Detect hate speech using qrnn and transformers. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection (pp. 176-180). find
  5. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, É., Ott, M., Zettlemoyer, L., & Stoyanov, V., 2020. Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 8440-8451). de Paula, A. F. M., & Schlicht, I. B., 2021a. AI-UPV at iberlef-2021 DETOXIS task: Toxicity detection in immigration-related web news comments using transformers and statistical models. arXiv preprint arXiv:2111.04530. de Paula, A. F. M., da Silva, R. F., & Schlicht, I. B., 2021b. Sexism prediction in spanish and english tweets using monolingual and multilingual bert and ensemble models. arXiv preprint arXiv:2111.04551. de Paula, A. F. M., & da Silva, R. F., 2022. Detection and Classification of Sexism on Social Media Using Multiple Languages, Transformers, and Ensemble Models. In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2022) co-located with the XXXVIII International Conference of the Spanish Society for Natural Language Processing (SEPLN 2022), CEUR Workshop proceedings (Vol. 3202, pp. 1-11). de Paula, A. F. M., Rosso, P., Bensalem, I., & Zaghouani, W., 2022. Upv at the arabic hate speech 2022 shared task: Offensive language and hate speech detection using transformers and ensemble models. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection (pp. 181-185). ↗
  6. Devlin, J., Chang, M. W., Lee, K., & Toutanova, K., 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗ our card
  7. Hadj Ameur, M. S., & Aliane, H., 2021. AraCOVID19-MFH: Arabic COVID-19 multi-label fake news & hate speech detection dataset. Procedia Computer Science, 189, 232-241. find
  8. Hassan, S., Samih, Y., Mubarak, H., Abdelali, A., Rashed, A., & Chowdhury, S. A., 2020. ALT submission for OSACT shared task on offensive language detection. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection (pp. 61-65). find
  9. Husain, F., & Uzuner, O., 2021. A survey of offensive language detection for the Arabic language. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 20(1), 1-44. ↗
  10. Lin, T., Wang, Y., Liu, X., & Qiu, X. (2022). A survey of transformers. AI Open. ↗
  11. Mahdaouy, A. E., Mekki, A. E., Oumar, A., Mousannif, H., & Berrada, I., 2022. Deep Multi-Task Models for Misogyny Identification and Categorization on Arabic Social Media. arXiv preprint arXiv:2206.08407. ↗ our card
  12. Mohammed, A. H., & Ali, A. H., 2021. Survey of BERT (bidirectional encoder representation transformer) types. In Journal of Physics: Conference Series (Vol. 1963, No. 1, p. 012173). IOP Publishing. find
  13. Mubarak, H., Darwish, K., Magdy, W., Elsayed, T., & Al-Khalifa, H., 2020. Overview of OSACT4 Arabic offensive language detection shared task. In Proceedings of the 4th Workshop on open-source arabic corpora and processing tools, with a shared task on offensive language detection (pp. 48-52). our card
  14. Mubarak, H., Al-Khalifa, H., & Al-Thubaity, A., 2022. Overview of OSACT5 Shared Task on Arabic Offensive Language and Hate Speech Detection. In LREC 2022 Workshop Language Resources and Evaluation Conference 20-25 June 2022 (p. 162). a Mubarak, H., Hassan, S., & Chowdhury, S. A., 2022. Emojis as anchors to detect arabic offensive language and hate speech. arXiv preprint arXiv:2201.06723. ↗ our card
  15. Mulki, H., & Ghanem, B., 2021. ArMI at FIRE2021: Overview of the First Shared Task on Arabic Misogyny Identification. Working Notes of FIRE, 820-830. find
  16. Ravichandiran, S., 2021. Getting Started with Google BERT: Build and train state-of-the-art natural language processing models using BERT. Packt Publishing Ltd. find
  17. Safaya, A., 2020. Arabic-alBERT. find
  18. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I., 2017. Attention is all you need. Advances in neural information processing systems, 30. our card
  19. Zampieri, M., Nakov, P., Rosenthal, S., Atanasova, P., Karadzhov, G., Mubarak, H., Derczynski, L., Pitenis, Z., & Çöltekin, Ç., 2020. SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020). In Proceedings of the Fourteenth Workshop on Semantic Evaluation (pp. 1425-1447). ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/paula-2023-transformers-ensemble-methods-solution-hate/