Elamur Research⌕
☰
home / papers / 2020 / OSACT4 Shared Tasks: Ensembled Stacked Classification for Offensive and Hate Speech in Arabic Tweets

OSACT4 Shared Tasks: Ensembled Stacked Classification for Offensive and Hate Speech in Arabic Tweets

Paper · 2020 · Hafiz Hassaan Saeed, Toon Calders and Faisal Kamiran

Summary

The paper describes a system for the OSACT4 2020 shared tasks on Arabic tweets. Subtask A asks whether a tweet is offensive, and subtask B asks whether it contains hate speech. One pipeline served both. The authors first tried classical classifiers over TF.IDF features, found that deep models beat them, and moved to neural networks.

For input representations they tested several pre-trained Arabic embeddings: FastText, Word2Vec, AraVec, N-Gram/Uni-Gram models and multilingual BERT. They then joined the two best, FastText and Word2Vec, into one 400-dimensional vector. Four architectures (CNN, BLSTM, BGRU, BLSTM+CNN) were each trained many times, and the F1 decision threshold was tuned during training. All of the resulting predictions were fed to a stacked ensemble of classical classifiers, which they call ESOTP.

On the official test set the system placed 6th of 35 on offensive language and 5th of 30 on hate speech. The work shows that cheap diversity, from repeated runs and stacking, can make predictions generalize more reliably than any single network in a low-resource language setting.

Key points

  • Embeddings: FastText (300 dimensions) and Word2Vec (100 dimensions, a skip-gram model trained on the Arabic CoNLL17 corpus) were concatenated into 400 dimensions. Any half missing for a word was sampled from a Gaussian using that embedding's mean and standard deviation.
  • Pre-processing:
    • Placeholder tokens, emojis, punctuation, Latin characters, digits and diacritics were removed.
    • Hamza, Ya, Ha and Qaf were normalized.
    • Character runs longer than 3 were collapsed.
    • Unknown words starting with Wa, Fa or Sa were split off as particles.
  • Base models and stacker:
    • The base models were CNN (Kim-style, kernel widths 1–5), BLSTM, BGRU and BLSTM+CNN, trained 250, 200, 70 and 30 times respectively.
    • This gave 550 thresholded predictions per validation sample.
    • Those predictions trained a stacker combining Naïve Bayes, Logistic Regression, SVM, Nearest Neighbours and Random Forest.
  • Cross-validation macro-F1 for offensive language (OFF):
    • SVM+TF.IDF scored 84.86.
    • The best single network, BLSTM, scored 89.02.
    • ESOTP scored 95.51, but this figure comes from re-splitting the validation set, so it is not directly comparable.
  • Hate speech (HS) and test behaviour:
    • On HS, BLSTM scored 76.83 and ESOTP scored 77.79.
    • Test submissions showed that CNN, BLSTM and BGRU overfit, while BLSTM+CNN underfit.
    • BLSTM with attention and BLSTM with added statistical features both lowered scores and were dropped.

Interesting ideas

  • I1. When two embedding tables are joined, fill whichever half is missing with noise drawn from that table's own mean and spread rather than zeros, so unknown words stay in-distribution.
  • I2. Peel off a known prefix particle only after a word fails lookup, so known vocabulary is never wrongly segmented while coverage still grows.
  • I3. Treat the decision threshold as a parameter tuned for the target metric inside training, rather than fixing it at 0.5.
  • I4. Replace a vote over many runs and architectures with a learned second-level classifier over their outputs, which absorbs both the overfit and the underfit members.
  • I5. The meta-classifier was tuned on a split of the validation data, which inflated its reported score; only the test submissions revealed which base models overfit.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): gparse could merge two pretrained input sources with distribution-matched fill for absent rows, so rare tokens do not become a degenerate zero vector.
  • I2 (tokens): the en lemmatizer could try guarded fallback splits for unknown forms only after the direct lexicon lookup misses, such as un-/re- prefixes or hyphenated compounds.
  • I3 (learn): acceptance thresholds for silver trees could be tuned per label or relation against a gold dev slice, maximizing F1 instead of using a fixed confidence cutoff.
  • I4 (learn): when several witness parsers disagree, a small explainable stacker over their per-arc votes could decide which witness to trust in which context.
  • I5 (learn): any learned combiner should be scored on data that neither it nor its witnesses ever touched, with that untouched split acting as a negative control against self-flattering numbers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 worksBias, fairness & safety · 517 works

Cite

Cite as Hafiz Hassaan Saeed, Toon Calders and Faisal Kamiran. 2020. OSACT4 Shared Tasks: Ensembled Stacked Classification for Offensive and Hate Speech in Arabic Tweets. Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{saeed2020osact4,
  title = {{OSACT4 Shared Tasks: Ensembled Stacked Classification for Offensive and Hate Speech in Arabic Tweets}},
  author = {Hafiz Hassaan Saeed, Toon Calders and Faisal Kamiran},
  booktitle = {Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection},
  year = {2020},
  url = {https://aclanthology.org/2020.osact-1.11/}
}

References

17 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Albadi, N., Kurdi, M., and Mishra, S. (2018). Are they our brothers? analysis and detection of religious hate speech in the arabic twittersphere. In 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 69–76. IEEE. find
  2. Chowdhury, A. G., Didolkar, A., Sawhney, R., and Shah, R. R. (2019). Arhnet - leveraging community interaction for detection of religious hate speech in arabic. In Fernando Alva-Manchego, et al., editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28 - August 2, 2019, Volume 2: Student Research Workshop, pages 273–280. Association for Computational Linguistics.
  3. Gambäck, B. and Sikdar, U. K. (2017). Using convolutional neural networks to classify hate-speech. In Proceedings of the First Workshop on Abusive Language Online, ALW@ACL 2017, Vancouver, BC, Canada, August 4, 2017, pages 85–90.
  4. Grave, E., Bojanowski, P., Gupta, P., Joulin, A., and Mikolov, T. (2018). Learning word vectors for 157 languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018.
  5. Haidar, B., Chamoun, M., and Serhrouchni, A. (2017). Multilingual cyberbullying detection system: Detecting cyberbullying in arabic content. In 2017 1st Cyber Security in Networking Conference (CSNet), pages 1–8. IEEE. find
  6. Kim, Y. (2014). Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1746–1751.
  7. Lee, S. and Kim, H. (2015). Why people post benevolent and malicious comments online. Commun. ACM, 58(11):74–79. ↗
  8. Magdy, W., Darwish, K., and Weber, I. (2015). # failedrevolutions: Using twitter to study the antecedents of isis support. arXiv preprint arXiv:1503.02401. ↗
  9. Mohaouchane, H., Mourhir, A., and Nikolov, N. S. (2019). Detecting offensive language on arabic social media using deep learning. In Mohammad A. Alsmirat et al., editors, Sixth International Conference on Social Networks Analysis, Management and Security, SNAMS 2019, Granada, Spain, October 22-25, 2019, pages 466– 471. IEEE. find
  10. Mubarak, H., Darwish, K., and Magdy, W. (2017). Abusive language detection on arabic social media. In Proceedings of the First Workshop on Abusive Language Online, pages 52–56.
  11. Mubarak, H., Darwish, K., Magdy, W., Elsayed, T., and Al-Khalifa, H. (2020). Overview of osact4 arabic offensive language detection shared task. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT), volume 4.
  12. Saeed, H. H., Shahzad, K., and Kamiran, F. (2018). Overlapping toxic sentiment classification using deep neural architectures. In 2018 IEEE International Conference on Data Mining Workshops, ICDM Workshops, Singapore, Singapore, November 17-20, 2018, pages 1361–1366. IEEE. find
  13. Santosh, T. Y. S. S. and Aravind, K. V. S. (2019). Hate speech detection in hindi-english code-mixed social media text. In Proceedings of the ACM India Joint International Conference on Data Science and Management of Data, COMAD/CODS 2019, Kolkata, India, January 3-5, 2019, pages 310–313. find
  14. Soliman, A. B., Eissa, K., and El-Beltagy, S. R. (2017). Aravec: A set of arabic word embedding models for use in arabic NLP. In Third International Conference On Arabic Computational Linguistics, ACLING 2017, November 5-6, 2017, Dubai, United Arab Emirates, pages 256–265. find
  15. Warner, W. and Hirschberg, J. (2012). Detecting hate speech on the world wide web. In Proceedings of the Second Workshop on Language in Social Media, pages 19–26, Montréal, Canada, June. Association for Computational Linguistics.
  16. Waseem, Z. and Hovy, D. (2016). Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In Proceedings of the NAACL student research workshop, pages 88–93.
  17. Yin, D., Xue, Z., Hong, L., Davison, B. D., Kontostathis, A., and Edwards, L. (2009). Detection of harassment on web 2.0. Proceedings of the Content Analysis in the WEB, 2:1–7. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/saeed-2020-osact4-shared-tasks-ensembled-stacked/