Elamur Research⌕
☰
home / papers / 2020 / Overview of OSACT4 Arabic Offensive Language Detection Shared Task

Overview of OSACT4 Arabic Offensive Language Detection Shared Task

Paper · 2020 · Hamdy Mubarak, Kareem Darwish, Walid Magdy et al.

Summary

The paper reports on the OSACT4 shared task, which asked systems to find abusive content in Arabic tweets. There were two binary subtasks. Subtask A was general offensiveness, covering vulgarity and insults aimed at people or groups. Subtask B was hate speech, meaning offensive content that targets a group by nationality, ethnicity, religion, gender, political or sports affiliation, and similar traits. The organizers reused the 10,000-tweet Arabic set from OffensEval2020. To get more offensive examples, they collected tweets containing the vocative particle "yA". One native annotator labelled the data, and three more annotators checked a sample. Systems were scored on CodaLab, with macro-averaged F1 as the official metric because the classes are very unbalanced. The best systems reached F1 90.5 on Subtask A and 95.2 on Subtask B. They combined Arabic-specific preprocessing with ensembles of classical classifiers and neural models. The task gives the field a common benchmark and a snapshot of what worked for Arabic offensive language around 2020.

Key points

  • Dataset: 10,000 tweets from OffensEval2020, split 70/10/20 into train/dev/test.
    • They were drawn from 660k tweets posted between April 15 and May 6, 2019.
    • Roughly 20% are labelled OFF (1,991 tweets) and about 5% HS (506 tweets).
  • Why the vocative filter: in random samples, under 2% of tweets were offensive. Tweets with one vocative raised this to about 5%, and tweets with two vocatives to about 20%.
  • Annotation quality: three extra annotators judged a balanced sample of 100 tweets (50 offensive, 50 not). Their agreement was a Fleiss's Kappa of 0.92.
  • Participation: 40 teams registered for Subtask A and 33 for Subtask B. Fourteen and 13 teams submitted test runs, and 11 wrote system papers.
  • Best results and methods:
    • Subtask A: F1 90.5 (Hassan et al.), using an ensemble of SVM on character n-grams, Mazajak embeddings, FastText, CNN+RNN and multilingual BERT.
    • Subtask B: F1 95.2 (Husain), using heavy preprocessing.
    • Other teams used SVM, logistic regression, CNN/RNN variants and fine-tuned AraBERT. Two teams used multitask learning.

Interesting ideas

  • I1. The organizers did not label random tweets. They first filtered by a cue of direct address (the vocative), which made the rare offensive class far more frequent in the pool to be annotated.
  • I2. Quality was checked cheaply: extra annotators judged a small set balanced between the two classes, giving a chance-corrected agreement score without relabelling everything.
  • I3. The winning hate-speech system replaced specific words with semantic categories (e.g. dog or monkey becomes ANIMAL), so the classifier learns patterns at the level of concept classes.
  • I4. Macro-averaged F1 was the official score, so the small positive class counted as much as the large negative one.
  • I5. Each team named two official runs before results were known, and the first run always beat the second, showing that dev-set choices carried over to test.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when building silver data for rare constructions, we could pre-filter the corpus with a syntactic cue from the UD tree (e.g. a vocative dependent) so annotation effort lands on the cases we actually need.
  • I2 (learn): to validate witness-parser or LLM-teacher labels, we could have a few independent judges score a small class-balanced sample and report kappa before trusting the full silver set.
  • I3 (frames): an insult speech-act detector could check whether a vocative or predicate noun belongs to an ontology class like an animal hypernym in WordNet, giving an explainable rule instead of a word list.
  • I4 (learn): evaluations of rare phenomena such as irony or implicature should report per-class and macro scores, so a model that ignores the minority class cannot look good.
  • I5 (learn): we could fix a primary configuration before touching held-out data and treat the alternate run as a check that dev improvements are real.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksSocial Sciences · 744 worksCommunication · 44 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 608 worksSemitic languages · 1,484 works

Cite

Cite as Hamdy Mubarak, Kareem Darwish, Walid Magdy, Tamer Elsayed and Hend Al-Khalifa. 2020. Overview of OSACT4 Arabic Offensive Language Detection Shared Task. Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{mubarak2020overview,
  title = {{Overview of OSACT4 Arabic Offensive Language Detection Shared Task}},
  author = {Hamdy Mubarak, Kareem Darwish, Walid Magdy, Tamer Elsayed and Hend Al-Khalifa},
  booktitle = {Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection},
  year = {2020},
  url = {https://aclanthology.org/2020.osact-1.7/}
}

References

16 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdellatif, M. and Elgammal, A. (2020). Sentiment analysis of imbalanced arabic data using ulmfit. OSACT, 4. find
  2. Abu Farha, I. and Magdy, W. (2020). Multitask learning for arabic offensive language and hate-speech detection. OSACT, 4. find
  3. Abuzayed, A. and Elsayed, T. (2020). Quick and simple approach for detecting hate speech in arabic tweets. OS-ACT, 4. find
  4. Team Abeer (Abuzayed and Elsayed, 2020) aialharbi (Alharbi and Lee, 2020) alisafaya alt (Hassan et al., 2020) find
  5. AMR-KELEG (Keleg et al., 2020) find
  6. Bushr (Haddad et al., 2020) elmadany (Elmadany et al., 2020) fatemah (Husain, 2020) hassaansaeed (Saeed et al., 2020) iaf7 (Abu Farha and Magdy, 2020) mabdellatif (Abdellatif and Elgammal, 2020) find
  7. Marc Djandji (Djandji et al., 2020) premjithb SAJA Affiliation Islamic University of Gaza, Palestine University of Birmingham, UK Koç University, Turkey Qatar Computing Research Institute, Qatar Faculty of Engineering, Ain Shams University, Egypt Damascus University, Syria University of British Columbia, NLP Lab, Canada Kuwait University, Dep. of Information Science, Kuwait University of Antwerpen, Belgium University of Edinburgh, UK Rutgers University, US American University of Beirut. Lebanon Center for Comp. Engineering and Networking, India Jordan University of Science and Technology, Jordan Table 2: List of participating teams in Subtasks A and B Alharbi, A. and Lee, M. (2020). Combining character and word embeddings for the detection of offensive language in arabic. OSACT, 4. find
  8. Djandji, M., Baly, F., antoun, w., and Hajj, H. (2020). Multi-task learning using arabert for offensive language detection. OSACT, 4. find
  9. Elmadany, A., Zhang, C., Abdul-Mageed, M., and Hashemi, A. (2020). Leveraging affective bidirectional transformers for offensive language detection. OSACT, 4. ↗
  10. Haddad, B., Orabe, Z., Al-Abood, A., and Ghneim, N. (2020). Arabic offensive language detection with attention-based deep neural networks. OSACT, 4. ↗
  11. Hassan, S., Samih, Y., Mubarak, H., Abdelali, A., Rashed, A., and Absar Chowdhury, S. (2020). Alt submission for osact shared task on offensive language detection. OS-ACT, 4. find
  12. Husain, F. (2020). Osact4 shared task on offensive language detection: Intensive preprocessing based approach. OSACT, 4. ↗
  13. Keleg, A., El-Beltagy, S. R., and Khalil, M. (2020). Asu opto at osact4 - offensive language detection for arabic text. OSACT, 4. find
  14. Saeed, H. H., Calders, T., and Kamiran, F. (2020). Ocast4 shared tasks: Ensembled stacked classification for offensive and hate speech in arabic tweets. OSACT, 4. Subtasks 1,2 1,2 1,2 1,2 1,2 1,2 1,2 1,2 1,2 1,2 1,2 1,2 1,2 1 find
  15. Team (Abdellatif and Elgammal, 2020) (Abu Farha and Magdy, 2020) Preprocessing simple tokenization, replace words below certain threshold with a special token character normalization and diacritic, kashida, repeated letter, and non-Arabic character removal character normalization and diacritic, kashida, repeated letter, and non-Arabic character removal (Abuzayed and Elsayed, 2020) find
  16. (Alharbi and Lee, 2020) character normalization and diacritic, kashida, repeated letter, and non-Arabic character removal. Also split AK (“yA”) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mubarak-2020-overview-osact4-arabic-offensive-language/