Elamur Research⌕
☰
home / papers / 2020 / ALT at SemEval-2020 Task 12: Arabic and English Offensive Language Identification in Social Media

ALT at SemEval-2020 Task 12: Arabic and English Offensive Language Identification in Social Media

Paper · 2020 · Sabit Hassan, Younes Samih, Hamdy Mubarak et al.

Summary

The paper reports the ALT team's entry for sub-task A of SemEval-2020 Task 12, which asks whether a tweet is offensive (OFF) or not (NOT). The team worked on Arabic and English. For Arabic, the 8,000 labelled training tweets were normalised and used with several classifiers, tuned by 5-fold cross-validation: fastText, multilingual BERT, a character-CNN plus BiLSTM network using Mazajak embeddings, and tf-idf n-gram SVMs. An SVM was further patched with a lexical list derived from valence scores. The final Arabic system takes a majority vote over three very different models and placed second of 53 teams, only 0.01 F1 below the winner. The English training data consisted of about 9M tweets labelled only with averaged machine confidence scores. The team turned these scores into silver labels by thresholding, added Wikipedia Detox, and fine-tuned multilingual BERT, which placed 18th of 82. A manual analysis of errors highlights two recurring problems: sarcasm and humour, and noisy gold labels.

Key points

  • Arabic: majority vote of SVM & ValenceList, C-LSTM and Mult-BERT reached macro-F1 90.16% on the 2,000-tweet test set (2nd/53; top was 90.17%). The best single model in cross-validation was SVM + ValenceList at 88.9%.
  • Mult-BERT alone was weak on Arabic tweets (82.9% in CV). The authors attribute this to the gap between its Wikipedia pretraining text and informal Twitter language.
  • ValenceList contains the 160 most frequent words with valence score 1 toward OFF. Any tweet the SVM marked NOT but containing one of these words was relabelled OFF.
  • English: training tweets came with a mean confidence and a standard deviation from several models instead of gold labels. Low-agreement items were dropped, and OLID 2019 served as the dev set. Mult-BERT trained on these labels plus Wikipedia Detox scored 89% on OLID 2019 and 91.14% on the 3,887-tweet test set (18th/82; top 92.04%).
  • In 100 sampled errors per language, humour/sarcasm caused 15% (Arabic) and 22% (English) of false positives, and 29% (Arabic) and 9% (English) of false negatives. Label noise accounted for roughly 13–29% across error types.

Interesting ideas

  • I1. Turn soft scores from many models into training labels by keeping only items where the mean is clearly on one side and the spread between models is small, so confusing cases never enter training.
  • I2. Place a tiny, inspectable lexical override after a statistical classifier: a short list of words that strongly signal one class flips only the predictions it can confidently correct.
  • I3. Build the ensemble from models that differ in kind (n-gram SVM, character/word neural net, pretrained transformer) and combine them by plain majority vote to cancel each model's typical mistakes.
  • I4. Hand-label a random sample of about 100 errors into a small taxonomy that includes "the gold label is wrong", which separates model weaknesses from dataset noise.
  • I5. Treat sarcasm and humour as a distinct failure class, because surface hostile words often mislead the classifier in both directions.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver trees agreed by witness parsers could be filtered the same way, keeping arcs only when parser confidences agree tightly and dropping high-variance sentences.
  • I2 (frames): An insult or hostile speech-act detector could use such a class-distinctive word list as an explainable rule layer that names the word responsible for each flip.
  • I3 (tree): The perceptron parser and gparse could be paired with a third parser of a different kind, with per-arc majority voting and each disagreement logged for explanation.
  • I4 (learn): Evaluation reports could add a fixed error-category audit with an explicit annotation-noise bucket, so self-induced repair rules are not built from bad gold data.
  • I5 (frames): The irony module could mark utterances with banter cues, so offensive-word evidence is weighed differently before deciding the speech act.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksCommunication · 97 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,263 worksSentiment & emotion · 972 worksSemitic languages · 2,908 works

Cite

Cite as Sabit Hassan, Younes Samih, Hamdy Mubarak and Ahmed Abdelali. 2020. ALT at SemEval-2020 Task 12: Arabic and English Offensive Language Identification in Social Media. Proceedings of the Fourteenth Workshop on Semantic Evaluation.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.semeval-1.249

@inproceedings{hassan2020alt,
  title = {{ALT at SemEval-2020 Task 12: Arabic and English Offensive Language Identification in Social Media}},
  author = {Sabit Hassan, Younes Samih, Hamdy Mubarak and Ahmed Abdelali},
  booktitle = {Proceedings of the Fourteenth Workshop on Semantic Evaluation},
  year = {2020},
  doi = {10.18653/v1/2020.semeval-1.249},
  url = {https://doi.org/10.18653/v1/2020.semeval-1.249}
}

References

21 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Hamdy Mubarak, Younes Samih, Sabit Hassan, and Kareem Darwish. 2020. Arabic dialect identification in the wild. ArXiv, abs/2005.06557. ↗
  2. Ibrahim Abu Farha and Walid Magdy. 2019. Mazajak: An online Arabic sentiment analyser. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 192–198, Florence, Italy, August. Association for Computational Linguistics.
  3. Azalden Alakrot, Liam Murray, and Nikola S. Nikolov. 2018. Towards accurate detection of offensive language in online communication in arabic. Procedia Computer Science, 142:315 – 320. Arabic Computational Linguistics. find
  4. N. Albadi, M. Kurdi, and S. Mishra. 2018. Are they our brothers? analysis and detection of religious hate speech in the arabic twittersphere. In 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 69–76, Aug. find
  5. Houda Bouamor, Sabit Hassan, and Nizar Habash. 2019. The MADAR shared task on Arabic fine-grained dialect identification. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 199–207, Florence, Italy, August. Association for Computational Linguistics.
  6. Michael Conover, Jacob Ratkiewicz, Matthew R Francisco, Bruno Gonçalves, Filippo Menczer, and Alessandro Flammini. 2011. Political polarization on twitter. ICWSM, 133:89–96. find
  7. Thomas Davidson, Dana Warmsley, Michael W. Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In ICWSM. ↗
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June. Association for Computational Linguistics. ↗
  9. Nemanja Djuric, Jing Zhou, Robin Morris, Mihajlo Grbovic, Vladan Radosavljevic, and Narayan Bhamidipati. 2015. Hate speech detection with comment embeddings. In Proceedings of the 24th International Conference on World Wide Web Companion, pages 29–30. International World Wide Web Conferences Steering Committee. find
  10. AbdelRahim Elmadany, Hamdy Mubarak, and Walid Magdy. 2018. Arsas : An arabic speech-act and sentiment corpus of tweets. find
  11. Sabit Hassan, Younes Samih, Hamdy Mubarak, Ahmed Abdelali, Ammar Rashed, and Shammur Absar Chowdhury. 2020. ALT submission for OSACT shared task on offensive language detection. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 61–65, Marseille, France, May. European Language Resource Association.
  12. Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hervé Jégou, and Tomas Mikolov. 2016. Fasttext.zip: Compressing text classification models. CoRR, abs/1612.03651. Quoc V. Le and Tomas Mikolov. abs/1405.4053. 2014. Distributed representations of sentences and documents. ↗
  13. CoRR, Ping Liu, Wen Li, and Liang Zou. 2019. Nuli at semeval-2019 task 6: Transfer learning for offensive language detection using bidirectional transformers. In SemEval@NAACL-HLT.
  14. Hamdy Mubarak and Kareem Darwish. 2019. Arabic offensive language classification on twitter. In International Conference on Social Informatics, pages 269–276. Springer. find
  15. Hamdy Mubarak, Kareem Darwish, and Walid Magdy. 2017. Abusive language detection on Arabic social media. In Proceedings of the First Workshop on Abusive Language Online, pages 52–56, Vancouver, BC, Canada, August. Association for Computational Linguistics.
  16. Hamdy Mubarak, Ammar Rashed, Kareem Darwish, Younes Samih, and Ahmed Abdelali. 2020. Arabic offensive language on twitter: Analysis and experiments. arXiv preprint arXiv:2004.02192. ↗
  17. Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Marcos Zampieri, and Preslav Nakov. 2020. A Large-Scale Semi-Supervised Dataset for Offensive Language Identification. In arxiv. find
  18. Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex machina: Personal attacks seen at scale. In Proceedings of the 26th International Conference on World Wide Web, WWW ’17, page 1391–1399, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee. ↗
  19. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019a. Predicting the Type and Target of Offensive Posts in Social Media. In Proceedings of NAACL.
  20. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019b. SemEval-2019 task 6: Identifying and categorizing offensive language in social media (OffensEval). In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 75–86, Minneapolis, Minnesota, USA, June. Association for Computational Linguistics.
  21. Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Çağrı Çöltekin. 2020. SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020). In Proceedings of SemEval. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hassan-2020-alt-semeval-2020-task-12/