Elamur Research⌕
☰
home / papers / 2020 / Multi-Task Learning using AraBert for Offensive Language Detection

Multi-Task Learning using AraBert for Offensive Language Detection

Paper · 2020 · Marc Djandji, Fady Baly, Wissam Antoun et al.

Summary

The paper describes the "BERTologists" team's system for the OSACT4 shared task on Arabic tweets. Subtask A asks whether a tweet is offensive. Subtask B asks whether it is hate speech. Both are scored by macro-F1. The data are hard for two reasons: tweets mix dialects with irregular spelling, and labels are very skewed, with hate speech only about 5% of training examples.

The authors fine-tune the pretrained AraBERT model and compare several ways of handling imbalance:

  • balanced batch sampling
  • a class-weighted cross-entropy loss
  • the two combined
  • a multilabel classifier
  • a multitask setup that learns both subtasks jointly

Input is segmented with Farasa to match AraBERT, and Twitter noise is stripped. The multitask variant performed best on the development set and stayed stable on the test set, taking second place in both subtasks. The study suggests that sharing supervision between closely related tasks helps a rare class more than resampling or loss reweighting do.

Key points

  • Training data: 5468 non-offensive vs 1371 offensive tweets, and 6489 non-hate vs 350 hate tweets. The dev set has 179 offensive and 44 hate tweets out of 1000.
  • Dev macro-F1, Multitask AraBERT: 90.15 for offensive and 83.41 for hate speech. Plain AraBERT scored 89.56 and 80.60.
  • Test macro-F1: 90 (Task A) and 82.28 (Task B), second place in both subtasks of OSACT4.
  • Imbalance methods: Balanced sampling and weighted loss each did worse than the baseline on their own. Combined, they gave the second-best Task A dev score (90.02) but the lowest hate-speech dev score of all variants (78.13).
  • Setup: 5 epochs, batch size 32, sequence length 256. Preprocessing used Farasa segmentation and removed user mentions, RT markers, URLs, <LF>, diacritics and emojis. Hashtag underscores were turned into spaces.

Interesting ideas

  • I1. Training a rare, nested label (hate speech is always also offensive) together with its broader parent label gives the rare class extra signal, and this beat reweighting tricks.
  • I2. The error analysis showed the model had absorbed the rule "no hate without offense," which is correct but made upstream offensive mistakes cascade into false hate predictions.
  • I3. In some cases where the model disagreed with the gold label, the gold label turned out to be wrong, exposing annotation errors in the dev set.
  • I4. Most remaining errors came from three sources: negative sentiment mistaken for offense, insulting words used harmlessly, and sarcasm or quotation of someone else's offensive words.
  • I5. Splitting hashtags at underscores recovers readable words from a single token before the model sees them.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): rare categories such as specific speech-act subtypes or irony could be trained jointly with their coarser parent category instead of alone with oversampling.
  • I2 (frames): when prag enforces hierarchical label constraints, the explanation trace should record which upstream decision triggered the downstream label, so cascading errors can be found and fixed.
  • I3 (learn): confident disagreements between Mova and gold treebank or test annotations could be queued for a gold-label audit rather than counted only as model errors.
  • I4 (frames): separating use from mention and attributing quoted speech to its real speaker in frames would address exactly these failure modes.
  • I5 (tokens): the en tokenizer could split hashtags and similar compound social-media tokens into words, so the tagger and parser get normal lexical items.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksCommunication · 97 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 works

Cite

Cite as Marc Djandji, Fady Baly, Wissam Antoun and Hazem Hajj. 2020. Multi-Task Learning using AraBert for Offensive Language Detection. Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{djandji2020multi,
  title = {{Multi-Task Learning using AraBert for Offensive Language Detection}},
  author = {Marc Djandji, Fady Baly, Wissam Antoun and Hazem Hajj},
  booktitle = {Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection},
  year = {2020},
  url = {https://aclanthology.org/2020.osact-1.16/}
}

References

24 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdelali, A., Darwish, K., Durrani, N., and Mubarak, H. (2016). Farasa: A fast and furious segmenter for arabic. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 11–16.
  2. Al-Hassan, A. and Al-Dossari, H. (2019). Detection of hate speech in social networks: a survey on multilingual corpus. In 6th International Conference on Computer Science and Information Technology. find
  3. Alakrot, A., Murray, L., and Nikolov, N. S. (2018). Towards accurate detection of offensive language in online communication in arabic. Procedia computer science, 142:315–320. find
  4. Albadi, N., Kurdi, M., and Mishra, S. (2018). Are they our brothers? analysis and detection of religious hate speech in the arabic twittersphere. In 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 69–76. IEEE. find
  5. AUBMind-Lab. (2020). https://github.com/aub-mind/arabert. ↗
  6. Basile, V., Bosco, C., Fersini, E., Nozza, D., Patti, V., Pardo, F. M. R., Rosso, P., and Sanguinetti, M. (2019). Semeval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 54–63.
  7. Caruana, R. (1997). Multitask learning. Machine learning, 28(1):41–75. find
  8. Cer, D., Yang, Y., Kong, S.-y., Hua, N., Limtiaco, N., John, R. S., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., et al. (2018). Universal sentence encoder. arXiv preprint arXiv:1803.11175. ↗
  9. Davidson, T., Warmsley, D., Macy, M., and Weber, I. (2017). Automated hate speech detection and the problem of offensive language. In Eleventh international aaai conference on web and social media. ↗
  10. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  11. Fosler-Lussier, E., Riloff, E., and Bangalore, S. (2012). Proceedings of the 2012 conference of the north american chapter of the association for computational linguistics: Human language technologies. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. find
  12. Haddad, H., Mulki, H., and Oueslati, A. (2019). T-hsab: A tunisian hate speech and abusive dataset. In International Conference on Arabic Language Processing, pages 251–263. Springer. find
  13. Mandl, T., Modha, S., Majumder, P., Patel, D., Dave, M., Mandlia, C., and Patel, A. (2019). Overview of the hasoc track at fire 2019: Hate speech and offensive content identification in indo-european languages. In Proceedings of the 11th Forum for Information Retrieval Evaluation, pages 14–17. find
  14. Mohaouchane, H., Mourhir, A., and Nikolov, N. S. (2019). find
  15. Detecting offensive language on arabic social media using deep learning. In 2019 Sixth International Conference on Social Networks Analysis, Management and Security (SNAMS), pages 466–471. IEEE. find
  16. Mubarak, H. and Darwish, K. (2019). Arabic offensive language classification on twitter. In International Conference on Social Informatics, pages 269–276. Springer. find
  17. Mubarak, H., Darwish, K., Magdy, W., Elsayed, T., and Al-Khalifa, H. (2020). Overview of osact4 arabic offensive language detection shared task. 4.
  18. Mulki, H., Haddad, H., Ali, C. B., and Alshabani, H. (2019). L-hsab: A levantine twitter dataset for hate speech and abusive language. In Proceedings of the Third Workshop on Abusive Language Online, pages 111–118.
  19. Qiu, M., Zhao, P., Zhang, K., Huang, J., Shi, X., Wang, X., and Chu, W. (2017). A short-term rainfall prediction model using multi-task convolutional neural networks. In 2017 IEEE International Conference on Data Mining (ICDM), pages 395–404. IEEE. find
  20. Schmidt, A. and Wiegand, M. (2017). A survey on hate speech detection using natural language processing. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, pages 1–10.
  21. Soliman, A. B., Eissa, K., and El-Beltagy, S. R. (2017). Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256–265. find
  22. Tynes, B. M., Giang, M. T., Williams, D. R., and Thompson, G. N. (2008). Online racial discrimination and psychological adjustment among adolescents. Journal of adolescent health, 43(6):565–569. ↗
  23. Waseem, Z. and Hovy, D. (2016). Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In Proceedings of the NAACL student research workshop, pages 88–93.
  24. Zampieri, M., Malmasi, S., Nakov, P., Rosenthal, S., Farra, N., and Kumar, R. (2019). Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval). arXiv preprint arXiv:1903.08983. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/djandji-2020-multi-task-learning-arabert-offensive/