Elamur Research⌕
☰
home / papers / 2020 / Leveraging Affective Bidirectional Transformers for Offensive Language Detection

Leveraging Affective Bidirectional Transformers for Offensive Language Detection

Paper · 2020 · AbdelRahim Elmadany, Chiyu Zhang, Muhammad Abdul-Mageed et al.

Summary

The paper describes a system built for the OSACT4 shared task on detecting offensive language (Sub-task A) and hate speech (Sub-task B) in Arabic tweets. The labelled data are small and skewed: only about a fifth of tweets are offensive and about one in twenty is hateful. The authors address this in two ways, both using BERT and no hand-crafted features. First, they harvest many extra training tweets from a large in-house collection. A tweet is kept only if it contains an item from a seed lexicon and its sentiment label, predicted by the AraNet toolkit, agrees with the target class. Second, they start fine-tuning from models already trained on sentiment or emotion, not from plain BERT. Both steps beat a vanilla BERT baseline. On the official TEST split, the best reported scores reach roughly 82–83 macro F1 for offensiveness. The work matters because offensive-content resources for Arabic are scarce. It also shows that cheap, noisy data expansion plus affect-oriented pretraining can help rare-class detection.

Key points

  • Shared task data: 10,000 manually annotated tweets, split by the organizers 70/10/20 into TRAIN (6,994), DEV (1,000) and TEST (2,000). Offensive tweets are about 20% of TRAIN and hateful ones about 5%, so macro F1 is the official metric.
  • Augmented sets: AUG-TRAIN-OFF has 480,777 tweets, 55% of them offensive. AUG-TRAIN-HS has 209,780 tweets, of which 10,489 are hateful.
  • Sentiment filter rationale: on TRAIN, AraNet tags 65% of offensive tweets and 72% of hateful tweets as negative. Hence positive-class tweets are drawn only from negatively predicted tweets, and negative-class tweets only from positively predicted ones.
  • The abstract reports 89.60% acc (82.31 macro F1) and 95.20% acc (70.51 macro F1) on TEST across the two tasks. BERT-EMO-AUG (emotion model plus augmented data) is the best offensive submission, at 89.35% acc and 82.85 F1.
  • The authors find that affective fine-tuning helps most for offensive language detection.

Interesting ideas

  • I1. When harvesting silver examples by keyword, require an independent classifier's prediction to agree with the intended label, which turns a loose lexicon match into a cross-checked sample.
  • I2. Before imposing a filter, the authors measured on gold TRAIN data how strongly sentiment really correlates with each class, so the filtering assumption was tested rather than assumed.
  • I3. Starting fine-tuning from a model trained on a neighbouring task (sentiment or emotion) gave better rare-class detection than generic pretraining, because the related signal was already encoded.
  • I4. The paper argues that word lists fail because offensiveness shifts with context and genre and new terms keep appearing, which motivates context-sensitive models.
  • I5. Scoring imbalanced binary tasks with macro F1 prevents high accuracy on the majority class from hiding weak minority-class performance.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver sentences collected by pattern could be kept only when an unrelated witness, such as a different parser or tagger, agrees with the assumed analysis.
  • I2 (learn): Any heuristic used to admit silver data should first be checked against gold data, and the measured agreement rate should be reported as a sanity control.
  • I3 (frames): Affect cues already computed for a text could serve as prior evidence when deciding whether an utterance is an insult or ironic attack, rather than learning that judgement from scratch.
  • I4 (frames): Treat insult or threat as a speech act decided by who is targeted and how, read from the frame, instead of flagging lexical items alone.
  • I5 (learn): Evaluations of rare phenomena such as irony or rare dependency relations should report per-class or macro scores alongside overall accuracy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of British ColumbiaVancouver · Canada98 works · 34,805 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 works

Cite

Cite as AbdelRahim Elmadany, Chiyu Zhang, Muhammad Abdul-Mageed and Azadeh Hashemi. 2020. Leveraging Affective Bidirectional Transformers for Offensive Language Detection. Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{elmadany2020leveraging,
  title = {{Leveraging Affective Bidirectional Transformers for Offensive Language Detection}},
  author = {AbdelRahim Elmadany, Chiyu Zhang, Muhammad Abdul-Mageed and Azadeh Hashemi},
  booktitle = {Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection},
  year = {2020},
  url = {https://aclanthology.org/2020.osact-1.17/}
}

References

26 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdul-Mageed, M., Zhang, C., Elmadany, A., Rajendran, A., and Ungar, L. (2019). Dianet: Bert and hierarchical attention multi-task learning of fine-grained dialect. arXiv preprint arXiv:1910.14243. ↗
  2. Abdul-Mageed, Muhammad, Z. C., Nagoudi, E. M. B., and Hashemi, A. (2020). Aranet: A deep learning toolkit for arabic social media. In The 4th Workshop on Open-Source Arabic Corpora and Processing Tools (OSACT4), LREC. ↗
  3. Abozinadah, E. (2017). Detecting abusive arabic language twitter accounts using a multidimensional analysis model. Ph.D. thesis. ↗
  4. Agrawal, S. and Awekar, A. (2018). Deep learning for detecting cyberbullying across multiple social media platforms. In European Conference on Information Retrieval, pages 141–153. Springer. find
  5. Alakrot, A., Murray, L., and Nikolov, N. S. (2018). Towards accurate detection of offensive language in online communication in arabic. Procedia computer science, 142:315–320. find
  6. Albadi, N., Kurdi, M., and Mishra, S. (2018). Are they our brothers? analysis and detection of religious hate speech in the arabic twittersphere. In 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 69–76. IEEE. find
  7. Badjatiya, P., Gupta, S., Gupta, M., and Varma, V. (2017). Deep learning for hate speech detection in tweets. In Proceedings of the 26th International Conference on World Wide Web Companion, pages 759–760. find
  8. Barberá, P. and Sood, G. (2015). Follow your ideology: Measuring media ideology on social networks. In Annual Meeting of the European Political Science Association, Vienna, Austria. Retrieved from http://www.gsood. com/research/papers/mediabias. pdf. ↗
  9. Chadefaux, T. (2014). Early warning signals for war in the news. Journal of Peace Research, 51(1):5–18. find
  10. Conover, M. D., Ratkiewicz, J., Francisco, M., Gonçalves, B., Menczer, F., and Flammini, A. (2011). Political polarization on twitter. In Fifth international AAAI conference on weblogs and social media. find
  11. Dadvar, M., Trieschnigg, D., Ordelman, R., and de Jong, F. (2013). Improving cyberbullying detection with user context. In European Conference on Information Retrieval, pages 693–696. Springer. find
  12. Darwish, K., Alexandrov, D., Nakov, P., and Mejova, Y. (2017). Seminar users in the arabic twitter sphere. In International Conference on Social Informatics, pages 91– 108. Springer. ↗
  13. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  14. Fortuna, P., Ferreira, J., Pires, L., Routar, G., and Nunes, S. (2018). Merging datasets for aggressive text identification. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018), pages 128– 139. find
  15. Georgakopoulos, S. V., Tasoulis, S. K., Vrahatis, A. G., and Plagianakos, V. P. (2018). Convolutional neural networks for toxic comment classification. In Proceedings of the 10th Hellenic Conference on Artificial Intelligence, pages 1–6. find
  16. Jay, T. and Janschewitz, K. (2008). The pragmatics of swearing. Journal of Politeness Research. Language, Behaviour, Culture, 4(2):267–288. ↗
  17. Kumar, R., Ojha, A. K., Malmasi, S., and Zampieri, M. (2018). Benchmarking aggression identification in social media. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018), pages 1–11.
  18. Kwok, I. and Wang, Y. (2013). Locate the hate: Detecting tweets against blacks. In Twenty-seventh AAAI conference on artificial intelligence. find
  19. Malmasi, S. and Zampieri, M. (2017). Detecting hate speech in social media. arXiv preprint arXiv:1712.06427. ↗
  20. Modha, S., Majumder, P., and Mandl, T. (2018). Filtering aggression from the multilingual social media feed. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018), pages 199–207. find
  21. Mubarak, H. and Darwish, K. (2019). Arabic offensive language classification on twitter. In International Conference on Social Informatics, pages 269–276. Springer. find
  22. Mubarak, H., Darwish, K., and Magdy, W. (2017). Abusive language detection on arabic social media. In Proceedings of the First Workshop on Abusive Language Online, pages 52–56.
  23. Nobata, C., Tetreault, J., Thomas, A., Mehdad, Y., and Chang, Y. (2016). Abusive language detection in online user content. In Proceedings of the 25th international conference on world wide web, pages 145–153. find
  24. Weber, I., Garimella, V. R. K., and Batayneh, A. (2013). Secular vs. islamist polarization in egypt on twitter. In Proceedings of the 2013 IEEE/ACM international conference on advances in social networks analysis and mining, pages 290–297. find
  25. Wiegand, M., Siegel, M., and Ruppenhofer, J. (2018). Overview of the germeval 2018 shared task on the identification of offensive language. find
  26. Zampieri, M., Malmasi, S., Nakov, P., Rosenthal, S., Farra, N., and Kumar, R. (2019). Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval). arXiv preprint arXiv:1903.08983. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/elmadany-2020-leveraging-affective-bidirectional-transformers-offensive/