Elamur Research⌕
☰
home / papers / 2020 / A Multi-Platform Arabic News Comment Dataset for Offensive Language Detection

A Multi-Platform Arabic News Comment Dataset for Offensive Language Detection

Summary

Most Arabic resources for detecting offensive language come from a single social network, usually Twitter or YouTube. Abusive commenting, however, happens on every platform. The authors build a dialectal Arabic dataset of news comments drawn from three platforms (Twitter, Facebook and YouTube), all posted under the accounts of one international news agency. They chose news because the posts themselves are not offensive, so any hostility in the replies reflects the commenters' own views.

From more than 100k collected comments they sampled 4000 and labelled them through crowdsourcing. Annotators had to pass a two-stage selection process, which matters for an under-represented language. Label quality was checked in two ways: agreement among the accepted workers, and comparison with expert annotations.

The paper also studies which words and emojis mark offensive comments. SVM classifiers trained on this data and on other public Arabic sets show that multi-platform training data helps models generalize across platforms, domains and dialects (Egyptian, Levantine). The dataset is publicly released.

Key points

  • The corpus contains 4000 annotated comments from Twitter, Facebook and YouTube. They were sampled from over 100k comments on posts a news agency published between 2011 and 2019. The data is released on GitHub (Arabic-Offensive-Multi-Platform-SocialMedia-Comment-Dataset).
  • Twitter's Standard Search API only returns replies from the past 7 days. To get the full reply history, the authors re-queried every 6 hours.
  • Filtering and cleaning steps:
    • comments needed at least 5 Arabic tokens, not counting emojis;
    • user mentions were replaced with USER.IDX and links with URL;
    • duplicate texts were removed.
  • Annotation used a two-step crowd-worker screening process. Quality was measured by inter-annotator agreement and by accuracy against expert labels.
  • SVM experiments covered in-domain and out-of-domain data, cross-platform tests and Egyptian/Levantine dialect sets. The external datasets were:
    • Mubarak et al. 2017: about 1.1k Egyptian tweets, plus about 32k deleted Aljazeera.net comments;
    • Mulki et al. 2019: L-HSAB, 5.8k Levantine tweets.

Interesting ideas

  • I1. The source posts were chosen to be neutral, so offensiveness can be credited to the replying user rather than the content they react to; this kind of clean attribution makes analysis easier in any stance study. — For Mova (frames): A test set of neutral prompts with charged replies would let the speech-act and irony modules be checked on whether they assign the stance to the right speaker.
  • I2. Crowd workers had to pass a screening stage, and their labels were then measured against an expert sample, so cheap annotators earned trust before their labels were used. — For Mova (learn): LLM teacher tiers could go through the same qualification against a small gold set before their silver labels are allowed into training.
  • I3. Generalization was tested by training on one platform or dialect and evaluating on another, which reveals models that only fit one source's style. — For Mova (learn): Holding out an entire genre (tweets, fables, textbooks) when evaluating the parsers would serve as a negative control against genre overfitting.
  • I4. Emojis were kept and analysed as signals of offensiveness instead of being stripped as noise, since they carry meaning in informal text. — For Mova (tokens): The tokenizer and tagger could give emojis their own token class and lemma, so later pragmatic stages can use them as evaluative cues.
  • I5. Mentions were anonymized with indexed placeholders (USER.IDX), which hides identities but keeps track of which person is which. — For Mova (frames): Coreference could rely on indexed placeholders like these when anonymizing input, so frames still distinguish participants without exposing names.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

30 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 23 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{chowdhury2020multi,
  title = {{A Multi-Platform Arabic News Comment Dataset for Offensive Language Detection}},
  author = {Chowdhury, Shammur Absar and Mubarak, Hamdy and Abdelali, Ahmed and Jung, Soon-gyo and Jansen, Bernard J. and Salminen, Joni},
  booktitle = {Proceedings of the Twelfth Language Resources and Evaluation Conference},
  year = {2020},
  url = {https://aclanthology.org/2020.lrec-1.761/}
}

References

35 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agarwal, S. and Sureka, A. (2014). A focused crawler for mining hate and extremism promoting videos on youtube. In Proceedings of the 25th ACM conference on Hypertext and social media, pages 294–296. ACM. find
  2. Al-Ajlan, M. A. and Ykhlef, M. (2018). Optimized twitter cyberbullying detection based on deep learning. In 2018 21st Saudi Computer Society National Computer Conference (NCC), pages 1–5. IEEE. find
  3. Alakrot, A., Murray, L., and Nikolov, N. S. (2018). Dataset construction for the detection of anti-social behaviour in online communication in arabic. Procedia Computer Science, 142:174–181. find
  4. Albadi, N., Kurdi, M., and Mishra, S. (2018). Are they our brothers? analysis and detection of religious hate speech in the arabic twittersphere. In 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 69–76. IEEE. find
  5. Badjatiya, P., Gupta, S., Gupta, M., and Varma, V. (2017). Deep learning for hate speech detection in tweets. In Proceedings of the 26th International Conference on World Wide Web Companion, pages 759–760. International World Wide Web Conferences Steering Committee. find
  6. Chatzakou, D., Kourtellis, N., Blackburn, J., De Cristofaro, E., Stringhini, G., and Vakali, A. (2017a). Hate is not binary: Studying abusive behavior of# gamergate on twitter. In Proceedings of the 28th ACM conference on hypertext and social media, pages 65–74. ACM. find
  7. Chatzakou, D., Kourtellis, N., Blackburn, J., De Cristofaro, E., Stringhini, G., and Vakali, A. (2017b). Mean birds: Detecting aggression and bullying on twitter. In Proceedings of the 2017 ACM on web science conference, pages 13–22. ACM. find
  8. Chatzakou, D., Kourtellis, N., Blackburn, J., De Cristofaro, E., Stringhini, G., and Vakali, A. (2017c). Measuring# gamergate: A tale of hate, sexism, and bullying. In Proceedings of the 26th international conference on world wide web companion, pages 1285–1290. International World Wide Web Conferences Steering Committee. find
  9. Conover, M. D., Ratkiewicz, J., Francisco, M., Gonçalves, B., Menczer, F., and Flammini, A. (2011). Political polarization on twitter. In Fifth international AAAI conference on weblogs and social media. find
  10. Dadvar, M., Trieschnigg, D., Ordelman, R., and de Jong, F. (2013). Improving cyberbullying detection with user context. In European Conference on Information Retrieval, pages 693–696. Springer. find
  11. Davidson, T., Warmsley, D., Macy, M., and Weber, I. (2017). Automated hate speech detection and the problem of offensive language. In Eleventh international aaai conference on web and social media. find our card
  12. Difallah, D., Filatova, E., and Ipeirotis, P. (2018). Demographics and dynamics of mechanical turk workers. In Proceedings of the eleventh acm international conference on web search and data mining, pages 135–143. ACM. find
  13. ElSherief, M., Nilizadeh, S., Nguyen, D., Vigna, G., and Belding, E. (2018). Peer to peer hate: Hate speech instigators and their targets. In Twelfth International AAAI Conference on Web and Social Media. find
  14. Falotico, R. and Quatto, P. (2015). Fleiss’ kappa statistic without paradoxes. Quality & Quantity, 49(2):463–470. find
  15. Gülaçtı, F. (2010). The effect of perceived social support on subjective well-being. Procedia-Social and Behavioral Sciences, 2(2):3844–3849. find
  16. Hosseinmardi, H., Mattson, S. A., Rafiq, R. I., Han, R., Lv, Q., and Mishra, S. (2015). Analyzing labeled cyberbullying incidents on the instagram social network. In International conference on social informatics, pages 49–66. Springer. find
  17. Intapong, P., Charoenpit, S., Achalakul, T., and Ohkura, M. (2017). Assessing symptoms of excessive sns usage based on user behavior and emotion: analysis of data obtained by sns apis. In 9th International Conference on Social Computing and Social Media, SCSM 2017 held as part of the 19th International Conference on Human-Computer Interaction, HCI International 2017, pages 71–83. Springer Verlag. find
  18. Jay, T. and Janschewitz, K. (2008). The pragmatics of swearing. Journal of Politeness Research. Language, Behaviour, Culture, 4(2):267–288. find
  19. Kumar, S., Hamilton, W. L., Leskovec, J., and Jurafsky, D. (2018). Community interaction and conflict on the web. In Proceedings of the 2018 World Wide Web Conference, pages 933–943. International World Wide Web Conferences Steering Committee. find
  20. Mondal, M., Silva, L. A., and Benevenuto, F. (2017). A measurement study of hate speech in social media. In Proceedings of the 28th ACM Conference on Hypertext and Social Media, pages 85–94. ACM. find
  21. Mubarak, H. and Darwish, K. (2019). Arabic offensive language classification on twitter. In International Conference on Social Informatics, pages 269–276. Springer. find
  22. Mubarak, H., Darwish, K., and Magdy, W. (2017). Abusive language detection on arabic social media. In Proceedings of the First Workshop on Abusive Language Online, pages 52–56. find our card
  23. Mulki, H., Haddad, H., Ali, C. B., and Alshabani, H. (2019). L-hsab: A levantine twitter dataset for hate speech and abusive language. In Proceedings of the Third Workshop on Abusive Language Online, pages 111–118. find
  24. Pavlopoulos, J., Malakasiotis, P., and Androutsopoulos, I. (2017). Deeper attention to abusive user content moderation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1125–1135. find
  25. Pitsilis, G. K., Ramampiaro, H., and Langseth, H. (2018). Detecting offensive language in tweets using deep learning. arXiv preprint arXiv:1801.04433. ↗
  26. Platt, J. (1998). Sequential minimal optimization: A fast algorithm for training support vector machines. find
  27. Rish, I. et al. (2001). An empirical study of the naive bayes classifier. In IJCAI 2001 workshop on empirical methods in artificial intelligence, volume 3, pages 41–46. find
  28. Ross, J., Zaldivar, A., Irani, L., and Tomlinson, B. (2009). Who are the turkers? worker demographics in amazon mechanical turk. Department of Informatics, University of California, Irvine, USA, Tech. Rep. find
  29. Salminen, J., Almerekhi, H., Milenkovic, M., Jung, S.-g., An, J., Kwak, H., and Jansen, B. J. (2018). Anatomy of Online Hate: Developing a Taxonomy and Machine Learning Models for Identifying and Classifying Hate in Online News Media. In Proceedings of The International AAAI Conference on Web and Social Media (ICWSM 2018), San Francisco, California, USA, June. find
  30. Salminen, J., Hopf, M., Chowdhury, S. A., Jung, S.-g., Almerekhi, H., and Jansen, B. J. (2020). Developing an online hate classifier for multiple social media platforms. Human-centric Computing and Information Sciences, 10(1). find
  31. Silva, L., Mondal, M., Correa, D., Benevenuto, F., and Weber, I. (2016). Analyzing the targets of hate in online social media. In Tenth International AAAI Conference on Web and Social Media. find
  32. Unsvåg, E. F. and Gambäck, B. (2018). The effects of user features on twitter hate speech detection. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2), pages 75–85. find
  33. Waldron, J. (2012). The harm in hate speech. Harvard University Press. find
  34. Wiegand, M., Siegel, M., and Ruppenhofer, J. (2018). Overview of the germeval 2018 shared task on the identification of offensive language. find
  35. Zhang, Z., Robinson, D., and Tepper, J. (2018). Detecting hate speech on twitter using a convolution-gru based deep neural network. In European Semantic Web Conference, pages 745–760. Springer. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chowdhury-2020-multi-platform-arabic-news-comment/