Elamur Research⌕
☰
home / papers / 2020 / Understanding and Detecting Dangerous Speech in Social Media

Understanding and Detecting Dangerous Speech in Social Media

Paper · 2020 · Ali Alshehri, El Moatez Billah Nagoudi and Muhammad Abdul-Mageed

Summary

The paper addresses the detection of dangerous speech in Arabic social media, focusing on direct threats of physical harm. The authors argue that such content is rare but serious, and that earlier work treated it only as a minor part of offensive or hateful language. They define dangerous language narrowly: a stated intention to physically hurt or damage someone as payback. This definition rules out non-physical threats and playful teasing, such as the exaggerated banter common among sports fans.

To find this rare phenomenon, they hand-built a lexicon of threat verbs covering several Arabic varieties. They used it to harvest a large collection of Arabic tweets and then manually labelled a subset as dangerous or not. They also characterize threatening language on Arabic Twitter and train classifiers on the labelled data. The strongest system, BERT-Emotion fine-tuned on the gold set, clearly beats a competitive baseline.

The work matters because it supplies a first dedicated resource and benchmark for a safety-critical category. Automated detection is needed because the volume of posts makes manual moderation impractical.

Key points

  • Dangerous speech is defined as expressing an aim to cause someone bodily pain, injury or damage as retaliation. It follows Fraser's (1998) two components: the speaker intends to act, and the act is unfavorable to the addressee.
  • The verb lexicon (Table 1) tags each entry by variety: Egyptian, Gulf, Levantine, MSA and Maghrebi, or all dialects. It also marks verbs used metaphorically (e.g., "pluck" or "mark" with a body part) and idiomatically (e.g., drinking someone's blood, erasing someone from the earth).
  • The best model is BERT-Emotion fine-tuned on the gold dangerous dataset. It reaches 77.97% accuracy and 59.60% macro F1, significantly above the baseline.
  • Because the labelled data is highly imbalanced, the authors treat F1 rather than accuracy as the main metric.
  • The motivation for Arabic is heavy Twitter use in the region; for example, Saudi Arabia has a reported 37% penetration, the highest worldwide. The datasets are released for research.

Interesting ideas

  • I1. Seed rare-phenomenon collection with a curated verb lexicon, then hand-annotate only a sample, because random sampling would almost never hit a threat.
  • I2. Each lexicon entry carries separate flags for literal, metaphorical and idiomatic use, so one verb can count as a threat only in certain uses.
  • I3. A threat is broken into two checkable parts: the speaker commits to a future act, and that act is harmful to the hearer.
  • I4. Teasing in a known context (sports rivalry) is explicitly excluded, so context and intent override surface wording.
  • I5. For a heavily skewed label, macro F1 rather than accuracy is reported, since a majority-class guesser would look deceptively good.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): rare constructions like threats, promises or conditionals could be harvested with a verb-list seed before silver labelling, so witness parsers and the LLM teacher are not wasted on empty samples.
  • I2 (world): the compiled verb-class knowledge could store a literal/figurative flag per sense, so "I'll kill you" in banter does not create a harm state in the text world.
  • I3 (frames): a threat speech act could be recognized structurally in prag as a commissive frame with the addressee as patient and a harm-class verb, which keeps the decision explainable.
  • I4 (frames): the irony/implicature layer could downgrade a commissive-harm frame when context shows playful rivalry, and log the reason.
  • I5 (learn): evaluations of rare speech acts should report per-class F1 alongside a majority-class negative control, to expose models that never fire.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of British ColumbiaVancouver · Canada98 works · 34,805 citations🎓University at Buffalo, State University of New YorkBuffalo · United States12 works · 318 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksPsychology · 1,552 worksSocial Psychology · 306 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Ali Alshehri, El Moatez Billah Nagoudi and Muhammad Abdul-Mageed. 2020. Understanding and Detecting Dangerous Speech in Social Media. Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{alshehri2020understanding,
  title = {{Understanding and Detecting Dangerous Speech in Social Media}},
  author = {Ali Alshehri, El Moatez Billah Nagoudi and Muhammad Abdul-Mageed},
  booktitle = {Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection},
  year = {2020},
  url = {https://aclanthology.org/2020.osact-1.6/}
}

References

28 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdul-Mageed, M., Zhang, C., Hashemi, A., and Nagoudi, E. M. B. (2019). Aranet: A deep learning toolkit for arabic social media. arXiv preprint arXiv:1912.13072. ↗
  2. Abozinadah, A., M. A. a. J. J. (2015). Detection of abusive accounts with arabic tweets. International Journal of Knowledge Engineering, Vol. 1, No. 2. ↗
  3. Alshehri, A., Nagoudi, A., Hassan, A., and Abdul-Mageed, M. (2018). Think before your click: Data and models for adult content in arabic twitter. The 2nd Text Analytics for Cybersecurity and Online Safety (TA-COS-2018), LREC. find
  4. Burnap, P. and Williams, M. L. (2014). Hate speech, machine classification and statistical modelling of information flows on twitter: Interpretation and communication for policy decision making. find
  5. Clarke, I. and Grieve, J. (2017). Dimensions of abusive language on twitter. Association for Computational Linguistics. find
  6. Davidson, T., Warmsley, D., Macy, M., and Weber, I. (2017). Automated hate speech detection and the problem of offensive language. arXiv preprint arXiv:1703.04009. ↗
  7. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  8. Farghaly, A. and Shaalan, K. (2009). Arabic natural language processing: Challenges and solutions. ACM Transactions on Asian Language Information Processing (TALIP), 8(4):1–22. ↗
  9. Fraser, B. (1998). Threatening revisited. Forensic linguistics, 5(2). find
  10. Gales, T. (2011). Identifying interpersonal stance in threatening discourse: An appraisal analysis. Discourse Studies, 13(1):27–46. ↗
  11. Gambäck, B. and Sikdar, U. K. (2017). Using convolutional neural networks to classify hate-speech. In Proceedings of the First Workshop on Abusive Language Online, pages 85–90.
  12. Gitari, N. D., Zuping, Z., Damien, H., and Long, J. (2015). A lexicon-based approach for hate speech detection. International Journal of Multimedia and Ubiquitous Engineering, 10(4):215–230. ↗
  13. Hardaker, C. and McGlashan, M. (2016). ’real men don’t hate women’: Twitter rape threats and group identity. Journal of Pragmatics, 91:80 – 93. ↗
  14. Hinduja, S. and Patchin, J. W. (2010). Bullying, cyberbullying, and suicide. Archives of suicide research, 14(3):206–221. ↗
  15. Iqbal, M. (2019). Twitter revenue and usage statistics in 2019. November. find
  16. Jay, T. and Janschewitz, K. (2008). The pragmatics of swearing. Journal of Politeness Research. Language, Behaviour, Culture, 4(2):267–288. ↗
  17. Kwok, I. and Wang, Y. (2013). Locate the hate: Detecting tweets against blacks. In AAAI. find
  18. Landis, J. R. and Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1):159–174. ↗
  19. Lenze, N. (2017). Social media in the arab world: Communication and public opinion in the gulf states. European Journal of Communication, 32(1):77–79. find
  20. Mubarak, H., Darwish, K., and Magdy, W. (2017). Abusive language detection on arabic social media. In Proceedings of the First Workshop on Abusive Language Online, pages 52–56.
  21. Òscar Garibo, i. O. (2019). Multilingual detection of hate speech against immigrants and women in twitter at semeval-2019 task 5: Frequency analysis interpolation for hate in speech detection. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 460–463. find
  22. Salem, F. (2017). The arab social media report 2017: Social media and the internet of things: Towards data-driven policymaking in the arab world. Vol. 7. find
  23. Singh, M., Bansal, D., and Sofat, S. (2016). Behavioral analysis and classification of spammers distributing pornographic content in social media. Social Network Analysis and Mining, 6(1):41, Jun. ↗
  24. Waseem, Z. and Hovy, D. (2016). Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In Proceedings of the NAACL student research workshop, pages 88–93.
  25. Waseem, Z., Davidson, T., Warmsley, D., and Weber, I. (2017). Understanding abuse: A typology of abusive language detection subtasks. CoRR, abs/1705.09899. ↗
  26. Waseem, Z. (2016). Are you a racist or am i seeing things? annotator influence on hate speech detection on twitter. In Proceedings of the first workshop on NLP and computational social science, pages 138–142.
  27. Wiegand, M., Siegel, M., and Ruppenhofer, J. (2018). Overview of the germeval 2018 shared task on the identification of offensive language. find
  28. Zampieri, M., Malmasi, S., Nakov, P., Rosenthal, S., Farra, N., and Kumar, R. (2019). Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval). arXiv preprint arXiv:1903.08983. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alshehri-2020-understanding-detecting-dangerous-speech-social/