Elamur Research⌕
☰
home / papers / 2019 / L-HSAB: A Levantine Twitter Dataset for Hate Speech and Abusive Language

L-HSAB: A Levantine Twitter Dataset for Hate Speech and Abusive Language

Paper · 2019 · Hala Mulki, Hatem Haddad, Chedi Bechikh Ali et al.

Summary

Arabic dialects have few public resources for detecting abusive language and hate speech, and Levantine dialects have almost none. This paper presents L-HSAB, which the authors describe as the first publicly available Levantine Twitter dataset for this task. They collected tweets from March 2018 to February 2019 using two sources: queries built around groups that are often targeted, and the timelines of prominent politicians, activists and TV presenters. After filtering, 5,846 Syrian and Lebanese tweets remained. Three native-speaker annotators labeled each tweet as Normal, Abusive or Hate. Their guidelines tied hate speech to identity-based demeaning of specific, pre-defined groups. Annotation quality was checked with several agreement measures, including Cohen's K and Krippendorff's α. The authors judge the dataset reliable, while noting that disagreement concentrates on the Hate class. Baseline experiments with SVM and NB on n-gram features showed NB ahead of SVM in both binary and three-way classification. The resource gives an underrepresented dialect group a benchmark for toxic content detection.

Key points

  • Size and labels: 57,058 tweets were retrieved and filtered to 6,000. After annotation, 5,846 remained: 3,650 Normal, 1,728 Abusive and 468 Hate. The domain is largely political, and the dialects are Syrian and Lebanese.
  • Annotators: three Levantine native speakers (one male, two female) holding a PhD or postdoc. The final label is the majority vote of at least two annotators.
  • Percent agreement: pairwise agreement ranged from 78.43% to 87.24%, and the S coefficient was 72.3%.
  • Cohen's K and Krippendorff's α: pairwise Cohen's K was 0.594–0.758, and Krippendorff's α was 76.5%. By the paper's own scale, this α falls in the "tentative" band of 0.67–0.8.
  • Baselines: SVM (LIBSVM) and NLTK NB were trained on unigram, bigram and trigram combinations with TF thresholds of 2 and 3. For the binary task, Hate was merged into Abusive. NB outperformed SVM in both setups, which the authors attribute to its binary presence/absence features.

Interesting ideas

  • I1. Listing target groups in advance: The guidelines name the groups that hate speech could target within this dataset's domain and time period. This makes the hate-versus-abuse boundary less subjective. The approach could transfer to any annotation task where a label depends on who is being attacked.
  • I2. Giving annotators local context: Annotators received notes on the ethnic, religious and regional affiliation of political parties. They also got a glossary of insulting nicknames for those groups. Coded or slang references are common in many languages, and such cheat sheets would help with them.
  • I3. Two-pronged data collection: Keyword queries about targeted groups were combined with harvesting replies to high-follower public figures. This boosts the share of rare toxic content and could help build any imbalanced corpus.
  • I4. Reporting several agreement measures: The authors report raw agreement, the S coefficient, pairwise Cohen's K and Krippendorff's α side by side. Pairwise scores also let them check whether annotator gender explained disagreements. Reporting this way shows where and between whom disagreement arises, not just one summary number.
  • I5. Two task granularities: Running both a merged binary task and a fine-grained three-class task shows how much difficulty the rare Hate class adds. Other benchmarks could use the same setup to show how difficulty grows with label granularity.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Manouba UniversityManouba · Tunisia3 works · 257 citations🎓University of CarthageTunis · Tunisia2 works · 257 citations🎓National School of Computer ScienceManouba · Tunisia2 works · 256 citations

Also on the byline: National Institute of Applied Science and Technology · Kırıkkale University · Konya Technical University.

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Social Sciences · 744 works Sociology and Political Science · 221 works

🧭 Topics 2 Sentiment & emotion · 608 works Bias, fairness & safety · 209 works

Cite

Cite as Hala Mulki, Hatem Haddad, Chedi Bechikh Ali and Halima Alshabani. 2019. L-HSAB: A Levantine Twitter Dataset for Hate Speech and Abusive Language. Proceedings of the Third Workshop on Abusive Language Online.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-3512

@inproceedings{mulki2019l,
  title = {{L-HSAB: A Levantine Twitter Dataset for Hate Speech and Abusive Language}},
  author = {Hala Mulki, Hatem Haddad, Chedi Bechikh Ali and Halima Alshabani},
  booktitle = {Proceedings of the Third Workshop on Abusive Language Online},
  year = {2019},
  doi = {10.18653/v1/w19-3512},
  url = {https://doi.org/10.18653/v1/w19-3512}
}

References

15 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Monirah A. Al-Ajlan and Mourad Ykhlef. 2018. Optimized twitter cyberbullying detection based on deep learning. In Proceedings of the 21st Saudi Computer Society National Computer Conference (NCC), pages 52–56. find
  2. Anna Schmidt and Michael Wiegand. 2017. A survey on hate speech detection using natural language processing. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, pages 1–10. our card
  3. Areej Al-Hassan and Hmood Al-Dossari. 2019. Detection of hate speech in social networks: A survey on multilingual corpus. Computer Science & Information Technology (CS & IT), 9(2):83–100. find
  4. Zeerak Waseem, Thomas Davidson, Dana Warmsley, and Ingmar Weber. 2017. Understanding abuse: A typology of abusive language detection subtasks. In Proceedings of the First Workshop on Abusive Language Online, pages 78–84. our card
  5. Azalden Alakrota, Liam Murray, and Nikola S.Nikolov. 2018. Dataset construction for the detection of anti-social behaviour in online communication in arabic. Procedia Computer Science, 142:174–181. find
  6. Nuha Albadi, Maram Kurdi, and Shivakant Mishra. 2018. Are they our brothers? analysis and detection of religious hate speech in the arabic twittersphere. In Proceedings of IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pages 69–76. find
  7. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596. ↗ our card
  8. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. ” O’Reilly Media, Inc.”. find
  9. Chih-Chung Chang and Chih-Jen Lin. 2011. Libsvm: A library for support vector machines. ACM transactions on intelligent systems and technology (TIST), 2(3):27. find
  10. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 1(20):37–46. ↗
  11. Ona de Gibert, Naiara Perez, Aitor Garcı́a-Pablos, and Montse Cuadros. 2018. Hate speech dataset from a white supremacy forum. In Proceedings of the 2nd Workshop on Abusive Language Online, pages 11– 20. our card
  12. Mari J Matsuda. 2018. Public response to racist speech: Considering the victim’s story. In Words that wound, pages 17–51. Routledge. ↗
  13. Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica: Biochemia medica, 22(3):276–282. ↗
  14. Hamdy Mubarak, Kareem Darwish, and Walid Magdy. 2017. Abusive language detection on arabic social media. In Proceedings of the First Workshop on Abusive Language Online, pages 52–56. our card
  15. John T. Nockleby. 2000. Hate Speech, volume 1. Encyclopedia of the American Constitution (2nd ed., edited by Leonard W. Levy, Kenneth L. Karst et al. New York: Macmillan, New York: Macmillan. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mulki-2019-l-hsab-levantine-twitter-dataset/