Elamur Research⌕
☰
home / papers / 2024 / Overview of the Hate Speech Detection in Turkish and Arabic Tweets (HSD-2Lang) Shared Task at CASE 2024

Overview of the Hate Speech Detection in Turkish and Arabic Tweets (HSD-2Lang) Shared Task at CASE 2024

Paper · 2024 · Gökçe Uludoğan, Somaiyeh Dehghan, Inanc Arin et al.

Summary

This overview describes HSD-2Lang, a shared task at the CASE 2024 workshop (co-located with EACL 2024) that benchmarks the detection of hateful tweets in two languages. Each tweet is classified as hateful or not. Subtask A uses Turkish tweets drawn from three topics: refugees, the Israel–Palestine conflict, and anti-Greek discourse. Subtask B covers anti-refugee Arabic tweets, with a small and heavily imbalanced training set. Systems were ranked by F1 on a hidden private portion of the test data, while a smaller public portion served as feedback during the test phase.

Every submission fine-tuned a BERT-family encoder. In the Turkish track, the winner combined a soft-voting ensemble of ConvBERTurk models with richer pooling and data translated from Arabic. In the Arabic track, the winner fully fine-tuned the tweet-pretrained AraBERTv0.2. The organizers observe that teams using the same backbone often finished far apart, which points to hyperparameter choices as a decisive factor, especially when data is scarce.

Key points

  • Subtask A data: 9,140 training and 2,295 test Turkish tweets across three topics. Subtask B data: 860 training tweets (only 82 hateful) and 522 test tweets (52 hateful).
  • Participation: 33 teams registered and 10 submitted for Subtask A. For Subtask B, 15 teams registered and 5 submitted.
  • The test sets were split at random, with 20% public for validation during the test phase and 80% private for the final ranking.
  • Subtask A winner DetectiveReDASers reached a private F1 of 0.69645 with ConvBERTurk. ReBERT and VRLLab placed second and third, both using TurkishBERTweet with LoRA.
  • Subtask B winner ReBERT reached a private F1 of 0.68354 with AraBERTv0.2. Team Curie (bert-base-arabic, raw tweets) came second and Team Uriel (AraBERTv0.2 with an extra 100-dimensional hidden layer) came third.

Interesting ideas

  • I1. The Turkish winner built the sentence vector by joining the [CLS] embedding with mean-pooled and max-pooled token states from the last layer, which gives the classifier both a summary view and signals from salient tokens.
  • I2. Several teams replaced emojis with textual descriptions before training, so these symbols receive real lexical content instead of becoming unknown tokens.
  • I3. The winning ensemble averaged probabilities from five copies of one architecture that differed only in random initialization, a cheap way to reduce variance from the seed.
  • I4. Cross-lingual augmentation helped the larger Turkish set but hurt the small Arabic set, which suggests auxiliary data can drown out a tiny target set.
  • I5. Keeping most of the test data hidden until the end, with only a small public slice for feedback, discourages tuning against the evaluation set; public and private rankings did in fact diverge here.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): gparse's sentence- or span-level heads (e.g. clause typing) could concatenate [CLS], mean and max states instead of relying on one pooled vector.
  • I2 (tokens): the en tokenizer could map emojis to descriptive lemmas with a POS tag, letting downstream frames and irony detection treat them as meaningful words.
  • I3 (learn): seed-only variants of a parser could serve as extra witnesses for silver data, and their disagreement could flag unstable sentences for the LLM teacher.
  • I4 (learn): when mixing converted treebanks or translated silver data into a small target domain, cap the auxiliary share and confirm the effect against a no-augmentation control.
  • I5 (answer): school-test and fable-QA benchmarks could keep a sealed private split, so self-induced repair rules are not quietly fitted to the visible questions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Sabancı ÜniversitesiIstanbul · Türkiye5 works · 228 citations🎓Boğaziçi UniversityIstanbul · Türkiye6 works · 400 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 4Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 worksEvaluation & benchmarks · 1,220 worksBias, fairness & safety · 517 works

Cite

Cite as Gökçe Uludoğan, Somaiyeh Dehghan, Inanc Arin, Elif Erol, Berrin Yanikoglu and Arzucan Özgür. 2024. Overview of the Hate Speech Detection in Turkish and Arabic Tweets (HSD-2Lang) Shared Task at CASE 2024. Proceedings of the 7th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2024).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.case-1.32

@inproceedings{uludogan2024overview,
  title = {{Overview of the Hate Speech Detection in Turkish and Arabic Tweets (HSD-2Lang) Shared Task at CASE 2024}},
  author = {Gökçe Uludoğan, Somaiyeh Dehghan, Inanc Arin, Elif Erol, Berrin Yanikoglu and Arzucan Özgür},
  booktitle = {Proceedings of the 7th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2024)},
  year = {2024},
  doi = {10.18653/v1/2024.case-1.32},
  url = {https://doi.org/10.18653/v1/2024.case-1.32}
}

References

14 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11–16 May 2020, page 9. ↗
  2. İnanç Arın, Zeynep Işık, Seçilay Kutal, Somaiyeh Dehghan, Arzucan Özgür, and Berrin Yanikoğlu. 2023. SIU2023-NST- Hate speech detection contest. In 2023 31st Signal Processing and Communications Applications Conference (SIU), pages 1–4. IEEE. find
  3. Ehsan Barkhordar, Işık S. Topçu, and Ali Hürriyetoğlu. 2024. Team Curie at HSD-2Lang 2024: Hate speech detection in Turkish and Arabic tweets using BERT-based models. In Proceedings of The Seventh Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE), Malta. Association for Computational Linguistics.
  4. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. CoRR, abs/1911.02116. ↗
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  6. Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. KUISAIL at SemEval-2020 task 12: BERT-CNN for offensive speech identification in social media. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2054–2059, Barcelona (online). International Committee for Computational Linguistics.
  7. Stefan Schweter. 2020. BERTurk - BERT models for Turkish. https://zenodo.org/records/ 3770924. ↗
  8. Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2019. Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7). Leibniz-Institut für Deutsche Sprache. find
  9. Utku Ugur Yagci, Ahmet Emirhan Kolcak, and Egemen Iscan. 2024. ReBERT at HSD-2Lang 2024: Fine-tuning BERT with AdamW for hate speech detection in Arabic and Turkish. In Proceedings of The Seventh Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE), Malta. Association for Computational Linguistics.
  10. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. LoRA: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. ↗
  11. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. ↗
  12. Ali Najafi and Onur Varol. 2023. TurkishBERTweet: Fast and reliable large language model for social media analysis. arXiv preprint arXiv:2311.18063. ↗
  13. Ali Najafi and Onur Varol. 2024. VRLLab at HSD-2Lang 2024: Turkish hate speech detection online with TurkishBERTweet. In Proceedings of The Seventh Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE), Malta. Association for Computational Linguistics. ↗
  14. Fatima Zahra Qachfar, Bryan E. Tuck, and Rakesh M. Verma. 2024. DetectiveReDASers at HSD-2Lang 2024: A new pooling strategy with cross-lingual augmentation and ensembling for hate speech detection in low-resource languages. In Proceedings of The Seventh Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE), Malta. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/uludogan-2024-overview-hate-speech-detection-turkish/