Elamur Research⌕
☰
home / papers / 2024 / Transformers at HSD-2Lang 2024: Hate Speech Detection in Arabic and Turkish Tweets Using BERT Based Architectures

Transformers at HSD-2Lang 2024: Hate Speech Detection in Arabic and Turkish Tweets Using BERT Based Architectures

Paper · 2024 · Kriti Singhal and Jatin Bedi

Summary

This shared-task paper from team Transformers addresses binary hate speech detection in tweets for two languages whose orthography and rich morphology make them harder than English: Turkish (HSD-2Lang 2024 Subtask A, covering several topical contexts) and Arabic (Subtask B, with very little labelled data). The authors fine-tuned off-the-shelf multilingual encoders: XLM RoBERTa Large for Turkish and mBERT for Arabic. They relied on cross-lingual transfer to make up for scarce in-language resources. Class imbalance was handled by randomly discarding majority-class tweets until both labels were equally frequent. Raw, unpreprocessed text gave better results than cleaned text, so no normalisation was applied. The systems reached F1 0.63258 in Subtask A (6th place) and 0.48101 in Subtask B (5th place). The winning teams scored 0.69644 and 0.68354. The work matters mainly as a simple, reproducible baseline. It shows how far plain multilingual fine-tuning goes, and where it falls short, especially with tiny Arabic training sets.

Key points

  • Subtask A data: 9,140 Turkish tweets from three context subsets (anti-refugee sentiment, Israel-Palestine conflict, anti-Greek discourse). The subsets were merged into one pool, with non-hateful tweets outnumbering hateful ones about 3 to 1.
  • Subtask B data: described as 1,000 Arabic tweets, but the reported counts are 778 non-hateful and 82 hateful, which do not add up. After undersampling, only 82 + 82 tweets remained.
  • Models and training: XLM RoBERTa Large was trained for 8 epochs with weighted Adam, NLL loss and an 80/20 split. mBERT was trained for 13 epochs with Adam, NLL loss and a 70/30 split.
  • Results: F1 0.63258 (rank 6) for Turkish and 0.48101 (rank 5) for Arabic. The gap to the top teams was about 0.06 and 0.20 respectively.
  • Future work proposed by the authors: ensembling several transformers and pooling multiple existing datasets.

Interesting ideas

  • I1. The authors checked whether conventional cleaning helps, and found that untouched tweets (emojis, links, symbols) trained a better transformer than cleaned ones, so preprocessing should be tested empirically rather than assumed.
  • I2. Pooling topically different subsets (refugees, Middle East conflict, Greek discourse) into one training set gives a single classifier that covers several contexts at once.
  • I3. Random undersampling to an exact 50/50 class ratio is a crude but transparent fix for imbalance, and the tiny Arabic set shows its cost: discarding most of the data hurts.
  • I4. Related work cited in the paper converts emojis and emoticons into textual labels instead of deleting them, so the meaning they carry is preserved.
  • I5. Reporting one's own F1 next to the shared-task winner's score makes the true gap visible, here roughly 0.06 versus 0.20.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Keep emojis, hashtags and URLs as typed tokens instead of stripping them, and evaluate any normalisation step against the raw-input baseline before adopting it.
  • I2 (learn): When building silver data, merge domain slices but keep the slice tag, so per-domain error rates can still be inspected after joint training.
  • I3 (learn): Any rebalancing of rare phenomena in training data should log how much was thrown away, and it should be compared against reweighting with a negative control.
  • I4 (frames): Map emojis to explicit sentiment or speech-act cues that the pragmatics layer can use for irony and hostility detection, rather than ignoring them.
  • I5 (answer): Always report test scores alongside a strong external reference, so improvements are judged against the real ceiling rather than in isolation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Thapar Institute of Engineering & TechnologyPatiāla · India6 works · 11 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 4Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 worksMultilingual & cross-lingual NLP · 828 worksBias, fairness & safety · 517 works

Cite

Cite as Kriti Singhal and Jatin Bedi. 2024. Transformers at HSD-2Lang 2024: Hate Speech Detection in Arabic and Turkish Tweets Using BERT Based Architectures. Proceedings of the 7th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2024).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.case-1.26

@inproceedings{singhal2024transformers,
  title = {{Transformers at HSD-2Lang 2024: Hate Speech Detection in Arabic and Turkish Tweets Using BERT Based Architectures}},
  author = {Kriti Singhal and Jatin Bedi},
  booktitle = {Proceedings of the 7th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2024)},
  year = {2024},
  doi = {10.18653/v1/2024.case-1.26},
  url = {https://doi.org/10.18653/v1/2024.case-1.26}
}

References

13 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abeer Abuzayed and Tamer Elsayed. 2020. "Quick and Simple Approach for Detecting Hate Speech in Arabic Tweets". In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 109–114, Marseille, France. European Language Resource Association.
  2. Emrah Budur, Rıza Özçelik, Tunga Gungor, and Christopher Potts. 2020. "Data and Representation for Turkish Natural Language Inference". In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8253– 8267, Online. Association for Computational Linguistics. find
  3. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555. ↗
  4. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. ↗
  5. İnanç Arın Elif Erol Berrin Yanikoglu Arzucan Özgür Gökçe Uludoğan, Somaiyeh Dehghan. 2024. Overview of the Hate Speech Detection in Turkish and Arabic Tweets (HSD-2Lang) Shared Task at CASE 2024. In Proceedings of the 7th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE), Malta. Association for Computational Linguistics.
  6. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735– 1780. ↗
  7. Fatemah Husain. 2020. "OSACT4 Shared Task on Offensive Language Detection: Intensive Preprocessing-Based Approach". In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 53–60, Marseille, France. European Language Resource Association. ↗
  8. Zehra Melce Hüsünbeyi, Didar Akar, and Arzucan Özgür. 2022. "Identifying Hate Speech Using Neural Networks and Discourse Analysis Techniques". In Proceedings of the First Workshop on Language Technology and Resources for a Fair, Inclusive, and Safe Society within the 13th Language Resources and Evaluation Conference, pages 32–41, Marseille, France. European Language Resources Association. find
  9. Shervin Malmasi and Mark Dras. 2014. "Arabic Native Language Identification". In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 180–186, Doha, Qatar. Association for Computational Linguistics.
  10. Karin C. Ryding. 2013. Teaching Arabic in the United States. In Kassem M Wahba, Zeinab A Taha, and Liz England, editors, Handbook for Arabic Language Teaching Professionals in the 21st Century. Routledge. find
  11. Kogilavani Shanmugavadivel, Malliga Subramanian, Prasanna Kumar Kumaresan, Bharathi Raja Chakravarthi, B Bharathi, Subalalitha Chinnaudayar Navaneethakrishnan, Lavanya Sambath Kumar, Thomas Mandl, Rahul Ponnusamy, Vasanth Palanikumar, et al. 2022. Overview of the Shared Task on Sentiment Analysis and Homophobia Detection of YouTube Comments in Code-Mixed Dravidian Languages. find
  12. Statista. 2022. Number of active Twitter users in selected countries. Accessed: January 24, 2024. find
  13. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/singhal-2024-transformers-hsd-2lang-2024-hate/