📄 Team Curie at HSD-2Lang 2024: Hate Speech Detection in Turkish and Arabic Tweets using BERT-based models

Summary

The paper describes Team Curie's entry to the HSD-2Lang shared task at CASE 2024. The task is binary hate speech classification of tweets in two settings: Turkish tweets grouped by topic (Subtask A) and Arabic tweets with only a small training set (Subtask B). The team fine-tuned two monolingual BERT models, both trained with AdamW: dbmdz/bert-base-turkish-128k-uncased for Turkish and asafaya/bert-base-arabic for Arabic.

For Turkish, they built a cleaning pipeline. It turns emojis into English descriptions and strips URLs, mentions, hashtags and extra whitespace. They also drew the validation set by stratifying over label and topic. For Arabic, cleaning lowered scores, so the tweets were left nearly raw.

The best public-leaderboard F1 was 0.719 for Turkish (fourth place) and 0.769 for Arabic (third place). The Turkish peak came from a submission made after the deadline. An ablation on Turkish shows which cleaning steps carry the gain and that resampling for balance did not help. The work offers a compact, reproducible baseline for classifying social-media text in lower-resource languages.

Key points

• Shared training setup: AdamW with learning rate 5×10⁻⁵, weight decay 0.01 and batch size 128, for 3 epochs (Turkish) and 5 epochs (Arabic). Turkish inputs were capped at 128 tokens.
• Turkish validation set: it was stratified per label and topic, with 70, 60 and 50 tweets per label for the topics Anti-Refugee, Israel-Palestine and Turkey-Greece.
• Subtask A F1 (public / private):
  • Late "peak" submission: 0.71889 / 0.66129.
  • Official submission: 0.71365 / 0.60790.
  • Competition best: 0.74876 / 0.69644.
• Subtask B F1 (public / private):
  • Team result: 0.76923 / 0.65853, the same for the official and peak submissions.
  • Competition best: 0.88888 / 0.68354.
• Turkish ablation:
  • Skipping hashtag processing caused the largest drop (0.67868 / 0.62391).
  • Skipping emoji conversion also hurt (0.69868 / 0.64391).
  • Newline and space cleanup made no difference.
  • Label balancing (0.70646 / 0.64332) and topic balancing (0.63917 / 0.60550) both scored below the unbalanced run.

Interesting ideas

• I1. Removing one preprocessing step at a time and re-scoring shows which normalizations matter and which are inert, such as whitespace cleanup here.
• I2. Emojis were not deleted but rewritten as textual descriptions, so a model trained on words can still read their meaning.
• I3. Cleaning that helped Turkish hurt Arabic, so preprocessing was treated as a choice to verify per language and data type, not a fixed default.
• I4. The validation split was stratified over label × topic combinations, so held-out data mirrored the thematic mix of training.
• I5. Official, post-deadline, public and private scores were reported separately, which exposes how much tuning against a public leaderboard inflates results.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Run the same leave-one-out test on each text-normalization rule before the tagger, so rules that add nothing can be dropped with evidence.
• I2 (tokens): The tokenizer could map each emoji to a description lemma with a part-of-speech, giving it a node in the UD tree that prag can later use for sentiment or irony cues.
• I3 (tokens): Keep normalization profiles separate per text source (fables, word problems, web text) and validate each on its own data.
• I4 (learn): Build evaluation sets stratified by genre × linguistic phenomenon, so a gain on one frequent pattern cannot hide a loss on a rare one.
• I5 (learn): Keep a sealed test split that is never used during repair-rule induction, and report its score next to the dev score as an overfitting check.