📄 ReBERT at HSD-2Lang 2024: Fine-Tuning BERT with AdamW for Hate Speech Detection in Arabic and Turkish

Summary

The authors present their entry to the HSD-2Lang 2024 shared task, which asks for binary hate speech classification of tweets in Turkish (Subtask A) and Arabic (Subtask B). They used only the organisers' data: 9140 labelled tweets for Turkish and 960 for Arabic. They first compared several Hugging Face checkpoints without any fine-tuning and kept the strongest ones, TurkishBERTweet and Arabertv02-twitter. Both were then fine-tuned with AdamW, and the team searched learning rate, batch size, epoch count and scheduler type. Input text was left almost untouched, and the Arabic training set was rebalanced by randomly removing part of its majority class. The Arabic system placed first on the private leaderboard with F1 0.74. The Turkish system placed second with F1 0.69, and both led the public leaderboard. The paper shows that careful hyperparameter tuning of domain-matched pretrained encoders is still a strong baseline. It also shows that model selection against a small public test slice can mislead, especially when training data is scarce and imbalanced.

Key points

• Data: Subtask A has 9140 training tweets with roughly 70/30 non-hate/hate labels. Subtask B has 960 tweets with roughly 90/10. For Subtask B, the team randomly kept only 80% of the negative examples.
• Turkish setup: TurkishBERTweet with its own preprocessing, which only turns URLs and emoticons into tags. Training ran on an A100 with batch 32, max length 256, learning rate 5e-5, epsilon 1e-8, and polynomial decay with 10% warmup.
• Unusually long training helped the Turkish model. Runs of 100 and 1000 epochs beat the usual 2–4 epochs by up to about 4%, and warmup also helped. An Adafactor run did clearly worse than the AdamW runs.
• Arabic setup: Arabertv02-twitter with no preprocessing, trained on a GTX 1070 with batch 8, max length 250, and linear decay without warmup. The best submitted run scored 0.89 public and 0.74 private F1.
• Tuning was guided only by the public test portion, which is 20% of the test data. In Table 1, one Arabic run with a lower public score (0.85) reached a higher private score (0.76) than the submitted model.

Interesting ideas

• I1. Before investing in fine-tuning, the authors ran each candidate pretrained checkpoint as-is on the training data and dropped the weaker ones, a cheap screen that saves compute.
• I2. Rather than aggressive balancing, they removed only a modest random share of majority-class examples, because the dataset was too small to lose much.
• I3. Preprocessing was kept minimal and reversible: URLs and emoticons were replaced by tags while punctuation and tone markers were kept.
• I4. The gap between public and private scores shows that choosing a model by a small visible test slice can pick the wrong configuration.
• I5. Choosing encoders pretrained on the same genre as the target text (tweets) mattered more than generic models.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): Candidate encoders for gparse could be screened by a quick frozen-feature or few-step probe on a UD treebank before committing to full training runs.
• I2 (learn): When building silver data where some labels dominate (e.g. frequent deprels or frame types), light undersampling of the dominant class could reduce bias without starving rare cases.
• I3 (tokens): The tokenizer in crate en could map URLs, emojis and handles to typed placeholder tokens, so the tagger sees a stable category while the original span stays recoverable for explanation.
• I4 (learn): Evaluation should keep a separate untouched held-out set and report variance across runs, so that repair rules or training choices are not overfitted to one dev slice.
• I5 (tree): Genre-matched pretraining or treebanks (e.g. web/social UD treebanks) could be used when gparse must handle informal text, instead of relying on a single news-trained model.