📄 GUCT at Arabic Hate Speech 2022: Towards a Better Isotropy for Hatespeech Detection

Summary

The authors took part in sub-task A of the OSACT 2022 shared task on Fine-grained Hate Speech Detection, a binary decision on whether a dialectal Arabic tweet is offensive. Tweets are short and give a language model little context, so the authors try to improve the input representations rather than the classifier. They first show that MARBERT's contextual embedding space is highly anisotropic: vectors crowd into a narrow cone, and rare words drift away from the origin. Its isotropy score is about 2.88e-08. They then apply the cluster-based post-processing of Rajaee and Pilehvar: K-means the vectors, centre each cluster, and project out a few dominant directions per cluster. This raises isotropy to roughly 0.97. Both raw and processed MARBERT vectors are fed to a biLSTM classifier. The isotropic version reaches 81.2% accuracy and 79.1% macro-F1 on the development set, against 77.2% F1 for raw vectors. On the test set it scores 76.5% accuracy and about 74.2% F1. The work suggests that cheap geometric post-processing of frozen embeddings can help downstream classification.

Key points

• Data: about 8.5K training and 1.2K development tweets from the OSACT5 dataset (Mubarak et al., 2022). Mentions, URLs, diacritics, elongations and non-Arabic characters were removed.
• Isotropy measurement: isotropy is computed as the ratio of the minimum to the maximum of a partition function taken over the eigenvectors of the embedding matrix. MARBERT scores 2.88e-08. After processing, scores are 0.9564 with k=1, 0.9743 with k=10 and 0.9767 with k=20.
• Setup: 10 clusters, with 8 directions removed per cluster (embedding dimension divided by 100). The biLSTM has two hidden layers of size 102 and is trained with AdamW at a learning rate of 3e-5 for 10 epochs. The best checkpoint is kept by macro-F1.
• Development results: macro-F1 was 79.1% with isotropic vectors and 77.2% with raw vectors. The confusion matrix shows 766 of 898 non-offensive and 271 of 368 offensive tweets classified correctly.
• Test results: 76.5% accuracy. The paper reports F1 as 74.2% in some places and 74.5% in others.
• Ablations: k=2, 10 and 20 give nearly identical scores. Removing 16 or 30 directions also changes little, which the authors attribute either to losing useful semantics or to the scarcity of high-frequency words in tweets.

Interesting ideas

• I1. Before trusting a pretrained encoder, the authors compute a single isotropy number for its embedding space, giving a cheap diagnostic of representation degeneration.
• I2. Instead of fine-tuning, they post-process frozen vectors by clustering them and removing each cluster's top principal directions, a training-free fix that restores spread.
• I3. The authors note that an anisotropic space inflates cosine similarity between unrelated items, so raw similarity scores are misleading.
• I4. They plot the embedding space with points coloured by word frequency, which visually exposes rare words clustering away from the origin.
• I5. Their ablations sweep the number of clusters and directions and find a flat plateau, showing the gain is not tied to one fragile setting.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): gparse could log the isotropy of its transformer token vectors per layer, so that a degenerate encoder is flagged before parser training starts.
• I2 (tree): this could be tried as a lightweight feature transform on gparse's encoder outputs before the arc and label scorers, with the change in LAS measured.
• I3 (world): any embedding-based matching of mentions to WordNet/Wikidata ontology entries should centre the vectors and remove dominant directions first, so similarity reflects meaning rather than a shared offset.
• I4 (learn): frequency-coloured projections of silver-data token vectors could reveal whether rare lemmas are poorly represented and need extra teacher-labelled examples.
• I5 (learn): Mova's evaluation could routinely include such hyperparameter sweeps alongside negative controls, so that a reported improvement holds across a range of settings.