📄 UMUTeam at SemEval-2023 Task 10: Fine-grained detection of sexism in English

Summary

The paper describes UMUTeam's system for EDOS (Explainable Detection of Online Sexism, SemEval-2023 Task 10). EDOS asks systems to flag sexist posts from Gab and Reddit and then explain them through a label hierarchy: a binary decision (Task A), four coarse categories (Task B) and eleven fine-grained vectors (Task C).

The team took RoBERTa-large and continued its masked-language-model pretraining on the organizers' two million unlabeled Gab and Reddit posts. They then fine-tuned this domain-adapted model separately for each subtask, using only the official labeled corpus. Before training, the text was lightly cleaned: mentions and hashtags were replaced with placeholders, emojis were dropped and slang contractions were expanded. Hyperparameters were tuned with a TPE search.

The system ranked 24th of 84 on Task A, 23rd of 69 on Task B and 13th of 63 on Task C. An ablation shows that domain adaptation always helped, and helped most on the sparse 11-way task. Error analysis traces most mistakes to rare or semantically overlapping classes. The work is a simple, reproducible baseline showing that unlabeled in-domain text pays off when labels are scarce.

Key points

• Data: The EDOS corpus has 20,000 posts, half from Gab and half from Reddit, split 14,000/2,000/4,000. About a quarter of the posts are sexist (4,854). Each post was labeled by three annotators, with expert adjudication of disagreements.
• Unlabeled data and pretraining: The organizers supplied 1M Gab and 1M Reddit unlabeled posts. MLM continuation of RoBERTa-large (355M parameters) used line-by-line sentences, a block size of 128, 3 epochs and batch size 32.
• Fine-tuning search: RayTune with TPE ran 10 trials. The chosen setting was about 15–20 epochs, batch 16, weight decay ≈0.18 and learning rate ≈1.19e-5, the same for all subtasks.
• Results (macro F1): Task A scored 0.8495, about 2.5 points behind the winner. Task B scored 0.6395 (rank 23/69 in the abstract, 24/70 in the results section). Task C scored 0.4793.
• Ablation: Compared with plain RoBERTa-large, domain adaptation raised Task A from 0.8426 to 0.8495, Task B from 0.6313 to 0.6395 and Task C from 0.4349 to 0.4793. The main confusions were between animosity, derogation and prejudiced discussions.

Interesting ideas

• I1. Before fine-tuning on a small labeled set, keep training the encoder's MLM objective on plentiful unlabeled text from the same genre; the gain grew as the label set got finer and sparser.
• I2. Report a clean before/after comparison of the same backbone with and without adaptation, so the effect of one design choice is isolated.
• I3. Read row-normalized confusion matrices to see which class pairs collapse, then explain the collapses by class frequency or by real semantic overlap.
• I4. Use a nested label scheme (binary, then 4 classes, then 11), where errors at the coarse level reappear in the fine one.
• I5. Normalize noisy social text before modeling, by mapping mentions and hashtags to placeholders and expanding slang contractions like "y'all".

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): continue pretraining the gparse encoder on raw text of our target genres (fables, school textbooks) before treebank training, especially to help rare relations.
• I2 (learn): treat "no adaptation" as a standing control run next to every adapted parser or tagger, so each claimed improvement has its own baseline.
• I3 (frames): apply the same per-pair confusion analysis to speech-act and irony labels in prag, to decide whether a failure needs more data or a sharper label definition.
• I4 (world): ontology-backed labels could be scored at each depth, so we can tell whether a mistake happens at the parent class or only among siblings.
• I5 (tokens): a normalization pass in crate en could expand such forms for tagging while keeping the original span, so every analysis still points back to the raw text.