Elamur Research⌕
☰
home / papers / 2023 / PCJ at SemEval-2023 Task 10: A Ensemble Model Based on Pre-trained Model for Sexism Detection and Classification in English

PCJ at SemEval-2023 Task 10: A Ensemble Model Based on Pre-trained Model for Sexism Detection and Classification in English

Paper · 2023 · Chujun Pu and Xiaobing Zhou

Summary

The paper presents team PCJ's system for SemEval-2023 Task 10, subtask A. The task is a binary decision on whether an English post from Gab or Reddit is sexist. The authors fine-tune two large encoders, RoBERTa-large and sup-SimCSE-RoBERTa-large, and attach several classification heads to them:

  • a BiLSTM;
  • a BiLSTM followed by a BiGRU, with pooled outputs combined with the pooler output;
  • a head that concatenates the <CLS> vectors of the last four hidden layers.

The resulting four models are merged by majority voting. Training uses the organizers' 14,000 labelled posts, which are heavily skewed toward the non-sexist class. The authors split them 8:1:1 with stratification and clean only the training and validation parts. On the official evaluation the ensemble reached a macro F1 of 0.8449, placing 28th of 84 teams. The ensemble's development-phase score stayed close to its score on the internal split, while the single models drifted more. This suggests that voting mainly buys stability. A per-class analysis shows that the minority "sexist" class remains clearly harder.

Key points

  • Data: 14,000 training posts, of which 3,398 are sexist, drawn from Gab and Reddit. The data was split 8:1:1 with stratified sampling. Emoji and "[xxx]" tokens were removed from train and validation, but the internal test part was left raw.
  • Four models:
    • RoBERTa-BiLSTM;
    • RoBERTa-BiLSTM-BiGRU, with average and max pooling plus the pooler output;
    • RoBERTa using the last-four-layer <CLS> concatenation with max pooling;
    • SimCSE-RoBERTa-BiLSTM.
  • Training setup: BertAdam optimizer, learning rate 2e-5 for RoBERTa and 1e-5 for the other parts, dropout 0.2, pad size 64, batch size 32.
  • Macro F1 on the internal test split: plain RoBERTa 0.8327, +BiLSTM 0.8311, +BiLSTM+BiGRU 0.8467, SimCSE+BiLSTM 0.8359, ensemble 0.8480.
  • Macro F1 on development-phase data: RoBERTa 0.8164, ensemble 0.8489.
  • Per-class results: precision and recall exceed 0.9 for "not sexist" but sit around 0.70–0.80 for "sexist". The authors attribute the gap to class imbalance.

Interesting ideas

  • I1. Several differently built heads on related encoders are combined by simple majority vote, which mainly improved how well results held up on a second evaluation set rather than raising peak accuracy.
  • I2. The [CLS] vectors from the last four encoder layers are concatenated instead of using only the top layer, on the premise that each layer captures text differently.
  • I3. The authors deliberately keep their internal test split unpreprocessed, so it resembles the raw competition input and gives a more honest score.
  • I4. They treat the gap between a model's score on its own split and on an external dev set as a generalization signal, separate from the absolute score.
  • I5. Reporting precision and recall per class exposed that the macro score hid a weak minority class.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): This supports building silver data from agreement among structurally different witness parsers, and suggests measuring agreement's value by stability across held-out sets.
  • I2 (tree): gparse could feed a learned mix of upper layers into its arc and label scorers and then check which layers help attachment decisions most.
  • I3 (learn): Evaluation sets should pass through the full pipeline from raw text, so that cleanup steps applied in training do not inflate tokenizer or parser scores.
  • I4 (learn): For each trained component, we could report the drop from in-domain to out-of-domain treebank as a standard metric alongside LAS.
  • I5 (frames): Speech-act and irony detectors should be evaluated per label, because rare categories like irony are exactly where aggregate scores mislead.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Yunnan UniversityKunming · China17 works · 40 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Chujun Pu and Xiaobing Zhou. 2023. PCJ at SemEval-2023 Task 10: A Ensemble Model Based on Pre-trained Model for Sexism Detection and Classification in English. Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.semeval-1.59

@inproceedings{pu2023pcj,
  title = {{PCJ at SemEval-2023 Task 10: A Ensemble Model Based on Pre-trained Model for Sexism Detection and Classification in English}},
  author = {Chujun Pu and Xiaobing Zhou},
  booktitle = {Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023)},
  year = {2023},
  doi = {10.18653/v1/2023.semeval-1.59},
  url = {https://doi.org/10.18653/v1/2023.semeval-1.59}
}

References

19 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Basant Agarwal and Namita Mittal. Text classification using machine learning methods-a survey. In Proceedings of the Second International Conference on Soft Computing for Problem Solving (SocProS 2012), December 28-30, 2012, pages 701–709. find
  2. Shikha Bordia and Samuel R Bowman. 2019. Identifying and reducing gender bias in word-level language models. arXiv preprint arXiv:1904.03035. ↗
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  4. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821. ↗
  5. Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does bert learn about the structure of language? In ACL 2019-57th Annual Meeting of the Association for Computational Linguistics.
  6. Hannah Rose Kirk, Wenjie Yin, Bertie Vidgen, and Paul Röttger. 2023. SemEval-2023 Task 10: Explainable Detection of Online Sexism. In Proceedings of the 17th International Workshop on Semantic Evaluation, Toronto, Canada. Association for Computational Linguistics.
  7. Qian Li, Hao Peng, Jianxin Li, Congying Xia, Renyu Yang, Lichao Sun, Philip S Yu, and Lifang He. 2020. A survey on text classification: From shallow to deep learning. arXiv preprint arXiv:2008.00364. ↗
  8. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  9. Hanjie Mai, Xiaobing Zhou, and Liqing Wang. A multi-task learning model for fine-grain dialogue social bias measurement. In Natural Language Processing and Chinese Computing: 11th CCF International Conference, NLPCC 2022, Guilin, China, September 24–25, 2022, Proceedings, Part II. find
  10. SreeJagadeesh Malla and PJA Alphonse. 2021. Covid-19 outbreak: An ensemble pre-trained deep learning model for detecting informative tweets. Applied Soft Computing, 107:107495. ↗
  11. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26. ↗
  12. Shervin Minaee, Nal Kalchbrenner, Erik Cambria, Narjes Nikzad, Meysam Chenaghlu, and Jianfeng Gao. 2021. Deep learning–based text classification: a comprehensive review. ACM computing surveys (CSUR), 54(3):1–40. ↗
  13. Ji Ho Park, Jamin Shin, and Pascale Fung. 2018. Reducing gender bias in abusive language detection. arXiv preprint arXiv:1808.07231. Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. Pre-trained models for natural language processing: A survey. Science China Technological Sciences. ↗
  14. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language understanding by generative pre-training. find
  15. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  16. Bingkun Wang, Donghong Shan, Aiwan Fan, Lei Liu, and Jingli Gao. 2021. A sentiment classification method of web social media based on multidimensional and multilevel modeling. IEEE Transactions on Industrial Informatics, 18(2):1240–1249. ↗
  17. Qing Yu, Ziyin Wang, and Kaiwen Jiang. 2021. Research on text classification based on bert-bigru model. In Journal of Physics: Conference Series, volume 1746, page 012019. IOP Publishing. find
  18. Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi. 2020. Revisiting few-sample bert fine-tuning. arXiv preprint arXiv:2006.05987. ↗
  19. Yu Zhang and Qiang Yang. 2021. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering, 34(12):5586–5609. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pu-2023-pcj-semeval-2023-task-10/