Elamur Research⌕
☰
home / papers / 2022 / SemEval-2022 Task 5: Multimedia Automatic Misogyny Identification

SemEval-2022 Task 5: Multimedia Automatic Misogyny Identification

Summary

This paper presents the MAMI shared task at SemEval-2022. MAMI asked systems to detect misogyny in internet memes, where the hateful meaning often comes from the combination of image and overlaid text and is frequently implicit. The organisers built a corpus of about 15k memes from Twitter, Reddit, 9GaG, Knowyourmeme, Imgur and anti-feminist communities, and released 10k for training and 1k for testing. Meme text was extracted with Google Cloud Vision. Three crowd annotators labelled each meme.

The task had two parts:

  • Sub-task A: a binary misogynous / not misogynous decision.
  • Sub-task B: multi-label typing into shaming, stereotype, objectification and violence.

The task attracted over 400 participants. Most teams fine-tuned pre-trained models such as RoBERTa and VisualBERT, and about half of those used ensembles. The best scores were 0.834 macro-F1 (A) and 0.731 weighted F1 (B). A cross-team error analysis showed recurring failures: frequent false positives on non-misogynous memes, which hints at model bias, and missed cases where only one modality carries the violent meaning. The benchmark matters because it shifts abuse detection from explicit text to multimodal, implicit content.

Key points

  • Data:
    • Balanced splits: 5000/5000 memes for training and 500/500 for testing.
    • The type labels overlap. Stereotype (56.2% of training) and objectification (44.0%) are the most common.
  • Inter-annotator agreement (Fleiss-κ):
    • 0.5767 for the binary label (moderate).
    • 0.3373 for the type labels (fair), computed with a MASI (Jaccard) distance for multi-label sets.
  • Participation:
    • 65 teams in Sub-task A and 41 in Sub-task B, from 13 countries.
    • 4214 runs in total, of which 166 were posted to the leaderboard.
    • India contributed the most teams (12).
  • Results:
    • Sub-task A ranged from 0.481 to 0.834.
    • Sub-task B ranged from 0.467 to 0.731.
    • Medians were nearly identical (0.679 vs 0.680). B is therefore harder only at the top of the ranking.
  • Baselines:
    • USE for text and VGG-16 for images.
    • A concatenated image+text model.
    • Flat and hierarchical multi-label models for Sub-task B.

Interesting ideas

  • I1. Grade test items by how many teams fail them. Errors are counted at thresholds of at least 25%, 50% and 75% of teams. This separates items that are hard for everyone from quirks of a single system. It is a cheap way to find systematically difficult examples in any shared task or model comparison.
  • I2. Use a set-aware kappa for overlapping labels. Combining Fleiss-κ with the MASI/Jaccard distance gives partial credit when annotators' label sets only partly overlap. Ordinary kappa would treat such cases as full disagreement, so this suits any multi-label annotation project.
  • I3. Pair the baselines to answer specific questions. Text-only, image-only and fused baselines show what each modality contributes. Flat and hierarchical multi-label variants test whether to predict "is it X" before "which kind of X". This design transfers well to any task with a parent label and subtypes.
  • I4. Collect positives from targeted sources. The organisers searched communities and events likely to contain the target phenomenon, rather than sampling randomly. This yields enough positive examples for a balanced benchmark of a rare phenomenon.
  • I5. Document contested label decisions explicitly. The organisers kept the annotators' majority view on benevolent sexism. They then reported that models seemed to flag it anyway. Recording such choices helps interpret "errors" that may actually reflect ambiguity in the label definition.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.semeval-1.74

@inproceedings{fersini2022semeval,
  title = {{SemEval-2022 Task 5: Multimedia Automatic Misogyny Identification}},
  author = {Fersini, Elisabetta and Gasparini, Francesca and Rizzi, Giulia and Saibene, Aurora and Chulvi, Berta and Rosso, Paolo and Lees, Alyssa and Sorensen, Jeffrey},
  booktitle = {Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)},
  year = {2022},
  doi = {10.18653/v1/2022.semeval-1.74},
  url = {https://doi.org/10.18653/v1/2022.semeval-1.74}
}

References

74 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Samyak Agrawal and Radhika Mamidi. 2022. Lastresort at semeval-2022 task 5: Towards misogyny identification using visual linguistic model ensembles and task-specific pretraining. In The 16th International Workshop on Semantic Evaluation. find
  2. Maja Brandt Andreasen. 2021. ‘rapeable’and ‘unra-peable’women: the portrayal of sexual violence in internet memes about\# metoo. Journal of Gender Studies, 30(1):102–113. find
  3. Maria Anzovino, Elisabetta Fersini, and Paolo Rosso. 2018. Automatic identification and classification of misogynistic language on twitter. In International Conference on Applications of Natural Language to Information Systems, pages 57–64. Springer. find
  4. Ayme Arango, Jesus Perez-Martin, and Arniel Labrada. 2022. Hateu at semeval-2022 task 5: Multimedia automatic misogyny identification. In The 16th International Workshop on Semantic Evaluation. find
  5. Giuseppe Attanasio, Debora Nozza, and Federico Bianchi. 2022. MilaNLP at semeval-2022 task 5: Using perceiver IO for detecting misogynous memes with text and image modalities. In The 16th International Workshop on Semantic Evaluation. find
  6. Shubham Kumar Barnwal, Ritesh Kumar, and Rajendra Pamula. 2022. IIT DHANBAD CODECHAMPS at semeval-2022 task 5: MAMI - multimedia automatic misogyny identification. In The 16th International Workshop on Semantic Evaluation. find
  7. Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in Twitter. In Proceedings of the 13th International Wor… find our card
  8. Mitra Behzadi, Ali Derakhshan, and Ian Harris. 2022. Mitra behzadi at semeval-2022 task 5 : Multimedia automatic misogyny identification method based on CLIP. In The 16th International Workshop on Semantic Evaluation. find
  9. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175. find our card
  10. Lei Chen and Hou Wei Chou. 2022. RIT boston at semeval-2022 task 5: Multimedia misogyny detection by using coherent visual and language features from CLIP model and data-centric AI principle. In The 16th International Workshop on Semantic Evaluation. find
  11. Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785– 794. find
  12. Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Uniter: Universal image-text representation learning. In European conference on computer vision, pages 104–120. Springer. find our card
  13. Pablo Cordon, Pablo Gonzalez Diaz, Jacinto Mata, and Victoria Pachón. 2022. I2c at semeval-2022 task 5: Identification of misogyny in internet memes. In The 16th International Workshop on Semantic Evaluation. find
  14. Statista Research Department. 2019. Share of u.s. adults who use social media 2019, by gender. find
  15. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171– 4186. find
  16. Alice H Eagly and Antonio Mladinic. 1989. Gender stereotypes and attitudes toward women and men. Personality and social psychology bulletin, 15(4):543–558. find
  17. Paolo Rosso Elisabetta Fersini, Debora Nozza. 2020. Ami @ evalita2020: Automatic misogyny identification. In Proceedings of the 7th evaluation campaign of Natural Language Processing and Speech tools for Italian (EVALITA 2020). CEUR.org. find
  18. Guy Emerson, Natalie Schluter, Gabriel Stanovsky, Ritesh Kumar, Alexis Palmer, and Nathan Schneider. 2022. Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022). Association for Computational Linguistics. find
  19. Tracie Farrell, Miriam Fernandez, Jakub Novotny, and Harith Alani. 2019. Exploring misogyny across the manosphere in reddit. In Proceedings of the 10th ACM Conference on Web Science, pages 87–96. find
  20. Elisabetta Fersini, Debora Nozza, and Paolo Rosso. 2018a. Overview of the evalita 2018 task on automatic misogyny identification (ami). In Proceedings of the 6th evaluation campaign of Natural Language Processing and Speech tools for Italian (EVALITA’18), Turin, Italy. CEUR.org. find our card
  21. Elisabetta Fersini, Paolo Rosso, and Maria Anzovino. 2018b. Overview of the task on automatic misogyny identification at ibereval 2018. In IberEval@ SEPLN, pages 214–228. find
  22. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. find
  23. Mary Anne Franks. 2011. Unwilling avatars: Idealism and discrimination in cyberspace. Colum. J. Gender & L., 20:224. find
  24. Simona Frenda, Bilal Ghanem, Manuel Montes-y Gómez, and Paolo Rosso. 2019. Online hate speech against women: Automatic identification of misogyny and sexism on twitter. Journal of Intelligent & Fuzzy Systems, 36(5):4743–4752. find
  25. José Antonio García-Díaz, Camilo Caparros-Laiz, and Rafael Valencia-García. 2022. UMUTeam at semeval-2022 task 5: Combining image and textual embeddings for multi-modal automatic misogyny identification. In The 16th International Workshop on Semantic Evaluation. find
  26. Francesca Gasparini, Ilaria Erba, Elisabetta Fersini, and Silvia Corchs. 2018. Multimodal classification of sexist advertisements. In ICETE (1), pages 565–572. find
  27. Peter Glick and Susan T Fiske. 1996. The ambivalent sexism inventory: Differentiating hostile and benevolent sexism. Journal of Personality and Social Psychology, 70(3):491–512. find
  28. Mohammad Habash, yahya Daqour, Malak AG Abdullah, and Mahmoud Al-Ayyoub. 2022. YMAI at semeval-2022 task 5: Detecting misogyny in memes using visualBERT and MMBT multimodal pre-trained models. In The 16th International Workshop on Semantic Evaluation. find
  29. Sherzod Hakimov, Gullal Singh Cheema, and Ralph Ewerth. 2022. TIB-VA at semeval-2022 task 5: A multimodal architecture for the detection and classification of misogynous memes. In The 16th International Workshop on Semantic Evaluation. find
  30. Chao Han, Jin Wang, and Xuejie Zhang. 2022. YNU-HPCC at semeval-2022 task 5: Multi-modal and multi-label emotion classification based on LXMERT. In The 16th International Workshop on Semantic Evaluation. find
  31. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735– 1780. find
  32. Álvaro Huertas-García, Helena Liz, Guillermo Villar-Rodríguez, Alejandro Martín, Javier Huertas-Tato, and David Camacho. 2022. AIDA-UPM at semeval-2022 task 5: Exploring multimodal late information fusion for multimedia automatic misogyny identification. In The 16th International Workshop on Semanti… find
  33. Milan Kalkenings and Thomas Mandl. 2022. University of hildesheim at semeval-2022 task 5: Combining deep text and image models for multimedia misogyny detection. In The 16th International Workshop on Semantic Evaluation. find
  34. Miroslav Kubat. 1999. Neural networks: a comprehensive foundation by simon haykin, macmillan, 1994, isbn 0-02-352781-7. The Knowledge Engineering Review, 13(4):409–412. find
  35. Da Li, Ming Yi, and Yukai He. 2022. AMS_ADRN at semeval-2022 task 5: A suitable image-text multimodal joint modeling method for multi-task misogyny identification. In The 16th International Workshop on Semantic Evaluation. find
  36. Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2020a. What does BERT with vision look at? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5265–5275, Online. Association for Computational Linguistics. find
  37. Qin Gu, Nino Meisinger, and Anna-Katharina Dick. 2022a. Qinian at semeval-2022 task 5: Multi-modal misogyny detection and classification. In The 16th International Workshop on Semantic Evaluation. find
  38. Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020b. Oscar: Object-semantics aligned pre-training for vision-language tasks. In European Conference on Computer Vision, pages 121–137. Springer. find our card
  39. Yimeng Gu, Ignacio Castro, and Gareth Tyson. 2022b. MMVAE at semeval-2022 task 5: A multi-modal multi-task VAE on misogynous meme detection. In The 16th International Workshop on Semantic Evaluation. find
  40. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  41. Gustavo Lorentz and Viviane Moreira. 2022. INF-UFRGS at semeval-2022 task 5: analyzing the performance of multimodal models. In The 16th International Workshop on Semantic Evaluation. find
  42. Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 13–23. find our card
  43. Shankar Mahadevan, Sean Benhur, Roshan Nayak, Malliga Subramanian, Kogilavani Shanmugavadivel, Kanchana Sivanraju, and Bharathi Raja Chakravarthi. 2022. Transformers at semeval-2022 task 5: A feature extraction based approach for misogynous meme detection. In The 16th International Workshop on Seman… find
  44. Paridhi Maheshwari and Sharmila Reddy Nangi. 2022. Teamotter at semeval-2022 task 5: Detecting misogynistic content in multimodal memes. In The 16th International Workshop on Semantic Evaluation. find
  45. Ahmed Mahmoud Mahran, Carlo Alessandro Borella, and Konstantinos Perifanos. 2022. Codec at semeval-2022 task 5: Multi-modal multi-transformer misog-ynic meme classification framework. In The 16th International Workshop on Semantic Evaluation. find
  46. Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157. find
  47. Arianna Muti, Katerina Korre, and Alberto Barrón-Cedeño. 2022. UniBO at semeval-2022 task 5: A multimodal bi-transformer approach to the binary and fine-grained identification of misogyny in memes. In The 16th International Workshop on Semantic Evaluation. find
  48. Marinella Paciello, Francesca D’Errico, Giorgia Saleri, and Ernestina Lamponi. 2021. Online sexist meme and its effects on moral and emotional processes in social media. Computers in Human Behavior, 116:106655. find
  49. Andrei Paraschiv, Mihai Dascalu, and Dumitru Clementin Cercel. 2022. UPB at semeval-2022 task 5: Enhancing UNITER with image sentiment and graph convolutional networks for multimedia automatic misogyny identification. In The 16th International Workshop on Semantic Evaluation. find
  50. Rebecca Passonneau. 2006. Measuring agreement on set-valued items (MASI) for semantic and pragmatic annotation. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06), Genoa, Italy. European Language Resources Association (ELRA). find our card
  51. Flor-Miriam Plaza-Del-Arco, M Dolores Molina-González, L Alfonso Ureña-López, and M Teresa Martín-Valdivia. 2020. Detecting misogyny and xenophobia in spanish tweets using language technologies. ACM Transactions on Internet Technology (TOIT), 20(2):1–19. find
  52. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748–8763. PM… find our card
  53. Tathagata Raha, Sagar Joshi, and Vasudeva Varma. 2022. IIITH at semeval-2022 task 5: A comparative study of deep learning models for identifying misogynous memes. In The 16th International Workshop on Semantic Evaluation. find
  54. Ailneni Rakshitha Rao and Arjun Rao. 2022. ASRtrans at semeval-2022 task 5: Transformer-based models for meme classification. In The 16th International Workshop on Semantic Evaluation. find
  55. Jason Ravagli and Lorenzo Vaiani. 2022. JRLV at semeval-2022 task 5: The importance of visual elements for misogyny identification in memes. In The 16th International Workshop on Semantic Evaluation. find
  56. Edgar Roman-Rangel, Jorge Fuentes-Pacheco, and Jorge Hermosillo Valadez. 2022. Vision-language approach to recognize misogynous content in memes. In The 16th International Workshop on Semantic Evaluation. find
  57. Gagan Sharma, Gajanan Sunil Gitte, Shlok Goyal, and Raksha Sharma. 2022a. IITR codebusters at semeval-2022 task 5: Misogyny identification using transformers. In The 16th International Workshop on Semantic Evaluation. find
  58. Mayukh Sharma, Ilanthenral Kandasamy, and Vasantha W B. 2022b. R2d2 at semeval-2022 task 5: Attention is only as good as its values! a multimodal system for identifying misogynist memes. In The 16th International Workshop on Semantic Evaluation. find
  59. Limor Shifman. 2013. Memes in a Digital World: Reconciling with a Conceptual Troublemaker. Journal of Computer-Mediated Communication, 18(3):362– 377. find
  60. Si Si, Huan Zhang, S Sathiya Keerthi, Dhruv Mahajan, Inderjit S Dhillon, and Cho-Jui Hsieh. 2017. Gradient boosted decision trees for high dimensional sparse output. In International conference on machine learning, pages 3182–3190. PMLR. find
  61. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. find our card
  62. Rajalakshmi Sivanaiah, Angel Deborah S, Sakaya Milton Rajendram, and Mirnalinee T T. 2022. TechSSN at semeval-2022 task 5: Multimedia automatic misogyny identification using deep learning models. In The 16th International Workshop on Semantic Evaluation. find
  63. Harshvardhan Srivastava. 2022. Poirot at semeval-2022 task 5: Leveraging graph network for misogynistic meme detection. In The 16th International Workshop on Semantic Evaluation. find
  64. Since some memes contain sensitive content, we provided an explicit advisory message to the annotators. find
  65. Dawn M Szymanski, Lauren B Moffitt, and Erika R Carr. 2011. Sexual objectification of women: Advances to theory and research 1ψ7. The Counseling Psychologist, 39(1):6–38. find
  66. A.1 Chen Tao and Jung jae Kim. 2022. taochen at semeval-2022 task 5: Multimodal multitask learning and ensemble learning. In The 16th International Workshop on Semantic Evaluation. find
  67. The job aims at labelling English memes shared by users on the web as misogynous or not misogynous. The first step is about collecting socio-demographic information about the annotators: • Gender: indicate your gender as female, male, unspecified Kathleen Van Royen, Karolien Poels, Heidi Vandebosch,… find
  68. Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2021. Ernie-vil: Knowledge enhanced vision-language representations through scene graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 3208– 3216. find
  69. Wentao Yu, Benedikt Boenninghoff, Jonas Röhrig, and Dorothea Kolossa. 2022. RubCSG at semeval-2022 task 5: Ensemble learning for identifying misogynous MEMEs. In The 16th International Workshop on Semantic Evaluation. find
  70. Age: please choose your age range between 18-15, 25-35, 35-45, 45-60, over-60 • Location: please indicate your country of birth The second step is about misogyny labelling. Annotators have to decide whether a meme is misogynous or not. If a meme is labelled as misogynous, then two other questions wi… find
  71. Jing Zhang and Yujin Wang. 2022. SRCB at semeval-2022 task 5: Pretraining based image to text late sequential fusion system for multimodal misogynous meme identification. In The 16th International Workshop on Semantic Evaluation. find
  72. JIN MEI ZHI, Zhou Mengyuan, Mengfei Yuan, Dou Hu, Xiyang Du, Lianxin Jiang, Yang Mo, and XiaoFeng Shi. 2022. PAIC at semeval-2022 task 5: Multimodal misogynous detection in MEMES with multi-task learning and multi-model fusion. In The 16th International Workshop on Semantic Evaluation. find
  73. Misogyny rating: the annotator should provide a rating about how much the meme is misogynous using stars, i.e. *, ** or ***. A.2 Annotation Guidelines We report here the annotation guidelines provided to the annotators participating in the crowdsourcing annotation process of the collected memes. Gui… find
  74. Ziming Zhou, Han Zhao, Jingjing Dong, Ning Ding, Xiaolong Liu, and Kangli Zhang. 2022. DD-TIG at semeval-2022 task 5: Investigating the relationships between multimodal and unimodal information in misogynous memes detection and classification. In The 16th International Workshop on Semantic Evaluatio… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/fersini-2022-semeval-2022-task-5-multimedia/