Elamur Research⌕
☰
home / papers / 2021 / MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targets

MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targets

Summary

The paper addresses automatic recognition of harmful memes, where an image and its overlaid text together can damage a person, organization, community or society, even when neither part looks offensive alone. It goes beyond earlier work on hate or offensiveness by framing two tasks: judging how harmful a meme is, and identifying which kind of social entity it targets. The authors build on their earlier COVID-19 HarMeme collection. They release Harm-C (COVID-19) and a new US-politics set, Harm-P, made of real-world memes gathered from the web and social platforms, filtered for quality and deduplicated. They also present MOMENTA, a deep multimodal network. It combines whole-meme representations from CLIP with finer cues, such as detected faces and image attributes, and links them to background context. Compared with ten unimodal and multimodal baselines, MOMENTA achieves the best accuracy on both tasks. It also transfers better across datasets, and LIME analyses show which image regions and words drove its decisions. The work matters because harmful content in memes is often subtle and depends on context.

Key points

  • Datasets:
    • Harm-C contains 3,544 memes and Harm-P contains 3,552 memes.
    • Each meme is labeled as very harmful, partially harmful, or harmless.
    • Harmful memes also carry a target label: individual, organization, community, or society.
  • Benchmark results: MOMENTA outperforms ten state-of-the-art baselines by 1.3–2.6 absolute accuracy points on both tasks.
  • Cross-dataset transfer: Compared with ViLBERT and Visual BERT, MOMENTA gains in F1:
    • 2.2–6.6 points for 2-class detection,
    • 1.1–9.0 points for 3-class detection,
    • 0.9–4.2 points for target identification.
    • The authors attribute much of this to the CLIP encoder, which was pre-trained on 400M image–text pairs.
  • Interpretability and errors:
    • In LIME visualizations, MOMENTA caught an insulting, partially occluded face of Joe Biden that ViLBERT missed.
    • Its failures involve memes whose harm comes only from implicit overall meaning.
  • Availability: Code and data are at github.com/LCS2-IIITD/MOMENTA, released for research use only. The datasets are distributed as links to the memes rather than the images themselves.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.findings-emnlp.379

@inproceedings{pramanick2021momenta,
  title = {{MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targets}},
  author = {Pramanick, Shraman and Sharma, Shivam and Dimitrov, Dimitar and Akhtar, Md. Shad and Nakov, Preslav and Chakraborty, Tanmoy},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2021},
  year = {2021},
  doi = {10.18653/v1/2021.findings-emnlp.379},
  url = {https://doi.org/10.18653/v1/2021.findings-emnlp.379}
}

References

5 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Byeongtae Ahn and Seok-Woo Jang. 2019. Multimodal approach for multimedia injurious contents blocking. Multimedia Tools and Applications, 79:16459–16472. find
  2. Michael A Alcorn, Qi Li, Zhitao Gong, Chengfei Wang, Long Mai, Wei-Shinn Ku, and Anh Nguyen. 2019. Strike (with) a pose: Neural networks are easily fooled by strange poses of familiar objects. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR ’19, pages 4845–4854… find
  3. Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. 2009. Evaluation measures for ordinal regression. In Proceedings of the Ninth International Conference on Intelligent Systems Design and Applications, ISDA ’09, pages 283–287, Pisa, Italy. find
  4. Ramy Baly, Georgi Karadzhov, Jisun An, Haewoon Kwak, Yoan Dinkov, Ahmed Ali, James Glass, and Preslav Nakov. 2020. What was written vs. who read it: News media profiling using text analysis and social media context. In Proceedings of the 58th Annual Meeting of the Association for Computational Lingu… find
  5. Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. 2019. ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In Proceedings of the 33rd International Conference on Neural Informatio… find

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pramanick-2021-momenta-multimodal-framework-detecting-harmful/