Elamur Research⌕
☰
home / papers / 2020 / SemEval-2020 Task 8: Memotion Analysis- the Visuo-Lingual Metaphor!

SemEval-2020 Task 8: Memotion Analysis- the Visuo-Lingual Metaphor!

Summary

This shared task opened Internet meme understanding as a benchmark, since meaning comes from the interplay of picture and overlaid caption. The organizers collected memes from 52 popular themes (e.g., Trump, Minions) via Google Images. They kept only clear images with English text and had each one labelled by five Amazon Mechanical Turk workers, resolving labels by majority vote. Captions were pulled out with Google Vision OCR and then corrected by crowd workers. The resulting set of 9,871 memes supports three subtasks:

  • A: sentiment polarity
  • B: multi-label detection of humour, sarcasm, offensiveness and motivation
  • C: the graded intensity of each of these

A GloVe+CNN+LSTM text model served as a baseline, and macro-F1 was the official metric. Top scores were 0.35 (A), 0.51 (B) and 0.32 (C), which shows how hard the problem remains. Participants disagreed on whether image, text or their fusion helps most. That disagreement makes the dataset a useful testbed for multimodal affect and offensive-content research.

Key points

  • Data splits: 1K trial samples, 6,992 training and 1,879 test memes. Source URLs are stored as metadata to address copyright concerns.
  • Intensity labels follow a Likert-style scale (e.g., not/slightly/mildly/very), while motivation is binary. Labels overlap heavily: most memes are sarcastic, many funny ones are also sarcastic, and a notable share are both motivational and offensive.
  • The baseline text branch uses 100-D GloVe vectors, a CNN with 64 filters of size 1×5, 1D max-pooling and an LSTM. A multi-level setup feeds the 4-way emotion output into the 14-dimensional intensity prediction.
  • Popular participant tools included BERT, XLNet, ResNet, Inception, VGG-16, DenseNet and GloVe. To cope with class skew, teams used GMM and Training Signal Annealing (TSA).
  • Planned extensions include code-mixed memes (Hinglish, Spanglish), meme recommendation and automatic flagging of offensive memes.

Interesting ideas

  • I1. OCR followed by human correction. Running machine OCR first and asking crowd workers only to fix its errors produces clean text cheaply. This would suit any corpus built from scanned or image-embedded text, such as Ukrainian signage or documents.
  • I2. Cascaded label prediction. Coarse emotion classes are predicted first and then used as input for fine-grained intensity. This hierarchical design could help any task where labels nest, such as topic → subtopic.
  • I3. Testing modality correlation before fusing. One team (Bonheme) applied CCA/DCCA and found image and text to be complementary rather than correlated. They concluded that alignment-based methods fit memes poorly. Running this check is a cheap way to choose a fusion strategy before building one.
  • I4. Annealing the training signal for imbalance. hit-mitlab combined TSA with RandAugment image augmentation to soften class skew and noise. This could carry over to small, skewed annotated datasets in other domains.
  • I5. Five annotators with majority vote. Collecting several labels per item addresses the subjectivity of affect labels. It also leaves room to study annotator disagreement later.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

91 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 89 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.semeval-1.99

@inproceedings{sharma2020semeval,
  title = {{SemEval-2020 Task 8: Memotion Analysis- the Visuo-Lingual Metaphor!}},
  author = {Sharma, Chhavi and Bhageria, Deepesh and Scott, William and PYKL, Srinivas and Das, Amitava and Chakraborty, Tanmoy and Pulabaigari, Viswanath and Gambäck, Björn},
  booktitle = {Proceedings of the Fourteenth Workshop on Semantic Evaluation},
  year = {2020},
  doi = {10.18653/v1/2020.semeval-1.99},
  url = {https://doi.org/10.18653/v1/2020.semeval-1.99}
}

References

53 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gupta Ambuje, Kataria Harsh, Mishra Souvik, Badal Tapas, and Mishra Vipul. 2020. Memotion Analysis at SemEval-2020 Task 8: Memes Classification using Hybrid classifier. In Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020), Barcelona, Spain, Sep. Association for Com… find
  2. Arup Baruah, Kaushik Amar Das, Ferdous Ahmed Barbhuiya, and Kuntal Dey. 2020. Iiitg-adbu at semeval-2020 task 8: A multimodal approach to detect offensive, sarcastic and humorous memes. In Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020), Barcelona, Spain, Decembe… find
  3. Manish Varma Vasudeva Boinepelli, Sravani Shrivastava. 2020. ”SemEval-2020 Task 8: An Overview of Simple Text Classification Methods for Meme”. In Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020), Barcelona, Spain, Sep. Association for Computational Linguistics. find
  4. Lisa Bonheme and Marek Grzes. 2020. SESAM at SemEval-2020 Task 8: Investigating the relationship between image and text in sentiment analysis of memes. In Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020), Barcelona, Spain, Sep. Association for Computational Lingui… find
  5. Jorge Ramón Fonseca Cacho, Kazem Taghva, and Daniel Alvarez. 2019. Using the google web 1t 5-gram corpus for ocr error correction. In 16th International Conference on Information Technology-New Generations (ITNG 2019), pages 505–511. Springer. find
  6. Abu Nowshed Chy, Umme Aymun Siddiqua, and Masaki Aono. 2020. CSECU KDE MA at SemEval-2020 task 8: A neural attention model for memotion analysis. In Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020), Barcelona, Spain, Sep. Association for Computational Linguistics. find
  7. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. 2019. find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. pages 4171–4186, Minneapolis, Minnesota, June. Association for Computational Linguistics. find
  9. Jean H French. 2017. Image-based memes as sentiment predictors. In 2017 International Conference on Information Society (i-Society), pages 80–85. IEEE. find
  10. Noam Gal, Limor Shifman, and Zohar Kampf. 2016. “it gets better”: Internet memes and the construction of collective identity. New Media & Society, 18(8):1698–1714. find
  11. De la Gretel, Liz, Sarracén Peña, Rosso Paolo, and Giachanou Anastasia. 2020. PRHLT-UPV at SemEval-2020 Task 8: Study of Multimodal Techniques for Memes Analysis. In Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020), Barcelona, Spain, Sep. Association for Computa… find
  12. Ruiz Guillermo, Tellez Eric S., Moctezuma Daniela, Miranda-Jiménez Sabino, Ramı́rez-delReal Tania, and Graff Mario. 2020. Deep Learning and Text Categorization approach for Memes classification at SemEval-2020 Task 8: Memotion Analysis. In Proceedings of the 14th International Workshop on Semantic… find
  13. Sunil Gundapu and Radhika Mamidi. 2020. gundapusunil at SemEval-2020 Task 8: Multimodal Memotion Analysis. In Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020), Barcelona, Spain, Sep. Association for Computational Linguistics. find
  14. Pradyumna Gupta, Himanshu Gupta, and Aman Sinha. 2020. DSC IIT-ISM at SemEval-2020 Task 8: Bi-Fusion Techniques for Deep Meme Emotion Analysis. In Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020), Barcelona, Spain, Sep. Association for Computational Linguistics. find
  15. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep residual learning for image recognition. CoRR, abs/1512.03385. find
  16. Jie Hu, Li Shen, and Gang Sun. 2017. Squeeze-and-excitation networks. CoRR, abs/1709.01507. — [arXiv:1709.01507](https://arxiv.org/abs/1709.01507) find
  17. Shrestha Ingroj and Rusert Jonathan. 2020. NLP UIOWA at SemEval-2020 Task 8: You’re not the only one cursed with knowledge - Multi branch model memotion analysis. In Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020), Barcelona, Spain, Sep. Association for Computati… find
  18. Bejan Irina. 2020. MemoSYS at SemEval-2020 Task 8: Memotion Analysis. In Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020), Barcelona, Spain, Sep. Association for Computational Linguistics. find
  19. Noman Islam, Zeeshan Islam, and Nazia Noor. 2017. A survey on optical character recognition system. arXiv preprint arXiv:1710.05703. — [arXiv:1710.05703](https://arxiv.org/abs/1710.05703) find
  20. Max Jaderberg, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Synthetic data and artificial neural networks for natural scene text recognition. arXiv preprint arXiv:1406.2227. — [arXiv:1406.2227](https://arxiv.org/abs/1406.2227) find
  21. Das Kaushik, Amar, Baruah Arup, Barbhuiya Ferdous, Ahmed, and Dey Kuntal. 2020. KAFK at SemEval-2020 Task 8: Extracting Features From Pre-trained Neural Networks To Classify Internet Memes. In Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020), Barcelona, Spain, Sep… find
  22. Vishal Keswani, Sakshi Singh, Suryansh Agarwal, and Ashutosh Modi. 2020. IITK at SemEval-2020 Task 8: Unimodal and Bimodal Sentiment Analysis of Internet Memes. In Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020), Barcelona, Spain, Sep. Association for Computation… find
  23. Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. 2020. The hateful memes challenge: Detecting hate speech in multimodal memes. find our card
  24. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 25, pages 1097–1105. Curran Associates, Inc. find
  25. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. find
  26. Yuan Li, Wang Jin, and Zhang Xuejie. 2020. YNU-HPCC at SemEval-2020 Task 8: Using a Parallel-Channel Model for Memotion Analysis. In Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020), Barcelona, Spain, Sep. Association for Computational Linguistics. find
  27. Chenxi Liu, Barret Zoph, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan L. Yuille, Jonathan Huang, and Kevin Murphy. 2017. Progressive neural architecture search. CoRR, abs/1712.00559. — [arXiv:1712.00559](https://arxiv.org/abs/1712.00559) find
  28. Sharma Mayukh, Kandasamy Ilanthenral, and Vasantha W.B. 2020. Memebusters at SemEval-2020 Task 8: Feature Fusion Model for Sentiment Analysis on Memes using Transfer Learning. In Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020), Barcelona, Spain, Sep. Association… find
  29. Jamshed Memon, Maira Sami, and Rizwan Ahmed Khan. 2020. Handwritten optical character recognition (ocr): A comprehensive systematic literature review (slr). arXiv preprint arXiv:2001.00139. — [arXiv:2001.00139](https://arxiv.org/abs/2001.00139) find
  30. Terufumi Morishita, Gaku Morio, Shota Horiguchi, Hiroaki Ozaki, and Toshinori Miyoshi. 2020. Hitachi at SemEval-2020 Task 8: Simple but Effective Modality Ensemble for Meme Emotion Recognition. In Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020), Barcelona, Spain,… find
  31. Joanna Isabelle Olszewska. 2015. Active contour based optical character recognition for automated scene understanding. Neurocomputing, 161:65–71. find
  32. V Peirson, L Abel, and E Meltem Tolunay. 2018. Dank learning: Generating memes using deep neural networks. arXiv preprint arXiv:1806.04510. — [arXiv:1806.04510](https://arxiv.org/abs/1806.04510) find
  33. Jeffrey Pennington, Richard Socher, and Christoper Manning. 2014. Glove: Global vectors for word representation. volume 14, pages 1532–1543, 01. find
  34. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. find
  35. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2018. Language models are unsupervised multitask learners. find
  36. Wasifur Rahman, Md Kamrul Hasan, Amir Zadeh, Louis-Philippe Morency, and Mohammed Ehsan Hoque. 2019. M-bert: Injecting multimodal information in the bert structure. find
  37. Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556. find our card
  38. Pranaydeep Singh, Nina Bauwelinck, and Els Lefever. 2020. LT3 at SemEval-2020 Task 8: Multi-Modal Multi-Task Learning for Memotion Analysis. In Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020), Barcelona, Spain, Sep. Association for Computational Linguistics. find
  39. Nikhil Sonnad. 2018. The world’s biggest meme is the word “meme” itself. find
  40. Andi Suciati and Indra Budi. Classi[U+FB01]cation. UI at SemEval-2020 Task 8: Text-Image Fusion for Sentiment Steve Durairaj Swamy, Shubham Laddha, Basil Abdussalam, Debayan Datta, and Anupam Jamatia. 2020. Nit-agartala-nlp-team at semeval-2020 task 8: Building multimodal classifiers to tackle inter… find
  41. Christian Szegedy, Sergey Ioffe, and Vincent Vanhoucke. 2016. Inception-v4, inception-resnet and the impact of residual connections on learning. CoRR, abs/1602.07261. — [arXiv:1602.07261](https://arxiv.org/abs/1602.07261) find
  42. Walinska Urszula and Potoniec Jedrzej. 2020. Urszula Walinska at SemEval-2020 Task 8: Memotion Analysis. In Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020), Barcelona, Spain, Sep. Association for Computational Linguistics. find
  43. George-Alexandru Vlad, George-Eduard Zaharia, Dumitru-Clementin Cercel, Costin-Gabriel Chiru, and Stefan Trausan-Matu. 2020. Upb at semeval-2020 task 8: Joint textual and visual modeling in a multi-task learning architecture for memotion analysis. In Proceedings of the 14th International Workshop on… find
  44. Urszula Walinska and Jedrzej Potoniec. 2020. Urszula Walinska at SemEval-2020 Task 8: Memotion Analysis. In Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020). find
  45. Amanda Williams, Clio Oliver, Katherine Aumer, and Chanel Meyers. 2016. Racial microaggressions and perceptions of internet memes. Computers in Human Behavior, 63:424–432. find
  46. Guo Xiaoyu, Ma Jing, and Zubiaga Arkaitz. 2020. NUAA-QMUL at SemEval 2020 Task 8:Utilizing BERT and DenseNet for Internet Meme Emotion Analysis. In Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020), Barcelona, Spain, Sep. Association for Computational Linguistics. find
  47. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. CoRR, abs/1906.08237. find our card
  48. Guo Yingmei, Huang Jinfa, Dong Yanlong, and Xu Mingxing. 2020. guoym at SemEval-2020 Task 8: Ensemble-based Classification of Visuo-Lingual Metaphor in Memes. In Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020), Barcelona, Spain, Sep. Association for Computational… find
  49. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. SemEval-2019 Task 6: Identifying and categorizing offensive language in social media (OffensEval). In 13th International Workshop on Semantic Evaluation, pages 75–86. ACL. find our card
  50. Liu Zehao, Osei-Brefo Emmanuel, Chen Siyuan, and Liang Huizhi. 2020. Semeval-2020 task 8: Gaussian mixture modelling (gmm)based sampling approach for multi-modal memotion analysis. In Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020), Barcelona, Spain, December. As… find
  51. Xingcheng Zhang, Zhizhong Li, Chen Change Loy, and Dahua Lin. 2016. Polynet: A pursuit of structural diversity in very deep networks. find
  52. Sicheng Zhao, Guiguang Ding, Tat-Seng Chua, Björn Schuller, and Kurt Keutzer. 2018. Affective image content analysis: A comprehensive survey. pages 5534–5541, 07. find
  53. Li Zhen, Zhang Yaojie, Xu Bing, and Zhao Tiejun. 2020. CN-HIT-MI.T at SemEval-2020 Task 8: Memotion Analysis Based on BERT. In Proceedings of the 14th International Workshop on Semantic Evaluation (SemEval-2020), Barcelona, Spain, Sep. Association for Computational Linguistics. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sharma-2020-semeval-2020-task-8-memotion/