Elamur Research
☰
home / papers / 2020 / The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes

The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2005.04790

Summary

Many vision-and-language benchmarks can be solved largely from a single modality, so they say little about genuine multimodal reasoning. The authors address this with a challenge set for classifying memes as hateful or not. In these memes, harmless text and a harmless picture can combine into an attack on a protected group. The memes were recreated from real social-media examples using licensed Getty Images stock photos and a custom tool, so the data can be redistributed and OCR noise is removed. Trained annotators labeled each meme against a strict hate speech definition. For hateful items, they also built "benign confounders": versions with a swapped image or swapped caption that make the meme non-hateful, so a unimodal shortcut fails. The paper tests unimodal baselines and multimodal fusion models of increasing complexity. Performance rises with fusion sophistication, but even the best pre-trained multimodal systems remain well below human accuracy. This makes the dataset useful both for measuring multimodal progress and for practical content moderation.

Key points

  • The source pool had over 1 million US public social-media images. After filtering and deduplication, 162k memes remained. Language and policy screening, plus removal of memes containing slurs, left about 46k candidates.
  • Replacement images came from Getty Images under a research redistribution license. Memes were stored as PNG and SVG, so the text can be edited or removed.
  • Annotators came from a third-party company, not a crowdsourcing platform. They had 4 hours of training and three pilot rounds, and spent about 27 minutes per final meme.
  • Each meme received five ratings on a 1–3 scale. Items with strong disagreement were sent to an expert, and the final output is a binary label.
  • Benign image confounders and benign text confounders flip hateful memes to non-hateful. This forces models to combine both modalities rather than rely on unimodal cues.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

263 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 252 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2005.04790 · arXiv 2005.04790

@misc{kiela2020hateful,
  title = {{The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes}},
  author = {Kiela, Douwe and Firooz, Hamed and Mohan, Aravind and Goswami, Vedanuj and Singh, Amanpreet and Ringshia, Pratik and Testuggine, Davide},
  year = {2020},
  doi = {10.48550/arxiv.2005.04790},
  eprint = {2005.04790},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2005.04790}
}

References

94 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. Analyzing the behavior of visual question answering models. arXiv preprint arXiv:1606.07356, 2016. find
  2. Firoj Alam, Ferda Ofli, and Muhammad Imran. Crisismmd: Multimodal twitter datasets from natural disasters. In Twelfth International AAAI Conference on Web and Social Media, 2018. find
  3. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425–2433, 2015. find
  4. John Arevalo, Thamar Solorio, Manuel Montes-y Gómez, and Fabio A González. Gated multimodal units for information fusion. In Proceedings of ICLR, 2017. find
  5. Marco Baroni. Grounding distributional semantics in the visual world. Language and Linguistics Compass, 10(1):3–13, 2016. find
  6. Andrew P Bradley. The use of the area under the roc curve in the evaluation of machine learning algorithms. Pattern recognition, 30(7):1145–1159, 1997. find
  7. Ozan Caglayan, Pranava Madhyastha, Lucia Specia, and Loïc Barrault. Probing the need for visual context in multimodal machine translation. arXiv preprint arXiv:1903.08678, 2019. find
  8. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015. find our card
  9. A. Das, S. Kottur, K. Gupta, A. Singh, D. Yadav, J. M. Moura, Yadav D., D. Parikh, and D. Batra. Visual dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 326–335, 2017. find
  10. Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. Racial bias in hate speech and abusive language detection datasets. In Proceedings of ACL, 2019. find our card
  11. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. Automated hate speech detection and the problem of offensive language. In Eleventh international aaai conference on web and social media, 2017. find our card
  12. H. De Vries, F. Strub, S. Chandar, O. Pietquin, H. Larochelle, and A. Courville. Guesswhat?! visual object discovery through multi-modal dialogue. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5503–5512, 2017. find
  13. Jean-Benoit Delbrouck and Stéphane Dupont. An empirical study on the effectiveness of images in multimodal neural machine translation. arXiv preprint arXiv:1707.00995, 2017. find
  14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. find
  15. Jacob Devlin, Saurabh Gupta, Ross Girshick, Margaret Mitchell, and C Lawrence Zitnick. Exploring nearest neighbor approaches for image captioning. arXiv preprint arXiv:1505.04467, 2015. — [arXiv:1505.04467](https://arxiv.org/abs/1505.04467) find
  16. Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, , and Lucy Vasserman. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 67–73, 2018. find
  17. Desmond Elliott. Adversarial evaluation of multimodal machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2974–2978, 2018. find
  18. Desmond Elliott, Stella Frank, Khalil Sima’an, and Lucia Specia. Multi30k: Multilingual english-german image descriptions. arXiv:1605.00459, 2016. find our card
  19. Paula Fortuna and Sérgio Nunes. A survey on automatic detection of hate speech in text. 51(4), 2018. find
  20. Antigoni Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. Large scale crowdsourcing and characterization of twitter abusive behavior. In Twelfth International AAAI Conference on… find
  21. I. Gallo, A. Calefati, and S. Nawaz. Multimodal classification fusion in real-world scenarios. In 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), volume 5, pages 36–41, 2017. find
  22. Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, et al. Evaluating nlp models via contrast sets. arXiv preprint arXiv:2004.02709, 2020. find our card
  23. Donald Geman, Stuart Geman, Neil Hallonquist, and Laurent Younes. Visual turing test for computer vision systems. Proceedings of the National Academy of Sciences, 112(12):3618–3623, 2015. find
  24. Jennifer Golbeck, Zahra Ashktorab, Rashad O. Banjo, Alexandra Berlinger, Siddharth Bhagwan, Cody Buntain, Paul Cheakalos, Alicia A. Geller, Quint Gergory, Rajesh Kumar Gnanasekaran, Raja Rajan Gunasekaran, Kelly M. Hoffman, Jenny Hottle, Vichita Jienjitlert, Shivika Khare, Ryan Lau, Marianna J. Mart… find
  25. Raul Gomez, Jaume Gibert, Lluis Gomez, and Dimosthenis Karatzas. Exploring hate speech detection in multimodal publications. In The IEEE Winter Conference on Applications of Computer Vision, pages 1470–1478, 2020. find
  26. Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6904–6913, 2017. find our card
  27. Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3608–3617, 2018. find
  28. Danna Gurari, Yinan Zhao, Meng Zhang, and Nilavra Bhattacharya. Captioning images taken by people who are blind. arXiv preprint arXiv:2002.08565, 2020. — [arXiv:2002.08565](https://arxiv.org/abs/2002.08565) find
  29. Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017. find
  30. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. find
  31. Homa Hosseinmardi, Sabrina Arredondo Mattson, Rahat Ibn Rafiq, Richard Han, Qin Lv, and Shivakant Mishra. Detection of cyberbullying incidents on the instagram social network. arXiv:1503.03909, 2015. — [arXiv:1503.03909](https://arxiv.org/abs/1503.03909) find
  32. Ting-Hao Kenneth Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, C. Lawrence Zitnick, Devi Parikh, Lucy Vanderwende, Michel Galley, and Margaret Mitchell. Visual storytelling. In Proceedings of NAACL-… find
  33. Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. arXiv preprint arXiv:1902.09506, 2019. find our card
  34. Z. Hussain, M. Zhang, X. Zhang, K. Ye, C. Thomas, Z. Agha, N. Ong, and A. Kovashka. Automatic understanding of image and video advertisements. pages 1705–1715, 2017. find
  35. Allan Jabri, Armand Joulin, and Laurens Van Der Maaten. Revisiting visual question answering baselines. In European conference on computer vision, pages 727–739. Springer, 2016. find
  36. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2901–2910, 201… find our card
  37. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759, 2016. find
  38. Divyansh Kaushik, Eduard Hovy, and Zachary C Lipton. Learning the difference that makes a difference with counterfactually-augmented data. arXiv preprint arXiv:1909.12434, 2019. find
  39. S. Kazemzadeh, V. Ordonez, M. Matten, and T. Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing, pages 787–798, 2014. find
  40. Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, and Davide Testuggine. Supervised multimodal bitransformers for classifying images and text. arXiv preprint arXiv:1909.02950, 2019. — [arXiv:1909.02950](https://arxiv.org/abs/1909.02950) find
  41. Douwe Kiela, Luana Bulat, Anita L Vero, and Stephen Clark. Virtual embodiment: A scalable long-term strategy for artificial intelligence research. arXiv preprint arXiv:1610.07432, 2016. — [arXiv:1610.07432](https://arxiv.org/abs/1610.07432) find
  42. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. find
  43. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32–73, 2… find our card
  44. J. Kruk, J. Lubin, K. Sikka, X. Lin, D. Jurafsky, and A. Divakaran. Integrating text and image: Determining multimodal document intent in instagram posts. In Proceedings of EMNLP, 2019. find
  45. Ritesh Kumar, Atul Kr Ojha, Shervin Malmasi, and Marcos Zampieri. Benchmarking aggression identification in social media. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018), pages 1–11, 2018. find
  46. Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557, 2019. find our card
  47. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014. find our card
  48. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. find
  49. Di Lu, Leonardo Neves, Vitor Carvalho, Ning Zhang, and Heng Ji. Visual attention model for name tagging in multimodal social media. In Proceedings of ACL, pages 1990–1999, 2018. find
  50. Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. arXiv preprint arXiv:1908.02265, 2019. find our card
  51. Jerry Ma and Denis Yarats. On the adequacy of untuned warmup for adaptive optimization. arXiv preprint arXiv:1910.04209, 2019. — [arXiv:1910.04209](https://arxiv.org/abs/1910.04209) find
  52. Y. Ma, Z. Xiang, Q. Du, and W. Fan. Effects of user-provided photos on hotel review helpfulness: An analytical approach with deep leaning. International Journal of Hospitality Management, 71:120–131, 2018. find
  53. S Malmasi and M Zampieri. Detecting hate speech in social media. arXiv:1712.06427, 2017. — [arXiv:1712.06427](https://arxiv.org/abs/1712.06427) find
  54. Shervin Malmasi and Marcos Zampieri. Challenges in discriminating profanity from hate speech. Journal of Experimental & Theoretical Artificial Intelligence, 30(2):187–202, 2018. find
  55. Mary L McHugh. Interrater reliability: the kappa statistic. Biochemia medica: Biochemia medica, 22(3):276–282, 2012. find
  56. A. Mogadala, M. Kalimuthu, and D. Klakow. Trends in integration of vision and language research: A survey of tasks, datasets, and methods. arXiv:1907.09358, 2019. — [arXiv:1907.09358](https://arxiv.org/abs/1907.09358) find
  57. C. C. Park and G. Kim. Expressing an image stream with a sequence of natural sentences. In Advances in neural information processing systems, pages 73–81, 2015. find
  58. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, pages 8024–8035, 2019. find our card
  59. Arnau Ramisa. Multimodal news article analysis. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, pages 5136–5140, 2017. find
  60. Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pages 91–99, 2015. find our card
  61. M. H. Ribeiro, P. H. Calais, Y. A. Santos, V. A. Almeida, and W. Meira Jr. Characterizing and detecting hateful users on twitter. In Twelfth International AAAI Conference on Web and Social Media, 2018. find
  62. Benet Oriol Sabat, Cristian Canton Ferrer, and Xavier Giro i Nieto. Hate speech in pixels: Detection of offensive memes towards automatic moderation. arXiv preprint 1910.02334, 2019. — [arXiv:1910.02334](https://arxiv.org/abs/1910.02334) find
  63. Maarten Sap, Dallas Card, Saadia Gabriela, Yejin Choi, and Noah A. Smith. The risk of racial bias in hate speech detection. In Proceedings of ACL, pages 1668–1678, 2019. find our card
  64. Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE International Conference on Computer Vision, pages 9339–9347, 2019. find
  65. Anna Schmidt and Michael Wiegand. A survey on hate speech detection using natural language processing. 2017. find our card
  66. Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565, 2018. find our card
  67. Kai Shu, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. Fake news detection on social media: A data mining perspective. arxiv:1708.01967, 2017. — [arXiv:1708.01967](https://arxiv.org/abs/1708.01967) find
  68. Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. arXiv preprint arXiv:2003.12462, 2020. find
  69. Amanpreet Singh, Vedanuj Goswami, Vivek Natarajan, Yu Jiang, Xinlei Chen, Meet Shah, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. Pythia-a platform for vision & language research. In SysML Workshop, NeurIPS, volume 2018, 2018. find
  70. Amanpreet Singh, Vedanuj Goswami, and Devi Parikh. Are we pretraining it right? digging deeper into visio-linguistic pretraining. arXiv preprint arXiv:2004.08744, 2020. — [arXiv:2004.08744](https://arxiv.org/abs/2004.08744) find
  71. Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8317–8326, 2019. find
  72. Vivek K. Singh, Souvick Ghosh, and Christin Jose. Toward multimodal cyberbullying detection. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems, pages 2090–2099, 2017. find
  73. M. Soleymani, D. Garcia, B. Jou, B. Schuller, S. F. Chang, and M. Pantic. A survey of multimodal sentiment analysis. Image and Vision Computing, 65:3–14, 2017. find
  74. L. Specia, S. Frank, K. Sima’an, and D. Elliott. A shared task on multimodal machine translation and crosslingual image description. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, 2016. find our card
  75. Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs. arXiv preprint arXiv:1811.00491, 2018. find our card
  76. Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. arXiv preprint arXiv:1503.01817, 2015. find
  77. Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015. find our card
  78. Prashanth Vijayaraghavan, Hugo Larochelle, and Deb Roy. Interpretable multi-modal hate speech detection. In AI for Social Good Workshop at the International Conference on Machine Learning, 2019. find
  79. Hoa Trong Vu, Claudio Greco, Aliia Erofeeva, Somayeh Jafaritazehjan, Guido Linders, Marc Tanti, Alberto Testoni, Raffaella Bernardi, and Albert Gatt. Grounded textual entailment. arXiv preprint arXiv:1806.05645, 2018. find
  80. Xin Wang, Devinder Kumar, Nicolas Thome, Matthieu Cord, and Frederic Precioso. Recipe recognition with large multimodal food dataset. In 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pages 1–6. IEEE, 2015. find
  81. Z. Waseem, T. Davidson, D. Warmsley, and I. Weber. Understanding abuse: A typology of abusive language detection subtasks. 2017. find our card
  82. Z. Waseem and D. Hovy. Hateful symbols or hateful people? predictive features for hate speech detection on twitter. pages 88–93, 2016. find
  83. Zeerak Waseem. Are you a racist or am I seeing things? annotator influence on hate speech detection on twitter. In Proceedings of the First Workshop on NLP and Computational Social Science, pages 138–142, 2016. find
  84. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-ofthe-art natural language processing. arXiv preprint arXiv:1910.03771, 2019. find our card
  85. Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: A novel task for fine-grained image understanding. arXiv preprint arXiv:1901.06706, 2019. — [arXiv:1901.06706](https://arxiv.org/abs/1901.06706) find
  86. Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. arXiv preprint arXiv:1611.05431, 2016. — [arXiv:1611.05431](https://arxiv.org/abs/1611.05431) find
  87. Fan Yang, Xiaochang Peng, Gargi Ghosh, Reshef Shilon, Hao Ma, Eider Moore, and Goran Predovic. Exploring deep multimodal fusion of text and photo for hate speech classification. In Proceedings of the Third Workshop on Abusive Language Online, pages 11–18, 2019. find
  88. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67–78, 2014. find
  89. J. Yu and J. Jiang. Adapting bert for target-oriented multimodal sentiment classification. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 5408–5414, 2019. find
  90. T. Zahavy, A. Magnani, A. Krishnan, and S. Mannor. Is a picture worth a thousand words? a deep multi-modal fusion architecture for product classification in e-commerce. arXiv:1611.09534, 2016. — [arXiv:1611.09534](https://arxiv.org/abs/1611.09534) find
  91. R. Zellers, Y. Bisk, A. Farhadi, and Y. Choi. pages 6720–6731, 2019. find
  92. Mingda Zhang, Rebecca Hwa, and Adriana Kovashka. Equal but not the same: Understanding the implicit relationship between persuasive images and text. arxiv:1807.08205, 2018. — [arXiv:1807.08205](https://arxiv.org/abs/1807.08205) find
  93. Haoti Zhong, Hao Li, Anna Squicciarini, Sarah Rajtmajer, Christopher Griffin, David Miller, and Cornelia Caragea. Content-driven detection of cyberbullying on the instagram social network. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, pages 3952–3958,… find
  94. Bolei Zhou, Yuandong Tian, Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. Simple baseline for visual question answering. arXiv preprint arXiv:1512.02167, 2015. — [arXiv:1512.02167](https://arxiv.org/abs/1512.02167) find