Elamur Research⌕
☰
home / papers / 2024 / Detecting Hate Speech in Amharic Using Multimodal Analysis of Social Media Memes

Detecting Hate Speech in Amharic Using Multimodal Analysis of Social Media Memes

Paper · 2024 · Melese Ayichlie Jigar, Abinew Ali Ayele, Seid Muhie Yimam et al.

Summary

On Ethiopian social media, hateful content often spreads through memes. In a meme, the picture and the overlaid words carry meaning together. Earlier Amharic detectors used only plain text or text recovered from memes by OCR, so they ignored the visual side.

The authors built a meme benchmark. During a four-month window, 74 trained volunteers collected about 10k memes posted on Telegram, Twitter and Facebook. These were filtered to 2,007 memes that contain both image and text and are not merely jokes. Three native speakers labeled each item as hate or non-hate in a custom web tool, HateMemAnno, and majority vote set the gold label.

For modelling, Tesseract OCR extracted the embedded text, which was turned into word2Vec features. Images were encoded with VGG16. The two vectors were joined by concatenation and passed to LSTM, BiLSTM and CNN classifiers.

The fused BiLSTM performed best, ahead of every text-only and image-only setup. This shows that visual cues compensate for noisy OCR in a low-resource script. The released resources give a base for Amharic multimodal moderation work.

Key points

  • Dataset: 2,007 memes, of which 919 are labeled hate and 1,088 non-hate. They were filtered from about 10k collected by 74 contributors through a Telegram group.
  • Annotation: three native Amharic speakers per meme, working in four batches of 500 in HateMemAnno. Agreement was 0.50 kappa (called Fleiss' in the abstract, Cohen's in the body).
  • Features: Tesseract OCR plus word2Vec for text, VGG16 for images, and Keras Concatenate for fusion.
  • Results (accuracy): BiLSTM reached 63% on text alone and 75% on multimodal input. CNN reached 69% on images alone.
  • Dataset, guidelines, tool, models and code are released at github.com/uhh-lt/AmharicHateSpeech.

Interesting ideas

  • I1. Code-mixed English words inside Amharic text were kept rather than deleted, because removing a word like "GENOCIDERS" would erase the hateful meaning.
  • I2. The error analysis traces each misclassification to its source: OCR failing on curved, distorted text; a word whose sense differs by region; and sarcasm the model missed.
  • I3. One Amharic word means "servant/slave" in Gojjam and Wollo but "young person" in Gondar, so offensiveness depends on the reader's dialect.
  • I4. Running each modality alone and then fused shows how much each input source actually contributes.
  • I5. A sarcastic jab at agriculture students was read as normal, showing that surface-literal classification misses ironic hostility.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): the tokenizer could keep foreign-language or foreign-script tokens as language-tagged spans, so their meaning still reaches the parse.
  • I2 (learn): evaluation reports could sort errors by the pipeline stage that caused them, so repair effort goes to the right component.
  • I3 (world): lexical senses in compiled knowledge could carry region or variety qualifiers, letting the world model choose a sense from the text's origin.
  • I4 (learn): ablating one evidence source at a time, such as a witness parser or knowledge layer, would quantify its real contribution as a built-in control.
  • I5 (frames): the irony and implicature module could flag mismatches between literal content and evaluative stance, marking such utterances as possibly hostile.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 4Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 worksVision & language · 657 worksBias, fairness & safety · 517 works

Cite

Cite as Melese Ayichlie Jigar, Abinew Ali Ayele, Seid Muhie Yimam and Chris Biemann. 2024. Detecting Hate Speech in Amharic Using Multimodal Analysis of Social Media Memes. Proceedings of the Fourth Workshop on Threat, Aggression & Cyberbullying @ LREC-COLING-2024.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{jigar2024detecting,
  title = {{Detecting Hate Speech in Amharic Using Multimodal Analysis of Social Media Memes}},
  author = {Melese Ayichlie Jigar, Abinew Ali Ayele, Seid Muhie Yimam and Chris Biemann},
  booktitle = {Proceedings of the Fourth Workshop on Threat, Aggression \& Cyberbullying @ LREC-COLING-2024},
  year = {2024},
  doi = {10.63317/3vffy9zhrm6n},
  url = {https://doi.org/10.63317/3vffy9zhrm6n}
}

References

44 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Zeleke Abebaw, Andreas Rauber, and Solomon Atnafu. 2022. Multi-channel convolutional neural network for hate speech detection in social media. In proceedings of the 9th EAI International Conference on the Advances of Science and Technology (ICAST), pages 603–618, Bahir Dar, Ethiopia. Springer. find
  2. Ayme Arango, Jesus Perez-Martin, and Arniel Labrada. 2022. HateU at SemEval-2022 task 5: Multimedia automatic misogyny identification. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 581–584, Seattle, WA, USA. Association for Computational Linguistics.
  3. Abinew Ali Ayele, Tadesse Destaw Belay, Seid Muhie Yimam, Skadi Dinter, Tesfa Tegegne Asfaw, and Chris Biemann. 2022a. Challenges of Amharic hate speech data Annotation using Yandex Toloka crowdsourcing platform. In Proceedings of the sixth Widening NLP Workshop (WiNLP), Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  4. Abinew Ali Ayele, Skadi Dinter, Tadesse Destaw Belay, Tesfa Tegegne Asfaw, Seid Muhie Yimam, and Chris Biemann. 2022b. The 5Js in Ethiopia: Amharic hate speech data annotation using Toloka Crowdsourcing Platform. In Proceedings of the 4th International Conference on Information and Communication Technology for Development for Africa (ICT4DA), pages 114– 120, Bahir Dar, Ethiopia. IEEE. find
  5. Abinew Ali Ayele, Skadi Dinter, Seid Muhie Yimam, and Chris Biemann. 2023a. Multilingual racial hate speech detection using transfer learning. In Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing, pages 41–48, Varna, Bulgaria. IN-COMA Ltd., Shoumen, Bulgaria. find
  6. Abinew Ali Ayele, Seid Muhie Yimam, Tadesse Destaw Belay, Tesfa Asfaw, and Chris Biemann. 2023b. Exploring Amharic hate speech data collection and classification approaches. In Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing (RANLP2023), pages 59–59, Varna, Bulgaria. Association for Computational Linguistics.
  7. Tadesse Destaw Belay, Abinew Ali Ayele, Getie Gelaye, Seid Muhie Yimam, and Chris Biemann. 2021. Impacts of homophone normalization on semantic models for Amharic. In Proceedings of the 3rd International Conference on Information and Communication Technology for Development for Africa (ICT4DA), pages 101–106, Bahir Dar, Ethiopia. IEEE. find
  8. Aruna Bhat, Vaibhav Vashisht, Vaibhav Raj Sahni, and Sumit Meena. 2023. Hate speech detection using multimodal meme analysis. In Proceedings of the 2nd International Conference on Applied Artificial Intelligence and Computing (ICAAIC), pages 1137–1142, Salem, India. IEEE. find
  9. João Bran and Adeline Hulin. 2023. Social Media 4 Peace: local lessons for global practices. Countering hate speech. the United Nations Educational, Scientific and Cultural Organization (UN-ESCO. find
  10. Rui Cao, Roy Ka-Wei Lee, Wen-Haw Chong, and Jing Jiang. 2022. Prompting for multimodal hateful meme classification. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 321–332, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  11. Michele Corazza, Stefano Menini, Pinar Arslan, Rachele Sprugnoli, Elena Cabrio, Sara Tonelli, and Serena Villata. 2018. Comparing different supervised approaches to hate speech detection. In Proceedings of The Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian (EVALITA 2018), pages 230–234. European Language Resources Association (ELRA), Turin, Italy. ↗
  12. Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. 2019. Racial bias in hate speech and abusive language detection datasets. In Proceedings of the Third Workshop on Abusive Language Online, pages 25–35, Florence, Italy. Association for Computational Linguistics. ↗
  13. Abreham Gebremedin Debele, Michael Melese and Woldeyohannis. 2022. Multimodal Amharic hate speech detection using deep learning. In 2022 International Conference on Information and Communication Technology for Development for Africa (ICT4DA), pages 102–107, Bahir Dar, Ethiopia. IEEE. find
  14. Naol Bakala Defersha and Kula Kekeba Tune. 2021. Detection of hate speech text in Afan Oromo social media using machine learning approach. Indian Journal of Science Technology, 14(31):2567–2578. find
  15. Mequanent Degu, Abebe Tesfahun, and Haymanot Takele. 2023. Amharic language hate speech detection system from facebook memes using deep learning system. Available at SSRN 4389914. find
  16. Eva D’hondt, Cyril Grouin, and Brigitte Grau. 2017. Generating a training corpus for OCR post-correction using encoder-decoder model. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1006–1014, Taipei, Taiwan. Asian Federation of Natural Language Processing. find
  17. Andargachew Mekonnen Gezmu, Binyam Ephrem Seyoum, Michael Gasser, and Andreas Nürnberger. 2018. Contemporary Amharic corpus: Automatically morpho-syntactically tagged Amharic corpus. In Proceedings of the First Workshop on Linguistic Resources for Natural Language Processing, pages 65–70, Santa Fe, NM, USA. Association for Computational Linguistics.
  18. Raul Gomez, Jaume Gibert, Lluis Gomez, and Dimosthenis Karatzas. 2020. Exploring hate speech detection in multimodal publications. In 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1459–1467, Snowmass Village, CO, USA. IEEE. find
  19. Eftekhar Hossain, Omar Sharif, and Mo-Mute: A hammed Moshiul Hoque. 2022. multimodal dataset for detecting hateful memes. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing: Student Research Workshop, pages 32–39, Online Only. Association for Computational Linguistics. find
  20. Oana Ignat, Jean Maillard, Vishrav Chaudhary, and Francisco Guzmán. 2022. OCR improves machine translation for low-resource languages. In Proceedings of the Association for Computational Linguistics: ACL 2022, pages 1164–1174, Dublin, Ireland. Association for Computational Linguistics. find
  21. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427–431, Valencia, Spain. Association for Computational Linguistics. Md. find
  22. Rezaul Karim, Sumon Kanti Dey, Tanhim Islam, Md Shajalal, and Bharathi Raja Chakravarthi. 2023. Multimodal hate speech detection from bengali memes and texts. In Proceedings of the first International Conference on Speech and Language Technologies for Low-Resource Languages, pages 293–308, Beijing, China. Springer International Publishing. find
  23. Simon Kemp. 2023. Digital 2023: Global overview report. Accessed Oct. 20, 2023, URL: https://datareportal.com/reports/digital-2023-global-overview-report. ↗
  24. Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. 2020. The hateful memes challenge: Detecting hate speech in multimodal memes. In proceedings of the 34th International Conference on Neural Information Processing Systems (NeurIPS 2020), pages 2611–2624, Vancouver, Canada. ↗
  25. Kosisochukwu Madukwe, Xiaoying Gao, and Bing Xue. 2020. In data we trust: A critical analysis of hate speech detection datasets. In Proceedings of the Fourth Workshop on Online Abuse and Harms, pages 150–161, Online. Association for Computational Linguistics. find
  26. Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. 2021. HateXplain: A benchmark dataset for explainable hate speech detection. In Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence, pages 14867– 14875, Palo Alto, CA, USA. Association for the Advancement of Artificial Intelligence. find
  27. Zewdie Mossie and Jenq-Haur Wang. 2018. Social network hate speech detection for Amharic In 4th International Conference language. on Natural Language Computing (NATL2018), pages 41–55, Dubai, United Arab Emirates. AIRCC Publishing. find
  28. Zewdie Mossie and Jenq-Haur Wang. 2020. Vulnerable community identification using hate speech detection on social media. Information Processing & Management, 57(3):1–16. ↗
  29. Konstantinos Perifanos and Dionysis Goutsos. 2021. Multimodal hate speech detection in Greek social media. Multimodal Technologies and Interaction, 5(7):1–10. Shraman Pramanick, Shivam Sharma, Dimitar Dimitrov, Md. ↗
  30. Shad Akhtar, Preslav Nakov, and Tanmoy Chakraborty. 2021. MOMENTA: A multimodal framework for detecting harmful memes and their targets. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4439–4455, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  31. Aneri Rana and Sonali Jha. 2022. Emotion based hate speech detection using multimodal learning. ArXiv, abs/2202.06218. ↗
  32. Paul Röttger, Haitham Seelawi, Debora Nozza, Zeerak Talat, and Bertie Vidgen. 2022. Multilingual HateCheck: Functional tests for multilingual hate speech detection models. In Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH), pages 154–169, Seattle, Washington (Hybrid). Association for Computational Linguistics. find
  33. Siva Sai, Naman Deep Srivastava, and Yashvardhan Sharma. 2022. Explorative application of fusion techniques for multimodal hate speech detection. SN Computer Science, 3(2):1–13. ↗
  34. Anna Schmidt and Michael Wiegand. 2017. A survey on hate speech detection using natural language processing. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, pages 1–10, Valencia, Spain. Association for Computational Linguistics.
  35. Jannik Strötgen and Michael Gertz. 2012. Temporal tagging on different domains: Challenges, strategies, and gold standards. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), pages 3746–3753, Istanbul, Turkey. European Language Resource Association (ELRA).
  36. Shardul Suryawanshi, Bharathi Raja Chakravarthi, Mihael Arcan, and Paul Buitelaar. 2020. Multimodal meme dataset (MultiOFF) for identifying offensive content in image and text. In Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying, pages 32–41, Marseille, France. European Language Resources Association (ELRA).
  37. Surafel Getachew Tesfaye and Kula Kakeba. 2020. Automated Amharic hate speech posts and comments detection model using recurrent neural network. Preprint. Version 1. ↗
  38. Riza Velioglu and Jewgeni Rose. 2020. Detecting hate speech in memes using multimodal deep learning approaches: Prize-winning solution to hateful memes challenge. arXiv preprint arXiv:2012.12975. ↗
  39. Getachew Assefa Woldemariam. 2020. The Language policy of federal Ethiopia: A case for reform. J. Ethiopian L., 32:83. find
  40. Baye Yimam. 1999. The verb to say in Amharic. Journal of Ethiopian Studies, 32(1):1–50. find
  41. Seid Muhie Yimam, Abinew Ali Ayele, and Chris Biemann. 2019. Analysis of the Ethiopic Twitter dataset for abusive speech in Amharic. In In Proceedings of International Conference On Language Technologies For All: Enabling Linguistic Diversity And Multilingualism Worldwide (LT4ALL 2019), pages 210v–214, Paris, France. ↗
  42. Seid Muhie Yimam, Abinew Ali Ayele, Gopalakrishnan Venkatesh, Ibrahim Gashaw, and Chris Biemann. 2021. Introducing various semantic models for Amharic: Experimentation and evaluation with multiple tasks and datasets. Future Internet, 13(11). ↗
  43. Yanling Zhou, Yanyan Yang, Han Liu, Xiufeng Liu, and Nick Savage. 2020. Deep learning based fusion approach for hate speech detection. IEEE Access, 8(1):128923–128929. ↗
  44. Frederike Zufall, Marius Hamacher, Katharina Kloppenborg, and Torsten Zesch. 2022. A legal approach to hate speech – operationalizing the EU’s legal framework against the expression of hatred as an NLP task. In Proceedings of the Natural Legal Language Processing Workshop 2022, pages 53–64, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jigar-2024-detecting-hate-speech-amharic-multimodal/