Elamur Research⌕
☰
home / papers / 2023 / Exploring Amharic Hate Speech Data Collection and Classification Approaches

Exploring Amharic Hate Speech Data Collection and Classification Approaches

Paper · 2023 · Abinew Ali Ayele, Seid Muhie Yimam, Tadesse Destaw Belay et al.

Summary

Hate speech detectors built for English do not carry over to Amharic. Earlier Amharic work mostly used binary labels on Facebook comments and seldom released data or models. The authors start from a Twitter archive of more than 18 million Fidäl-script tweets that they have collected since 2014. They select 3.8 million tweets posted during a turbulent period in Ethiopia (October 2020 to November 2021) and pass them through a multi-step pipeline: retweet removal, language identification, filtering with a community-sourced hate/offensive lexicon, anonymization and near-duplicate removal. Two native speakers labelled 15.1k tweets as hate, offensive, normal or unsure, and a curator settled disagreements. The team then compared classical classifiers with deep models, including fine-tuned AmFLAIR and AmRoBERTa. AmFLAIR performed best at 72% F1, a clear gain over a previous crowdsourced Amharic dataset that reached only 50%. A manual error analysis shows that much of the remaining difficulty comes from context, sarcasm and annotation bias rather than from the models alone. The guidelines, lexicon, data, models and scripts are released openly.

Key points

  • Data funnel: 3.8M tweets shrank to 902k after dropping retweets and non-Amharic text (langdetect), then to 153k after lexicon filtering. Shingling with a 25% Jaccard cutoff left 130k unique tweets, which means about 33% were near-duplicates.
  • Deleted content: About 12% of archived tweets were no longer available on Twitter, and roughly 9% came from suspended accounts. Samples of these were annotated in a pilot study.
  • Annotation: 15.1k tweets were labelled with four classes (hate, offensive, normal, unsure). Two annotators reached a Cohen's kappa of 0.48, and a curator decided the gold labels. The earlier crowdsourced Toloka dataset had a kappa of only 0.34.
  • Models: AmFLAIR achieved the top F1 of 72%. Both contextual embedding models (AmFLAIR, AmRoBERTa) beat the classical baselines.
  • Error audit: 63.6% of errors were the model's fault, 28.8% were annotator mistakes, and 7.6% could not be judged. The leading causes were annotation bias (29.7%), sarcasm (13.6%), keyword association (11%) and missing context (8.5%).

Interesting ideas

  • I1. The authors kept two separate unlabelled pools, one with lexicon keywords and one without, so the sample does not only reflect what the seed lexicon can find; this guards against a classifier that just learns the keywords.
  • I2. They removed near-duplicates with shingling plus a Jaccard threshold, because copy-and-tweak reposts would otherwise inflate the corpus and leak between splits.
  • I3. In their audit, experts assigned each wrong prediction to the model, to the annotator, or to an "undecidable" bucket, which showed that almost a third of the "errors" were really label errors.
  • I4. They added an explicit "unsure" class so that annotators could flag items that lack background context instead of guessing.
  • I5. Sarcasm and irony turned out to be a major error source, and the paper notes they confused even human annotators, so they deserve dedicated handling rather than being left to a general classifier.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver data and evaluation sets could mix keyword-selected and keyword-free samples, so that negative controls reveal whether a component depends on surface triggers.
  • I2 (learn): Applying the same dedup step before building silver treebanks or train/test splits would stop near-identical sentences from inflating parser and reader scores.
  • I3 (learn): Disagreements between Mova and gold data could be triaged the same way, separating gold-label bugs from real failures before repair rules are induced.
  • I4 (frames): Speech-act and implicature outputs could include an explicit "insufficient context" verdict instead of a forced label, which keeps every decision explainable.
  • I5 (frames): Mova's irony module could compare the literal polarity of a frame with the world state it describes, and so catch "positive-looking" texts that are actually ironic.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Universität HamburgHamburg · Germany79 works · 5,422 citations🎓Bahir Dar UniversityBahir Dar · Ethiopia11 works · 261 citations🎓Wollo UniversityDessie · Ethiopia2 works · 64 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 4Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 worksBias, fairness & safety · 517 worksSpeech & spoken language · 480 works

Cite

Cite as Abinew Ali Ayele, Seid Muhie Yimam, Tadesse Destaw Belay, Tesfa Asfaw and Chris Biemann. 2023. Exploring Amharic Hate Speech Data Collection and Classification Approaches. Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{ayele2023exploring,
  title = {{Exploring Amharic Hate Speech Data Collection and Classification Approaches}},
  author = {Abinew Ali Ayele, Seid Muhie Yimam, Tadesse Destaw Belay, Tesfa Asfaw and Chris Biemann},
  booktitle = {Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing},
  year = {2023},
  doi = {10.26615/978-954-452-092-2\_006},
  url = {https://doi.org/10.26615/978-954-452-092-2_006}
}

References

34 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Zeleke Abebaw, Andreas Rauber, and Solomon Atnafu. 2021. Multi-channel convolutional neural network for hate speech detection in social media. In International Conference on Advances of Science and Technology, pages 603–618, Bahir Dar, Ethiopia. Springer. find
  2. Fabio Del Vigna, Andrea Cimino, Felice Dell’Orletta, Marinella Petrocchi, and Maurizio Tesconi. 2017. Hate me, hate me not: Hate speech detection on facebook. In Proceedings of the First Italian Conference on Cybersecurity (ITASEC17), pages 86–95, Venice, Italy. ITASEC17. find
  3. Zeleke Abebaw, Andreas Rauber, and Solomon Atnafu. 2022. Design and implementation of a multichannel convolutional neural network for hate speech detection in social networks. Revue d’Intelligence Artificielle, 36(2):175–183. ↗
  4. Christoph Demus, Jonas Pitz, Mina Schütz, Nadine Probol, Melanie Siegel, and Dirk Labudde. 2022. Detox: A comprehensive dataset for German offensive language and conversation analysis. In Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH), pages 143–153, Seattle, WA, USA. Association for Computational Linguistics. find
  5. Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. FLAIR: An easy-to-use framework for state-of-the-art NLP. In NAACL 2019, 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 54–59, Minneapolis, MN, USA.
  6. Mai ElSherief, Vivek Kulkarni, Dana Nguyen, William Yang Wang, and Elizabeth Belding. 2018. Hate lingo: A target-based linguistic analysis of hate speech in social media. In Twelfth International AAAI Conference on Web and Social Media, pages 42–51, Palo Alto, CA, USA. ↗
  7. Nuha Albadi, Maram Kurdi, and Shivakant Mishra. 2018. Are they our brothers? analysis and detection of religious hate speech in the arabic twitter-Paula Fortuna, Monica Dominguez, Leo Wanner, and Zeerak Talat. 2022. Directions for NLP practices applied to online hate speech detection. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11794–11805, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  8. Zewdie Mossie and Jenq-Haur Wang. 2018. Social network hate speech detection for amharic language. In 4th International Conference on Natural Language Computing (NATL2018), pages 41–55, Dubai, United Arab Emirates. AIRCC Publishing. ↗
  9. Antigoni Maria Founta, Despoina Chatzakou, Nicolas Kourtellis, Jeremy Blackburn, Athena Vakali, and Ilias Leontiadis. 2019. A unified deep learning architecture for abuse detection. In Proceedings of the 10th ACM conference on web science, pages 105–114, New York City, NY, USA. Association for Computing Machinery. find
  10. Zewdie Mossie and Jenq-Haur Wang. 2020. Vulnerable community identification using hate speech detection on social media. Information Processing & Management, 57(3):1–16. ↗
  11. Antigoni Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large scale crowdsourcing and characterization of twitter abusive behavior. In Twelfth International AAAI Conference on Web and Social Media, pages 491–500, Palo Alto, CA, USA. ↗
  12. Hala Mulki, Hatem Haddad, Chedi Bechikh Ali, and Halima Alshabani. 2019. L-hsab: A levantine twitter dataset for hate speech and abusive language. In Proceedings of the third workshop on abusive language online, pages 111–118, Florence, Italy.
  13. John Paul Mueller and Luca Massaron. 2021. Machine learning for dummies. John Wiley & Sons. find
  14. Ray Oshikawa, Jing Qian, and William Yang Wang. 2018. A survey on natural language processing for fake news detection. arXiv preprint arXiv:1811.00770. ↗
  15. Andargachew Mekonnen Gezmu, Binyam Ephrem Seyoum, Michael Gasser, and Andreas Nürnberger. 2018. Contemporary amharic corpus: Automatically morpho-syntactically tagged amharic corpus. In Proceedings of the First Workshop on Linguistic Resources for Natural Language Processing, pages 65–70, Santa Fe, NM, USA. Association for Computational Linguistics.
  16. Nedjma Ousidhoum, Zizheng Lin, Hongming Zhang, Yangqiu Song, and Dit-Yan Yeung. 2019. Multilingual and multi-aspect hate speech analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4675– 4684, Hong Kong, China. Association for Computational Linguistics.
  17. Tobias Glasmachers. 2017. Limits of end-to-end learning. In Proceedings of the Ninth Asian Conference on Machine Learning, pages 17–32, Seoul, Korea. PMLR. ↗
  18. Fabio Poletto, Marco Stranisci, Manuela Sanguinetti, Viviana Patti, and Cristina Bosco. 2017. Hate speech annotation: Analysis of an italian twitter corpus. In 4th Italian Conference on Computational Linguistics, CLiC-it 2017, volume 2006, pages 1–6, Rome, Italy. CEUR-WS. find
  19. Tommi Gröndahl, Luca Pajola, Mika Juuti, Mauro Conti, and N Asokan. 2018. All you need is" love" evading hate speech detection. In Proceedings of the 11th ACM workshop on artificial intelligence and security, pages 2–12, Toronto, ON, Canada. Association for Computing Machinery. ↗
  20. Paul Röttger, Debora Nozza, Federico Bianchi, and Dirk Hovy. 2022a. Data-efficient strategies for expanding hate speech detection into under-resourced languages. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5674–5691, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  21. Simon Kemp. 2023. DIGITAL 2023: GLOBAL OVERVIEW REPORT. Technical report, DataReportal. Last accessed: July 16, 2023. find
  22. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  23. Paul Röttger, Haitham Seelawi, Debora Nozza, Zeerak Talat, and Bertie Vidgen. 2022b. Multilingual Hate-Check: Functional tests for multilingual hate speech detection models. In Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH), pages 154–169, Seattle, WA, USA. Association for Computational Linguistics. find
  24. Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. 2021. Hatexplain: A benchmark dataset for explainable hate speech detection. In Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence, pages 14867–14875, Palo Alto, CA, USA. Association for the Advancement of Artificial Intelligence. find
  25. Abiodun Salawu and Asemahagn Aseres. 2015. Language policy, ideologies, power, and the ethiopian media. South African Journal for Communication Theory and Research, 41(1):71–89. ↗
  26. Fenna Miedema. 2018. Sentiment analysis with long short-term memory networks. Vrije Universiteit Amsterdam, 1:1–17. find
  27. Surafel Getachew Tesfaye and Kula Kakeba. 2020. Automated amharic hate speech posts and comments detection model using recurrent neural network. Research Square, DOI: https://doi.org/10.21203/rs.3.rs-114533/v1, pages 1–14. ↗
  28. Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In Proceedings of the NAACL student research workshop, pages 88–93, San Diego, CA, USA. Association for Computational Linguistics.
  29. Matthew L Williams, Pete Burnap, Amir Javed, Han Liu, and Sefa Ozalp. 2020. Hate in the machine: Anti-black and anti-muslim social media posts as predictors of offline racially and religiously aggravated crime. The British Journal of Criminology, 60(1):93– 117. ↗
  30. Kevin Winter and Roman Kern. 2019. Know-center at SemEval-2019 task 5: Multilingual hate speech detection on Twitter using CNNs. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 431–435, Minneapolis, MN, USA. Association for Computational Linguistics. find
  31. Seid Muhie Yimam, Abinew Ali Ayele, and Chris Biemann. 2019. Analysis of the Ethiopic Twitter Dataset for Abusive Speech in Amharic. In In Proceedings of International Conference On Language Technologies For All: Enabling Linguistic Diversity And Multilingualism Worldwide (LT4ALL 2019), pages 210v–214, Paris, France. ↗
  32. Seid Muhie Yimam, Abinew Ali Ayele, Gopalakrishnan Venkatesh, Ibrahim Gashaw, and Chris Biemann. 2021. Introducing various semantic models for amharic: Experimentation and evaluation with multiple tasks and datasets. Future Internet, 13(11):1–18. ↗
  33. Seid Muhie Yimam, Iryna Gurevych, Richard Eckart de Castilho, and Chris Biemann. 2013. WebAnno: A flexible, web-based and visually supported system for distributed annotations. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 1–6, Sofia, Bulgaria. Association for Computational Linguistics.
  34. Tom Young, Devamanyu Hazarika, Soujanya Poria, and Erik Cambria. 2018. Recent trends in deep learning based natural language processing [review article]. IEEE Computational Intelligence Magazine, 13(3):55–75. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ayele-2023-exploring-amharic-hate-speech-data/