Elamur Research⌕
☰
home / papers / 2023 / AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languages

AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languages

Summary

NLP research on African languages is scarce, largely because labelled data is lacking. This is especially true for sentiment in noisy social-media text, where code-switching, tone and multiple scripts complicate analysis. The authors address this gap with AfriSenti, a Twitter benchmark covering 14 African languages from four language families. Native speakers labelled each tweet as positive, negative or neutral. The collection combines ten newly annotated datasets with four existing ones that the team re-curated, and it was gathered using keywords and geographic location. Alongside the data, the authors release sentiment lexicons and describe the practical difficulties of collecting and annotating each language. They report baselines with several multilingual PLMs in per-language, combined multilingual, and zero-shot cross-lingual settings. Models pre-trained on African languages perform best, particularly at larger scale. The resource powered SemEval 2023 Task 12 and gives researchers a public foundation for building and evaluating sentiment systems in under-served languages.

Key points

  • AfriSenti contains more than 110,000 tweets in amh, arq, ary, hau, ibo, kin, pt-MZ, pcm, orm, swa, tir, twi, tso and yor. Six languages are Afro-Asiatic, six are Niger-Congo, one is an English Creole and one is Indo-European.
  • In terms of writing systems, ten languages use Latin script, while Arabic script and Ethiopic/Geez script account for two each. Algerian and Moroccan Arabic appear in both Arabic and Latin characters on social media.
  • The data was used in AfriSenti-SemEval (SemEval 2023 Task 12), the first Afrocentric SemEval shared task, which drew more than 200 participants. Datasets and lexicons are released on GitHub, but commercial use and use by state actors is restricted.
  • In multilingual training on all languages combined, AfroXLMR-large achieves the top average F1 of 71.2. It beats AfroXLMR-base, XLM-R-large and XLM-T-base by over 2.5 points.
  • For the test-only languages orm and tir, Hausa is the strongest zero-shot source (about 47.1 and 68.6 F1). It even surpasses a multilingually trained model, and Yorùbá also transfers well to tir.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

104 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 100 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.emnlp-main.862

@inproceedings{muhammad2023afrisenti,
  title = {{AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languages}},
  author = {Muhammad, Shamsuddeen Hassan and Abdulmumin, Idris and Ayele, Abinew Ali and Ousidhoum, Nedjma and Adelani, David Ifeoluwa and Yimam, Seid Muhie and Ahmad, Ibrahim Sa'id and Beloucif, Meriem and Mohammad, Saif M. and Ruder, Sebastian and Hourrane, Oumaima and Brazdil, Pavel and Jorge, Alipio and Ali, Felermino Dário Mário António and David, Davis and Osei, Salomey and Bello, Bello Shehu and Ibrahim, Falalu and Gwadabe, Tajuddeen and Rutunda, Samuel and Belay, Tadesse and Messelle, Wendimu Baye and Balcha, Hailu Beshada and Chala, Sisay Adugna and Gebremichael, Hagos Tesfahun and Opoku, Bernard and Arthur, Stephen},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.862},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.862}
}

References

66 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ife Adebara and Muhammad Abdul-Mageed. 2022. Towards afrocentric NLP for African languages: Where we are and where we can go. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3814–3841, Dublin, Ireland. Association for Computat… find
  2. David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D’souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, Stephen Mayhew, Israel Abebe Azime, Shamsuddeen H. Muhammad, Chris Chinenye Emezue, Joyce Nakatumba-Nabende, Perez Ogayo, Are… find our card
  3. David Ifeoluwa Adelani, Graham Neubig, Sebastian Ruder, Shruti Rijhwani, Michael Beukman, Chester Palen-Michel, Constantine Lignos, Jesujoba O. Alabi, Shamsuddeen H. Muhammad, Peter Nabende, Cheikh M. Bamba Dione, Andiswa Bukula, Rooweither Mabuya, Bonaventure F. P. Dossou, Blessing Sibanda, Happy B… find our card
  4. Basant Agarwal and Namita Mittal. 2016. Machine learning approach for sentiment analysis. In Prominent feature extraction for sentiment analysis, pages 21–45. Springer. find
  5. Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, and Dietrich Klakow. 2022. Adapting pre-trained language models to African languages via multilingual adaptive fine-tuning. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4336–4349, Gyeongju, Republic… find our card
  6. Nana Aba Appiah Amfo and Jemima Anderson. 2019. Multilingualism and language policies in the african context: lessons from ghana. find
  7. Jason Angel, Segun Taofeek Aroyehun, Antonio Tamayo, and Alexander Gelbukh. 2020. NLP-CIC at SemEval-2020 task 9: Analysing sentiment in code-switching language using a simple deep-learning classifier. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 957–962, Barcelona (online… find
  8. JoAnne Banks-Wallace. 2002. Talk that talk: Storytelling and analysis rooted in african american oral tradition. Qualitative Health Research, 12(3):410– 426. PMID: 11918105. find
  9. Francesco Barbieri, Luis Espinosa Anke, and Jose Camacho-Collados. 2022a. Xlm-t: Multilingual language models in twitter for sentiment analysis and beyond. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 258– 266. find our card
  10. Francesco Barbieri, Luis Espinosa Anke, and Jose Camacho-Collados. 2022b. XLM-T: Multilingual language models in Twitter for sentiment analysis and beyond. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 258–266, Marseille, France. European Language Resources Ass… find our card
  11. Tadesse Destaw Belay, Abinew Ali Ayele, Getie Gelaye, Seid Muhie Yimam, and Chris Biemann. 2021. Impacts of homophone normalization on semantic models for amharic. In 2021 International Conference on Information and Communication Technology for Development for Africa (ICT4DA), pages 101–106. find
  12. Erik Cambria, Qian Liu, Sergio Decherchi, Frank Xing, and Kenneth Kwok. 2022. Senticnet 7: a commonsense-based neurosymbolic ai framework for explainable sentiment analysis. Proceedings of LREC 2022. find
  13. Jan Carter-Black. 2007. Teaching cultural competence: An innovative strategy grounded in the universality of storytelling as depicted in african and african american storytelling traditions. Journal of Social Work Education, 43(1):31–50. find
  14. Quintin Gee. 2005. Review of script displays of african languages by current software. New Review of Hypermedia and Multimedia, 11:247 – 255. find
  15. Isaac Caswell, Theresa Breiner, Daan van Esch, and Ankur Bapna. 2020. Language ID in the wild: Unexpected challenges on the path to a thousand-language web text corpus. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6588–6608, Barcelona, Spain (Online). Inter… find our card
  16. Itisha Gupta and Nisheeth Joshi. 2020. Enhanced twitter sentiment analysis using hybrid approach and by accounting local contextual semantic. Journal of intelligent systems, 29(1):1611–1625. find
  17. Chenhua Chen, Zhiyang Teng, Zhongqing Wang, and Yue Zhang. 2022. Discrete opinion tree induction for aspect-based sentiment analysis. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2051–2064. find
  18. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  19. Aida Mostafazadeh Davani, Mark D’iaz, and Vinodkumar Prabhakaran. 2021. Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Transactions of the Association for Computational Linguistics, 10:92–110. find our card
  20. Peter Sheridan Dodds, Eric M Clark, Suma Desu, Morgan R Frank, Andrew J Reagan, Jake Ryland Williams, Lewis Mitchell, Kameron Decker Harris, Isabel M Kloumann, James P Bagrow, et al. 2015. Human language reveals a universal positivity bias. Proceedings of the national academy of sciences, 112(8):238… find
  21. David M. Eberhard, Gary F. Simons, and Charles D. Fennig. 2020. Ethnologue: Languages of the World. Twenty-third edition. Dallas, Texas: SIL International. Url: http://www.ethnologue.com. ↗
  22. Abdeljalil El Abdouli, Larbi Hassouni, and Houda Anoun. 2017. Sentiment analysis of moroccan tweets using naive bayes algorithm. International Journal of Computer Science and Information Security (IJCSIS), 15(12). find
  23. Chris Chinenye Emezue, Hellina Hailu Nigatu, Cynthia Thinwa, Helper Zhou, Shamsuddeen Hassan Muhammad, Lerato Louis, Idris Abdulmumin, Samuel Gbenga Oyerinde, Benjamin Ayoade Ajibade, Olanrewaju Samuel, et al. 2022. The african stopwords project: Curating stopwords for african languages. In 3rd Work… find
  24. Rosa Falotico and Piero Quatto. 2015. Fleiss’ kappa statistic without paradoxes. Quality & Quantity, 49(2):463–470. find
  25. William L Hamilton, Jure Leskovec, and Dan Jurafsky. 2016. Diachronic word embeddings reveal statistical laws of semantic change. arXiv preprint arXiv:1605.09096. find our card
  26. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. find our card
  27. Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. 2020. SMART: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization. In Proceedings of the 58th Annual Meeting of the Association for Computational Lingui… find our card
  28. Dr Kaur et al. 2022. Incorporating sentimental analysis into development of a hybrid classification model: A comprehensive study. International Journal of Health Sciences, 6:1709–1720. find
  29. C Maria Keet. 2021. Natural language generation requirements for social robots in sub-saharan africa. In 2021 IST-Africa Conference (IST-Africa), pages 1–8. IEEE. find
  30. Hoai Bac Le and Huy Nguyen. 2020. Twitter sentiment analysis using machine learning techniques. In International Conference on Computer Science, Applied Mathematics and Applications. find
  31. Paul M. A. Lewis. 2009. Ethnologue : languages of the world. find
  32. Wei Liang, Yuming Shi, Chi K. Tse, Jing Liu, Yanli Wang, and Xunqiang Cui. 2009. Comparison of co-occurrence networks of the chinese and english languages. Physica A: Statistical Mechanics and its Applications, 388(23):4901–4909. find
  33. Yunlong Liang, Fandong Meng, Jinan Xu, Yufeng Chen, and Jie Zhou. 2022. Msctd: A multimodal sentiment chat translation dataset. arXiv preprint arXiv:2202.13645. find
  34. Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, and Graham Neubig. 2019. Choosing transfer languages for cross-lingual learning. In Proceedings of the 57th Annual Meeting of… find our card
  35. Bing Liu. 2020. Sentiment analysis: Mining opinions, sentiments, and emotions. Cambridge university press. find
  36. Gati L Martin, Medard E Mswahili, and Young-Seob Jeong. 2021. Sentiment classification in swahili language using multilingual bert. arXiv preprint arXiv:2104.09006. — [arXiv:2104.09006](https://arxiv.org/abs/2104.09006) find
  37. M’hamed Mataoui, Omar Zelmati, and Madiha Boumechache. 2016. A proposed lexicon-based sentiment analysis approach for the vernacular algerian arabic. Research in Computing Science, 110(1):55– 70. find
  38. Granville J Matheson. 2019. We need to talk about reliability: making better use of test-retest studies for study design and interpretation. PeerJ, 7:e6918. find
  39. Million Meshesha and C. V. Jawahar. 2008. Indigenous scripts of african languages. Indilinga: African Journal of Indigenous Knowledge Systems, 6:132–142. find
  40. Saif Mohammad. 2016. A practical guide to sentiment annotation: Challenges and solutions. In Proceedings of the 7th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 174–179, San Diego, California. Association for Computational Linguistics. find
  41. Saif M. Mohammad. 2022. Ethics sheet for automatic emotion recognition and sentiment analysis. Computational Linguistics, 48(2):239–278. find
  42. Saif M. Mohammad. 2023. Best practices in the creation and use of emotion lexicons. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, Dubrovnik, Croatia. Association for Computational Linguistics. find
  43. Saif M. Mohammad, Svetlana Kiritchenko, and Xiao-Dan Zhu. 2013. Nrc-canada: Building the state-ofthe-art in sentiment analysis of tweets. In International Workshop on Semantic Evaluation. find
  44. Leila Moudjari, Karima Akli-Astouati, and Farah Benamara. 2020. An Algerian corpus and an annotation platform for opinion and emotion analysis. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1202–1210, Marseille, France. European Language Resources Association. find
  45. Shamsuddeen Hassan Muhammad, Idris Abdulmumin, Seid Muhie Yimam, David Ifeoluwa Adelani, Ibrahim Said Ahmad, Nedjma Ousidhoum, Abinew Ali Ayele, Saif Mohammad, Meriem Beloucif, and Sebastian Ruder. 2023. SemEval-2023 task 12: Sentiment analysis for African languages (AfriSenti-SemEval). In Proceedin… find
  46. Shamsuddeen Hassan Muhammad, David Ifeoluwa Adelani, Sebastian Ruder, Ibrahim Sa’id Ahmad, Idris Abdulmumin, Bello Shehu Bello, Monojit Choudhury, Chris Chinenye Emezue, Saheed Salahudeen Abdullahi, Anuoluwapo Aremu, Alípio Jorge, and Pavel Brazdil. 2022. NaijaSenti: A Nigerian Twitter sentiment cor… find our card
  47. Kelechi Ogueji, Yuxin Zhu, and Jimmy Lin. 2021a. Small data? no problem! exploring the viability of pretrained multilingual language models for low-resourced languages. In Proceedings of the 1st Workshop on Multilingual Representation Learning, pages 116–126, Punta Cana, Dominican Republic. Associat… find our card
  48. Kelechi Ogueji, Yuxin Zhu, and Jimmy J. Lin. 2021b. Small data? no problem! exploring the viability of pretrained multilingual language models for low-resourced languages. Proceedings of the 1st Workshop on Multilingual Representation Learning. find our card
  49. Aissam Outchakoucht and Hamza Es-Samaali. 2021. Moroccan dialect -darija- open dataset. find our card
  50. Tal Perry. 2021. LightTag: Text annotation platform. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 20–27, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  51. Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, and Rada Mihalcea. 2020. Beneath the tip of the iceberg: Current challenges and new directions in sentiment analysis research. IEEE Transactions on Affective Computing. find
  52. Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark D’iaz. 2021. On releasing annotator-level labels and information in datasets. ArXiv, abs/2110.05699. find our card
  53. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1–67. find
  54. Justus J Randolph. 2005. Free-marginal multirater kappa (multirater k [free]): An alternative to fleiss’ fixed-marginal multirater kappa. Online submission. find
  55. Andrew J. Reagan, Lewis Mitchell, Dilan Kiley, Christopher M. Danforth, and Peter S. Dodds. 2016. The emotional arcs of stories are dominated by six basic shapes. EPJ Data Science, 5(1):1–12. Copyright -EPJ Data Science is a copyright of Springer, 2016; Last updated - 2017-02-06. find
  56. Sebastin Santy, Anirudh Srinivasan, and Monojit Choudhury. 2021. BERTologiCoMix: How does code-mixing interact with multilingual BERT? In Proceedings of the Second Workshop on Domain Adaptation for NLP, pages 111–121, Kyiv, Ukraine. Association for Computational Linguistics. find
  57. Kevin P Scannell. 2011. Statistical unicodification of african languages. Language resources and evaluation, 45(3):375–386. find
  58. Laura Sterponi and Paul F. Lai. 2014. Culture and language development. In Farzad Sharifian, editor, The Routledge Handbook of Language and Culture, pages 339–356. Routledge, London, UK. find
  59. Maite Taboada, Julian Brooke, Milan Tofiloski, Kimberly D. Voll, and Manfred Stede. 2011. Lexicon-based methods for sentiment analysis. Computational Linguistics, 37:267–307. find our card
  60. S Thara and Prabaharan Poornachandran. 2018. Code-mixing: A brief survey. In 2018 International Conference on Advances in Computing, Communications and Informatics (ICACCI), pages 2382–2388. find
  61. Peter D. Turney. 2002. Thumbs up or thumbs down? semantic orientation applied to unsupervised classification of reviews. In Annual Meeting of the Association for Computational Linguistics. find our card
  62. Ashima Yadav and Dinesh Kumar Vishwakarma. 2020. Sentiment analysis using deep learning architectures: a review. Artificial Intelligence Review, 53(6):4335– 4385. find
  63. Seid Muhie Yimam, Hizkiel Mitiku Alemayehu, Abinew Ali Ayele, and Chris Biemann. 2020. Exploring amharic sentiment analysis from social media texts: Building annotation tools and classification models. In International Conference on Computational Linguistics. find
  64. Seid Muhie Yimam, Abinew Ali Ayele, and Chris Biemann. 2019. Analysis of the Ethiopic Twitter Dataset for Abusive Speech in Amharic. In In Proceedings of International Conference On Language Technologies For All: Enabling Linguistic Diversity And Multilingualism Worldwide (LT4ALL 2019), pages 1–5, P… find
  65. Seid Muhie Yimam, Iryna Gurevych, Richard Eckart de Castilho, and Chris Biemann. 2013. WebAnno: A flexible, web-based and visually supported system for distributed annotations. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 1–… find our card
  66. Lei Zhang, Shuai Wang, and Bing Liu. 2018. Deep learning for sentiment analysis: A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8(4):e1253. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/muhammad-2023-afrisenti-twitter-sentiment-analysis-benchmark/