Elamur Research⌕
☰
home / papers / 2022 / NaijaSenti: A Nigerian Twitter Sentiment Corpus for Multilingual Sentiment Analysis

NaijaSenti: A Nigerian Twitter Sentiment Corpus for Multilingual Sentiment Analysis

Summary

Sentiment analysis tools are mostly built for data-rich languages, so Nigerian languages have had little support. This paper presents NaijaSenti, which the authors describe as the first large, human-labelled Twitter sentiment corpus for Hausa, Igbo, Yorùbá and Nigerian Pidgin (Naija). The abstract puts it at roughly 30,000 tweets per language, and many of them are code-mixed.

The Twitter API cannot filter by these languages, so the authors built their own collection pipeline. It combines:

  • query terms checked by native speakers, paired with emojis,
  • location-restricted searches,
  • hashtags and handles of local-language news accounts,
  • automatic language identification,
  • pre-filtering with a machine-translated sentiment lexicon.

Native speakers then labelled the tweets using a multi-class scheme. The authors benchmarked several pre-trained multilingual models and transfer strategies. Models that had seen African-language text, such as AfriBERTa, or that went through language-adaptive fine-tuning, generally performed best.

The datasets, models, lexicons and code are all released. This gives a reusable base for sentiment research and other NLP work in under-represented languages.

Key points

  • The corpus covers four languages: Hausa (hau), Igbo (ibo), Nigerian Pidgin (pcm) and Yorùbá (yor). Code-switching is common; about 43% of Igbo tweets mix Igbo with English.
  • Of six earlier sentiment datasets for these languages, only two were openly available. Several of them relied on translation rather than native annotation.
  • The annotation guide is adapted from Mohammad (2016). It defines five classes, including POS, NEG, NEU and MIX (the fifth class name is cut off in the provided text).
  • Tweets were cleaned before annotation:
    • retweets and duplicates were dropped,
    • URLs and mentions were stripped,
    • text was lowercased,
    • tweets shorter than three words were discarded.
  • Fine-tuning used HuggingFace with batch size 32, max sequence length 128 and 20 epochs. The learning rate was 5e-5 for most models and 2e-5 for XLM-R and RemBERT so they would converge. The release also includes manually built sentiment lexicons for Hausa, Igbo and Yorùbá.

Interesting ideas

  • I1. Geographic radius instead of an unsupported language filter. The authors restricted searches to circles of about 25 miles around areas where each language dominates. This helps whenever an API has no language parameter for your target language.
  • I2. Watch for shared stopwords across languages. Common words can belong to several neighbouring languages; for example, querying "nke" for Igbo returned Hausa tweets. Seed-word queries therefore need a separate language-ID pass afterwards. For Pidgin, which off-the-shelf detectors do not support, the authors built a simple n-gram detector from the stopword list.
  • I3. Pre-filter with a cheap translated lexicon before paying annotators. Machine-translating the Affin lexicon and keeping only tweets that contain its terms raises the share of sentiment-bearing items sent for annotation. Combining stopwords with happy or sad emojis serves the same purpose.
  • I4. Cap tweets per conversation. In the Hausa handle-based data, a single @bbchausa thread made up 54% of the tweets. Limiting how many tweets one conversation can contribute keeps a few popular threads from skewing the corpus by topic.
  • I5. Adapt the encoder to the language before fine-tuning. Language-adaptive fine-tuning, or choosing models pre-trained on African-language text, gave the strongest results. This suggests that a short pass of continued pre-training on unlabelled target-language text is a sensible default for any low-resource classification task.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

69 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 67 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{muhammad2022naijasenti,
  title = {{NaijaSenti: A Nigerian Twitter Sentiment Corpus for Multilingual Sentiment Analysis}},
  author = {Muhammad, Shamsuddeen Hassan and Adelani, David Ifeoluwa and Ruder, Sebastian and Ahmad, Ibrahim Sa'id and Abdulmumin, Idris and Bello, Bello Shehu and Choudhury, Monojit and Emezue, Chris Chinenye and Abdullahi, Saheed Salahudeen and Aremu, Anuoluwapo and Jorge, Alípio and Brazdil, Pavel},
  booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation Conference},
  year = {2022},
  doi = {10.63317/2anx3vqhsxdm},
  url = {https://doi.org/10.63317/2anx3vqhsxdm}
}

References

70 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abubakar, A. I., Roko, A., Bui, A. M., and Saidu, I. (2021). An enhanced feature acquisition for sentiment analysis of english and hausa tweets. International Journal of Advanced Computer Science and Applications, 12(9). find
  2. Adelani, D. I., Abbott, J., Neubig, G., D’souza, D., Kreutzer, J., Lignos, C., Palen-Michel, C., Buzaaba, H., Rijhwani, S., Ruder, S., Mayhew, S., Azime, I. A., Muhammad, S. H., Emezue, C. C., Nakatumba-Nabende, J., Ogayo, P., Anuoluwapo, A., Gitau, C., Mbaye, D., Alabi, J., Yimam, S. M., Gwadabe, T… find our card
  3. Al-Twairesh, N., Al-Khalifa, H., Al-Salman, A., and Al-Ohali, Y. (2017). Arasenti-tweet: A corpus for arabic sentiment analysis of saudi tweets. Procedia Computer Science, 117:63–72. find
  4. Alabi, J., Amponsah-Kaakyire, K., Adelani, D., and España-Bonet, C. (2020). Massive vs. curated embeddings for low-resourced languages: the case of Yorùbá and Twi. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 2754–2762, Marseille, France, May. European Language Reso… find
  5. Beddor, B. (2019). Subjective Disagreement. Noûs, 53(4):819–851, December. find
  6. Brum, H. B. and Nunes, M. d. G. V. (2017). Building a sentiment corpus of tweets in brazilian portuguese. arXiv preprint arXiv:1712.08917. find
  7. Caswell, I., Breiner, T., van Esch, D., and Bapna, A. (2020). Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6588–6608, Barcelona, Spain (Online). International… find our card
  8. Chau, E. C., Lin, L. H., and Smith, N. A. (2020). Parsing with multilingual BERT, a small corpus, and a small treebank. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1324–1334, Online, November. Association for Computational Linguistics. find
  9. Chung, H. W., Fevry, T., Tsai, H., Johnson, M., and Ruder, S. (2021). Rethinking embedding coupling in pre-trained language models. In International Conference on Learning Representations. find our card
  10. Clark, K., Luong, M.-T., Le, Q. V., and Manning, C. D. (2020). ELECTRA: Pre-training text encoders as discriminators rather than generators. In ICLR. find
  11. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. (2020). Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages… find
  12. Davani, A. M., Díaz, M., and Prabhakaran, V. (2021). Dealing with disagreements: Looking beyond the majority vote in subjective annotations. arXiv preprint arXiv:2110.05719. find our card
  13. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (… find
  14. Eberhard, D. M., Simons, G. F., and (eds.), C. D. F. (2022). Ethnologue: Languages of the world. twenty-third edition. find
  15. Fleiss, J. L., Levin, B., and Paik, M. C. (2013). Statistical methods for rates and proportions. john wiley & sons. find
  16. ∀, ., Nekoto, W., Marivate, V., Matsila, T., Fasubaa, T., Fagbohungbe, T., Akinola, S. O., Muhammad, S., Kabongo Kabenamualu, S., Osei, S., Sackey, F., Niyongabo, R. A., Macharm, R., Ogayo, P., Ahia, O., Berhe, M. M., Adeyemi, M., Mokgesi-Selinga, M., Okegbemi, L., Martinus, L., Tajudeen, K., Degila… find our card
  17. Fornaciari, T., Uma, A., Paun, S., Plank, B., Hovy, D., and Poesio, M. (2021). Beyond black & white: Leveraging annotator disagreement via soft-label multi-task learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu… find
  18. Go, A., Bhayani, R., and Huang, L. (2009). Twitter sentiment classification using distant supervision. CS224N project report, Stanford, 1(12):2009. find
  19. Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., and Smith, N. A. (2020). Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342–8360, Online, July. As… find
  20. He, P., Gao, J., and Chen, W. (2021). Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. ArXiv, abs/2111.09543. find our card
  21. Honkanen, M. and Müller, J. (2021). Interjections and emojis in nigerian online communication. World Englishes. find
  22. Howard, J. and Ruder, S. (2018). Universal Language Model Fine-tuning for Text Classification. In Proceedings of ACL 2018. find
  23. Hu, M. and Liu, B. (2004). Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 168–177. find
  24. Iyanda, A. R. and Abegunde, O. (2019). Predicting sentiment in yorùbá written texts: A comparison of machine learning models. In Proceedings of SAI Intelligent Systems Conference, pages 416–431. Springer. find
  25. Jaggar, P. (2001). Hausa. London Oriental and African language library. John Benjamins Publishing Company. find
  26. Joshi, P., Santy, S., Budhiraja, A., Bali, K., and Choudhury, M. (2020). The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282– 6293, Online, July. Association for Computationa… find our card
  27. Jovanoski, D., Pachovski, V., and Nakov, P. (2021). Sentiment analysis in twitter for macedonian. arXiv preprint arXiv:2109.13725. — [arXiv:2109.13725](https://arxiv.org/abs/2109.13725) find
  28. Jurman, G., Riccadonna, S., and Furlanello, C. (2012). A comparison of mcc and cen error measures in multi-class prediction. PLOS ONE, 7(8):1–8, 08. find
  29. Kolajo, T., Daramola, O., and Adebiyi, A. (2019). Sentiment analysis on naija-tweets. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 338–343. find
  30. Kralj Novak, P., Smailović, J., Sluban, B., and Mozetič, I. (2015). Sentiment of emojis. PLoS ONE, 10(12):e0144296. find
  31. Kwaik, K. A., Chatzikyriakidis, S., Dobnik, S., Saad, M., and Johansson, R. (2020). An arabic tweets sentiment analysis dataset (atsad) using distant supervision and self training. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive… find
  32. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692. find
  33. Mohammad, S. M. and Turney, P. D. (2013). Crowdsourcing a word-emotion association lexicon. Computational Intelligence, 29(3):436–465. find
  34. Mohammad, S. (2016). A Practical Guide to Sentiment Annotation: Challenges and Solutions. In Proceedings of the 7th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 174–179, San Diego, California. Association for Computational Linguistics. find
  35. Moudjari, L., Akli-Astouati, K., and Benamara, F. (2020). An algerian corpus and an annotation platform for opinion and emotion analysis. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 1202–1210. find
  36. Mozetič, I., Grčar, M., and Smailović, J. (2016). Multilingual twitter sentiment classification: The role of human annotators. PloS one, 11(5):e0155036. find
  37. Muller, B., Anastasopoulos, A., Sagot, B., and Seddah, D. (2021). When being unseen from mBERT is just the beginning: Handling new languages with multilingual language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human… find our card
  38. Nakov, P., Kozareva, Z., Ritter, A., Rosenthal, S., Stoyanov, V., and Wilson, T. (2019). Semeval-2013 task 2: Sentiment analysis in twitter. find
  39. Nangia, N. and Bowman, S. R. (2019). Human vs. muppet: A conservative estimate of human performance on the glue benchmark. arXiv preprint arXiv:1905.10425. find
  40. Nasim, Z. and Ghani, S. (2020). Sentiment analysis on urdu tweets using markov chains. SN Computer Science, 1(5):1–13. find
  41. Nwofe, E. S. (2017). Pro-biafran activists and the call for a referendum: A sentiment analysis of ‘bi-afraexit’on twitter after uk’s vote to leave the european union. Journal of Ethnic and Cultural Studies, 4(1):65. find
  42. Ogbuju, E. and Onyesolu, M. (2019). Development of a general purpose sentiment lexicon for Igbo language. In Proceedings of the 2019 Workshop on Widening NLP, page 1, Florence, Italy, August. Association for Computational Linguistics. find
  43. Ogueji, K., Zhu, Y., and Lin, J. (2021). Small data? no problem! exploring the viability of pretrained multilingual language models for low-resourced languages. In Proceedings of the 1st Workshop on Multilingual Representation Learning, pages 116–126, Punta Cana, Dominican Republic, November. Associ… find our card
  44. Ohiri-Aniche, C. (2007). Stemming the tide of centrifugal forces in Igbo orthography. Dialectical Anthropology, 31(4):423–436. find
  45. Olagunju, T., Oyebode, O., and Orji, R. (2020). Exploring key issues affecting african mobile ecommerce applications using sentiment and thematic analysis. IEEE Access, 8:114475–114486. find
  46. Olaleye, S. A., Sanusi, I. T., and Salo, J. (2018). Sentiment analysis of social commerce: a harbinger of online reputation management. International Journal of Electronic Business, 14(2):85–102. find
  47. Onyenwe, I., Nwagbo, S., Mbeledogu, N., and Onyedinma, E. (2020). The impact of political party/candidate on the election results from a sentiment analysis perspective using\# anambrade-cides2017 tweets. Social Network Analysis and Mining, 10(1):1–17. find
  48. Orimaye, S. O., Alhashmi, S. M., and Eu-gene, S. (2012). Sentiment analysis amidst ambiguities in youtube comments on yoruba language (nollywood) movies. In Proceedings of the 21st International Conference on World Wide Web, pages 583–584. find
  49. Oyebode, O. and Orji, R. (2019). Social media and sentiment analysis: The nigeria presidential election 2019. In 2019 IEEE 10th Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON), pages 0140–0146. IEEE. find
  50. Oyewusi, W. F., Adekanmbi, O., and Akinsande, O. (2020). Semantic enrichment of nigerian pidgin english for contextual sentiment classification. arXiv preprint arXiv:2003.12450. find
  51. Pak, A. and Paroubek, P. (2010). Twitter as a corpus for sentiment analysis and opinion mining. In LREC, volume 10, pages 1320–1326. find our card
  52. Pang, B. and Lee, L. (2007). Opinion mining and sentiment analysis. Found. Trends Inf. Retr., 2:1–135. find
  53. Perry, T. (2021). LightTag: Text Annotation Platform. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 20–27, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  54. Pfeiffer, J., Vuli, I., Gurevych, I., and Ruder, S. (2020). MAD-X: An Adapter-based Framework for Multi-task Cross-lingual Transfer. In Proceedings of EMNLP 2020. find our card
  55. Pfeiffer, J., Vulić, I., Gurevych, I., and Ruder, S. (2021). UNKs everywhere: Adapting multilingual language models to new scripts. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10186–10203, Online and Punta Cana, Dominican Republic, November. Asso… find
  56. Potts, C., Wu, Z., Geiger, A., and Kiela, D. (2021). DynaSent: A dynamic benchmark for sentiment analysis. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pa… find
  57. Rakhmanov, O. (2020). A comparative study on vectorization and classification techniques in sentiment analysis to classify student-lecturer comments. Procedia Computer Science, 178:194–204. find
  58. Refaee, E. and Rieser, V. (2014a). An arabic twitter corpus for subjectivity and sentiment analysis. In LREC, pages 2268–2273. find
  59. Refaee, E. and Rieser, V. (2014b). An Arabic Twitter corpus for subjectivity and sentiment analysis. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 2268–2273, Reykjavik, Iceland, May. European Language Resources Association (ELRA). find
  60. Rosenthal, S., Farra, N., and Nakov, P. (2017). Semeval-2017 task 4: Sentiment analysis in twitter. In Proceedings of the 11th international workshop on semantic evaluation (SemEval-2017), pages 502– 518. find our card
  61. Umoh, U., Eyoh, I., Isong, E., Ekong, A., and Peter, S. (2020). Using interval type-2 fuzzy logic to analyze igbo emotion words. Journal of Fuzzy Extension and Applications, 1(3):217–240. find
  62. UzZaman, N., Llorens, H., Derczynski, L., Allen, J., Verhagen, M., and Pustejovsky, J. (2013). Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations. In Second Joint Conference on Lexical and Computational Semantics (* SEM), Volume 2: Proceedings of the Seventh… find
  63. Warstadt, A., Singh, A., and Bowman, S. R. (2019). Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641. find
  64. Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. (2021). Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359. find our card
  65. Wicaksono, A. F., Vania, C., Distiawan, B., and Adriani, M. (2014). Automatically building a corpus for sentiment analysis on indonesian tweets. In Proceedings of the 28th Pacific Asia Conference on Language, Information and Computing, pages 185–194. find
  66. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. (2020). Transformers: State-of-the-art nat… find our card
  67. Xia, M., Zheng, G., Mukherjee, S., Shokouhi, M., Neubig, G., and Awadallah, A. H. (2021). Metaxl: Meta representation transformation for low-resource cross-lingual learning. arXiv preprint arXiv:2104.07908. find
  68. Yang, X., Macdonald, C., and Ounis, I. (2018). Using word embeddings in twitter election classification. Information Retrieval Journal, 21(2):183–207. find
  69. Yimam, S. M., Alemayehu, H. M., Ayele, A., and Biemann, C. (2020). Exploring amharic sentiment analysis from social media texts: Building annotation tools and classification models. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1048–1060. find
  70. Årup Nielsen, F. (2011). A new anew: Evaluation of a word list for sentiment analysis in microblogs. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/muhammad-2022-naijasenti-nigerian-twitter-sentiment-corpus/