Elamur Research⌕
☰
home / papers / 2020 / Exploring Amharic Sentiment Analysis from Social Media Texts: Building Annotation Tools and Classification Models

Exploring Amharic Sentiment Analysis from Social Media Texts: Building Annotation Tools and Classification Models

Paper · 2020 · Seid Muhie Yimam, Hizkiel Mitiku Alemayehu, Abinew Ayele et al.

Summary

The paper addresses sentiment analysis for Amharic, a low-resource language. Amharic lacks annotated corpora and annotation infrastructure. Mainstream crowdsourcing services such as MTurk do not support Ethiopian requesters, and most local users work on phones over slow connections. After trying a spreadsheet and then a web annotator, the authors built ASAB, an annotation bot that runs inside Telegram. It has built-in quality checks and pays annotators automatically with mobile vouchers. From a large collection of Amharic tweets, they selected those containing sentiment-lexicon words, and each selected tweet received labels from three annotators. On this data they compared classical supervised classifiers with deep FLAIR-based models. The best system was a FLAIR classifier using network embeddings computed over a distributional thesaurus. Error analysis shows that sarcasm and idiomatic, figurative language are the main obstacles. Some errors were also noise from annotators. The dataset, lexicon, bot, code and models are released openly, giving a reusable recipe for annotating other phone-first, low-resource languages.

Key points

  • Source data is ETD-AM: over 300k tweets in Ethiopic script from December 2019 to January 2020, a period of major political events.
  • An extended lexicon (435 positive, 660 negative entries) filtered the tweets down to 9.4k, and each was labelled by three Telegram users.
  • The labels are positive, negative, neutral and mixed. Results are reported for all four classes and for a "cleaned" three-class setting with mixed removed.
  • ASAB quality control and rewards:
    • one known-answer control item is inserted per 6 tweets;
    • three consecutive mistakes trigger a warning, and continued random answering leads to a block;
    • a voucher is issued after 50 annotated tweets.
  • Model findings:
    • deep models beat classical ones, and fine-tuned multilingual Flair beats the non-fine-tuned version;
    • DeepWalk and Role2Vec network embeddings outperform Word2Vec;
    • NearestCentroid over-predicts "mixed" and misses neutral and positive tweets, so the authors suggest ensembles.

Interesting ideas

  • I1. Gold items with known answers are hidden among real items at a fixed rate, and repeated failures escalate from a warning to a block, so quality is policed continuously rather than afterwards.
  • I2. A seed lexicon is used to choose which raw texts are worth paying to annotate, which concentrates a small budget on informative items.
  • I3. Disagreements between model and human labels are split into "annotator was wrong" and "model was wrong," with concrete causes such as UI lag producing misclicks.
  • I4. Embeddings learned over a word-similarity graph (a distributional thesaurus) worked better than plain word vectors for this small-data task.
  • I5. Idioms whose literal words look neutral, like the "abdomen burn" phrase meaning roughly "let it go," were misread by the model.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Known-answer sentences could be slipped into batches sent to LLM teacher tiers or witness parsers, so an unreliable witness is detected and down-weighted before its silver trees are accepted.
  • I2 (learn): Sentences could be pre-filtered for target constructions (rare deprels, specific VerbNet classes) before requesting teacher annotation, so expensive labels land where the parser is weak.
  • I3 (learn): Parser-vs-treebank disagreements should be audited for gold errors before they count as failures, and confirmed gold errors could feed a correction list.
  • I4 (world): Graph embeddings over WordNet or the knowledge graph could serve as an explainable similarity signal, since each neighbour traces back to an explicit edge.
  • I5 (frames): A non-compositional idiom flag could block literal frame construction and route such phrases to implicature or irony handling instead.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Universität HamburgHamburg · Germany79 works · 5,422 citations🎓University of BolognaBologna · Italy17 works · 1,821 citations🎓Bahir Dar UniversityBahir Dar · Ethiopia11 works · 261 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 972 worksSemitic languages · 2,908 works

Cite

Cite as Seid Muhie Yimam, Hizkiel Mitiku Alemayehu, Abinew Ayele and Chris Biemann. 2020. Exploring Amharic Sentiment Analysis from Social Media Texts: Building Annotation Tools and Classification Models. Proceedings of the 28th International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.coling-main.91

@inproceedings{yimam2020exploring,
  title = {{Exploring Amharic Sentiment Analysis from Social Media Texts: Building Annotation Tools and Classification Models}},
  author = {Seid Muhie Yimam, Hizkiel Mitiku Alemayehu, Abinew Ayele and Chris Biemann},
  booktitle = {Proceedings of the 28th International Conference on Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.coling-main.91},
  url = {https://doi.org/10.18653/v1/2020.coling-main.91}
}

References

53 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mesfin Abate and Yaregal Assabie. 2014. Development of Amharic Morphological Analyzer Using Memory-Based Learning. In International Conference on Natural Language Processing, NLP 2014, pages 1–13, Warsaw, Poland. find
  2. Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence labeling. In https://github.com/uhh-lt/ASAB Proceedings of the 27th International Conference on Computational Linguistics, pages 1638–1649, Santa Fe, NM, USA. ↗
  3. Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. FLAIR: An easy-to-use framework for state-of-the-art NLP. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 54–59, Minneapolis, MN, USA.
  4. Md Shad Akhtar, Asif Ekbal, and Pushpak Bhattacharyya. 2016. Aspect based sentiment analysis in Hindi: resource creation and evaluation. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 2703–2709, Portorož, Slovenia. find
  5. Girma Neshir Alemneh, Andreas Rauber, and Solomon Atnafu. 2019. Dictionary Based Amharic Sentiment Lexicon Generation. In International Conference on Information and Communication Technology for Development for Africa, pages 311–326, Bahir Dar, Ethiopia. ↗
  6. N. Anitha, B. Anitha, and S. Pradeepa. 2013. Sentiment classification approaches. International Journal of Innovations in Engineering and Technology (IJIET), 3(1):22–31. find
  7. Gilbert Badaro, Ramy Baly, Hazem Hajj, Wassim El-Hajj, Khaled Bashir Shaban, Nizar Habash, Ahmad Al-Sallab, and Ali Hamdi. 2019. A survey of opinion mining in arabic: a comprehensive system perspective covering challenges and advances in tools, resources, models, applications, and visualizations. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 18(3):1–52. find
  8. Josemar A. Caetano, Hélder S. Lima, Mateus F. Santos, and Humberto T. Marques-Neto. 2018. Using sentiment analysis to define Twitter political users’ classes and their homophily during the 2016 American presidential election. Journal of Internet Services and Applications, 9(1):1–15. ↗
  9. Hongyun Cai, Vincent W. Zheng, and Kevin Chen-Chuan Chang. 2018. A comprehensive survey of graph embedding: Problems, techniques, and applications. IEEE Transactions on Knowledge and Data Engineering, 30(9):1616–1637. ↗
  10. Corinna Cortes and Vladimir Vapnik. 1995. Support-Vector Networks. Machine learning, 22(11):273–297. ↗
  11. Padraig Cunningham and Sarah Jane Delany. 2020. k-Nearest Neighbour Classifiers: 2nd Edition (with Python examples). eprint: 2004.04523, archivePrefix: arXiv, primaryClass:cs.CL. find
  12. Alessia D’Andrea, Fernando Ferri, Patrizia Grifoni, and Tiziana Guzzo. 2015. Approaches, tools and applications for sentiment analysis implementation. International Journal of Computer Applications, 125(3):26–33. ↗
  13. Paulo De Souza Bermejo, José Pereira, Daniely Barbosa, and Daniel Oliveira. 2019. The application of the sentiment analysis technique in social media as a tool for social management practices at the governmental level. Revista de Administracao Publica, 53:235–251. find
  14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, MN, USA.
  15. Fiona M. Dickinson, Mary McCauley, Barbara Madaj, and Nynke van den Broek. 2019. Using electronic tablets for data collection for healthcare service and maternal health assessments in low resource settings: lessons learnt. BMC Health Services Research, 19(1):336. ↗
  16. Ahmed Fadhil and Adolfo Villafiorita. 2017. An adaptive learning with gamification & conversational UIs: The rise of CiboPoliBot. In Adjunct publication of the 25th conference on user modeling, adaptation and personalization, pages 408–412, Bratislava, Slovakia. find
  17. Ronen Feldman. 2013. Techniques and applications for sentiment analysis. Communications of the Association for computing machinery, 56(4):82–89. ↗
  18. Mark A Finlayson and Tomaž Erjavec. 2017. Overview of Annotation Creation: Processes and Tools. In Handbook of Linguistic Annotation, pages 167–191. Springer. ↗
  19. Rama Rohit Reddy Gangula and Radhika Mamidi. 2018. Resource Creation Towards Automated Sentiment Analysis in Telugu (a Low Resource Language) and Integrating Multiple Domain Sources to Enhance Sentiment Prediction. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), pages 627–634, Miyazaki, Japan. find
  20. Selama Gebremeskel. 2010. Sentiment Mining Model for Opinionated Amharic Texts. unpublished, Online: http://etd.aau.edu.et/handle/123456789/3029. ↗
  21. Andargachew Mekonnen Gezmu, Binyam Ephrem Seyoum, Michael Gasser, and Andreas Nürnberger. 2018. Contemporary Amharic Corpus: Automatically Morpho-Syntactically Tagged Amharic Corpus. In Proceedings of the First Workshop on Linguistic Resources for Natural Language Processing, pages 65–70, Alexandria, Egypt.
  22. William L. Hamilton, Rex Ying, and Jure Leskovec. 2017. Representation Learning on Graphs: Methods and Applications. IEEE Data Engineering Bulletin, pages 1–24. ↗
  23. Eduard Hovy and Julia Lavid. 2010. Towards a ‘science’of corpus annotation: a new methodological challenge for corpus linguistics. International journal of translation, 22(1):13–36. find
  24. Abhik Jana and Pawan Goyal. 2018. Can network embedding of distributional thesaurus be combined with word vectors for better representation? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 463–473, New Orleans, LA, USA. find
  25. Dan Jurafsky and James H. Martin. 2019. Speech and language processing (3rd ed. draft). Online version: https://web.stanford.edu/~jurafsky/slp3/. ↗
  26. Erick Kauffmann, Jesús Peral, David Gil, Antonio Ferrández, Ricardo Sellers-Rubio, and Higinio Mora. 2019. Managing marketing decision-making with sentiment analysis: An evaluation of the main product features using text data mining. Sustainability, 11(15):4235. ↗
  27. Moshe Koppel and Jonathan Schler. 2005. The importance of neutral examples for learning sentiment. In Workshop on the Analysis of Informal and Formal Information Exchange During Negotiations (FINEXIN), pages 100–109, Ottawa, ON, Canada. ↗
  28. Robert J. Lennox, Diogo Veríssimo, William M. Twardek, Colin R. Davis, and Ivan Jarić. 2020. Sentiment analysis as a measure of conservation culture in scientific literature. Conservation Biology, 34(2):462–471. ↗
  29. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. CoRR. eprint: 1907.11692, primaryClass:cs.CL.
  30. Bing Liu. 2012. Sentiment analysis and opinion mining. Synthesis lectures on human language technologies, 5(1):1–167. ↗
  31. Noam Lupu and Kristin Michelitch. 2018. Advances in survey methods for the developing world. Annual Review of Political Science, 21:195–214. ↗
  32. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed Representations of Words and Phrases and Their Compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2, NIPS’13, page 3111–3119, Red Hook, NY, USA. ↗
  33. Shervin Minaee, Nal Kalchbrenner, Erik Cambria, Narjes Nikzad, Meysam Chenaghlu, and Jianfeng Gao. 2020. Deep Learning Based Text Classification: A Comprehensive Review. eprint: 2004.03705, archivePrefix: arXiv, primaryClass:cs.CL. find
  34. Preslav Nakov, Sara Rosenthal, Zornitsa Kozareva, Veselin Stoyanov, Alan Ritter, and Theresa Wilson. 2013. SemEval-2013 Task 2: Sentiment Analysis in Twitter. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 312–320, Atlanta, GE, USA.
  35. Pooja Pandey and Sharvari Govilkar. 2015. A framework for sentiment analysis in Hindi using HSWN. International Journal of Computer Applications, 119(19). ↗
  36. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in python. Journal of Machine Learning Research, 12:2825–2830. ↗
  37. Wondwossen Philemon and Wondwossen Mulugeta. 2014. A Machine Learning Approach to Multi-Scale Sentiment Analysis of Amharic Online Posts. HiLCoE Journal of Computer Science and Technology, 2(2):8. find
  38. Rudy Prabowo and Mike Thelwall. 2009. Sentiment analysis: A combined approach. Journal of Informetrics, 3(2):143 – 157. ↗
  39. James Pustejovsky and Amber Stubbs. 2012. Natural Language Annotation for Machine Learning: A guide to corpus-building for applications. O’Reilly Media, Inc. find
  40. Eugen Ruppert, Manuel Kaufmann, Martin Riedl, and Chris Biemann. 2015. JoBimViz: A web-based visualization for graph-based distributional semantic models. In Proceedings of ACL-IJCNLP 2015 System Demonstrations, pages 103–108, Beijing, China. find
  41. Marta Sabou, Kalina Bontcheva, Leon Derczynski, and Arno Scharl. 2014. Corpus Annotation through Crowdsourcing: Towards Best Practice Guidelines. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 859–866, Reykjavik, Iceland. find
  42. Özge Sevgili, Alexander Panchenko, and Chris Biemann. 2019. Improving neural entity disambiguation with graph embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 315–322, Florence, Italy. find
  43. Pontus Stenetorp, Sampo Pyysalo, Goran Topić, Tomoko Ohta, Sophia Ananiadou, and Jun’ichi Tsujii. 2012. BRAT: a web-based tool for NLP-assisted text annotation. In Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 102–107, Avignon, France.
  44. Amirsina Torfi, Rouzbeh A. Shirvani, Yaser Keneshloo, Nader Tavvaf, and Edward A. Fox. 2020. Natural language processing advancements by deep learning: A survey. eprint: 2003.01200, archivePrefix: arXiv, prima-ryClass:cs.CL. find
  45. Karsten Tymann, Matthias Lutz, Patrick Palsbröker, and Carsten Gips. 2019. GerVADER -A German adaptation of the VADER sentiment analysis tool for social media texts. In ”Lernen, Wissen, Daten, Analysen - LWDA2019, pages 178–189, Berlin, Germany. find
  46. Charu Virmani, Anuradha Pillai, and Dimple Juneja. 2017. Extracting Information from Social Network using NLP. International Journal of Computational Intelligence Research, 13(4):621–630. find
  47. Aobo Wang, Cong Duy Vu Hoang, and Min-Yen Kan. 2013. Perspectives on crowdsourcing annotations for natural language processing. Language resources and evaluation, 47(1):9–31. ↗
  48. Michael Wojatzki, Eugen Ruppert, Sarah Holschneider, Torsten Zesch, and Chris Biemann. 2017. GermEval 2017: Shared Task on Aspect-based Sentiment in Social Media Customer Feedback. In Proceedings of the GSCL GermEval Shared Task on Aspect-based Sentiment in Social Media Customer Feedback, pages 1–12, Berlin, Germany. ↗
  49. Mobile Time Spent 2018. Will Smartphones Remain Ascendant? Technical report, emarketer.com. Online: https://www.emarketer.com/content/mobile-time-spent-2018. ↗
  50. Seid Muhie Yimam, Abinew Ali Ayele, and Chris Biemann. 2019. Analysis of the Ethiopic Twitter Dataset for Abusive Speech in Amharic. In In Proceedings of International Conference On Language Technologies For All: Enabling Linguistic Diversity And Multilingualism Worldwide (LT4ALL 2019), pages 1–5, Paris, France. ↗
  51. Tom Young, Devamanyu Hazarika, Soujanya Poria, and Erik Cambria. 2018. Recent trends in deep learning based natural language processing [review article]. IEEE Computational Intelligence Magazine, 13:55–75. ↗
  52. Bianca Zadrozny and Charles Elkan. 2002. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, page 694–699, New York, NY, USA. find
  53. Radim Řehůřek and Petr Sojka. 2011. Gensim – Statistical Semantics in Python. In EuroScipy 2011, Paris, France. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yimam-2020-exploring-amharic-sentiment-analysis-social/