Elamur Research⌕
☰
home / papers / 2020 / A Sentiment Analysis Dataset for Code-Mixed Malayalam-English

A Sentiment Analysis Dataset for Code-Mixed Malayalam-English

Summary

Social media comments in India are often written in a mix of languages and typed in Roman script instead of native scripts. Sentiment classifiers trained on monolingual text handle such input poorly, and before this work no labelled resource existed for Malayalam-English code-mixed text. The authors built one from YouTube comments on 2019 Malayalam movie trailers. They automatically removed comments in a single language, comments in Malayalam script, emojis, and sentences that were too short or too long, which left 6,738 code-mixed sentences. Volunteer bilingual annotators labelled each sentence with one of five categories through Google Forms. Disagreements were resolved in stages by bringing in additional annotators. The resulting gold standard reaches a Krippendorff's alpha above 0.8. The authors then used it to benchmark several common classification approaches, including DME, CDME, 1DConv and BERT. They describe it as the first code-mixed sentiment dataset for Malayalam, and they release both the data and the code. This gives an under-resourced Dravidian language a supervised training and evaluation resource for informal, script-mixed user text.

Key points

  • Data collection: 116,711 raw comments were scraped with youtube-comment-scraper, using results for the search "Malayalam movie 2019".
  • Filtering: The langdetect library was used to drop monolingual comments, and comments in Malayalam script were also discarded. Only sentences of 5–15 words were kept, giving a final corpus of 6,738 sentences.
  • Label scheme: Five classes were used, following Mohammad (2016): Positive, Negative, Mixed feelings, Neutral, and Not in intended language.
  • Annotators: Six volunteers took part (2 male, 4 female, all postgraduates, 5 schooled in English medium). Agreement on the dataset is a Krippendorff's alpha above 0.8.
  • Benchmarks and release: Baselines include DME, CDME, 1DConv and BERT. The dataset and code are on GitHub (MalayalamMixSentiment).

Interesting ideas

  • I1. Escalating annotation: Each sentence starts with two annotators. A third is added only on conflict, and two more only if all three still disagree. This spends annotation effort where items are genuinely ambiguous, which suits volunteer-driven projects with limited labour.
  • I2. Filtering by detector confidence: Instead of searching for code-mixed text directly, the authors discard comments that a language detector confidently assigns to one language. This is a cheap way to bootstrap a mixed-language corpus for any pair where a dedicated code-mixing detector does not exist.
  • I3. Length limits for annotators: Sentences were restricted to 5–15 words mainly because longer ones are hard to label consistently. This trades some coverage for label quality, a choice other crowd or volunteer annotation efforts could adopt explicitly.
  • I4. Catch-all and concrete-example labels: The scheme includes a "Not in intended language" label to absorb noise. Neutral is defined with concrete examples, such as subscription requests or questions about release dates. Both choices reduce forced, inconsistent labelling of off-topic social media content.
  • I5. Lightweight form controls: Forms were capped at 100 sentences, and collecting email addresses limited each annotator to a single submission. With consent, the forms also gathered annotator demographics (gender, education, schooling medium). This makes annotator diversity documentable using only free tools such as Google Forms.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

81 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 77 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{chakravarthi2020sentiment,
  title = {{A Sentiment Analysis Dataset for Code-Mixed Malayalam-English}},
  author = {Chakravarthi, Bharathi Raja and Jose, Navya and Suryawanshi, Shardul and Sherly, Elizabeth and McCrae, John P.},
  booktitle = {Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL)},
  year = {2020},
  doi = {10.5281/zenodo.4015233},
  url = {https://doi.org/10.5281/zenodo.4015233}
}

References

55 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agrawal, R., Chenthil Kumar, V., Muralidharan, V., and Sharma, D. (2018). No more beating about the bush : A step towards idiom handling for Indian language NLP. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May. European L… find
  2. Androutsopoulos, J. (2013). Code-switching in computer-mediated communication. Pragmatics of computer-mediated communication, pages 667–694. find
  3. Balage Filho, P. P., Brun, C., and Rondeau, G. (2012). A graphical user interface for feature-based opinion mining. In Proceedings of the Demonstration Session at the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5–8, Mo… find
  4. Bali, K., Sharma, J., Choudhury, M., and Vyas, Y. (2014). “I am borrowing ya mixing ?” an analysis of English-Hindi code mixing in Facebook. In Proceedings of the First Workshop on Computational Approaches to Code Switching, pages 116–126, Doha, Qatar, October. Association for Computational Linguist… find
  5. Barman, U., Das, A., Wagner, J., and Foster, J. (2014). Code mixing: A challenge for language identification in the language of social media. In Proceedings of the First Workshop on Computational Approaches to Code Switching, pages 13–23, Doha, Qatar, October. Association for Computational Linguisti… find
  6. Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T. (2017a). Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. find
  7. Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T. (2017b). Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. find
  8. Chakravarthi, B. R., Arcan, M., and McCrae, J. P. (2018). Improving Wordnets for Under-Resourced Languages Using Machine Translation. In Proceedings of the 9th Global WordNet Conference. find
  9. The Global WordNet Conference 2018 Committee. find
  10. Chakravarthi, B. R., Arcan, M., and McCrae, J. P. (2019a). Comparison of different orthographies for machine translation of under-resourced dravidian languages. In 2nd Conference on Language, Data and Knowledge (LDK 2019). Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik. find
  11. Chakravarthi, B. R., Arcan, M., and McCrae, J. P. (2019b). WordNet gloss translation for under-resourced languages using multilingual neural machine translation. In Proceedings of the Second Workshop on Multilingualism at the Intersection of Knowledge Bases and Machine Translation, pages 1–7, Dublin… find
  12. Chakravarthi, B. R., Priyadharshini, R., Stearns, B., Jayapal, A., S, S., Arcan, M., Zarrouk, M., and McCrae, J. P. (2019c). Multilingual multimodal machine translation for Dravidian languages utilizing phonetic transcription. In Proceedings of the 2nd Workshop on Technologies for MT of Low Resource… find
  13. Chakravarthi, B. R., Muralidaran, V., Priyadharshini, R., and McCrae, J. P. (2020). Corpus creation for sentiment analysis in code-mixed Tamil-English text. In Proceedings of the 1st Joint Workshop of SLTU (Spoken Language Technologies for Under-resourced languages) and CCURL (Collaboration and Comp… find our card
  14. Cieliebak, M., Deriu, J. M., Egger, D., and Uzdilli, F. (2017). A Twitter corpus and benchmark resources for German sentiment analysis. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, pages 45–51, Valencia, Spain, April. Association for Computation… find
  15. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. Mona Diab, et al., editors. (2014). Proceedings of the First Workshop on Computational Approaches to Code Switching, Doha, Q… find
  16. Esuli, A. and Sebastiani, F. (2006). Sentiwordnet: A publicly available lexical resource for opinion mining. In In Proceedings of the 5th Conference on Language Resources and Evaluation (LREC’06, pages 417–422. find our card
  17. WordNet: An Electronic Lexical Database. Bradford Books. Habimana, O., Li, Y., Li, R., Gu, X., and Yu, G. (2019). Sentiment analysis using deep learning approaches: an overview. Science China Information Sciences, 63(1):111102, Dec. find
  18. Hu, M. and Liu, B. (2004). Mining and summarizing customer reviews. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’04, page 168–177, New York, NY, USA. Association for Computing Machinery. find
  19. Jiang, Q., Chen, L., Xu, R., Ao, X., and Yang, M. (2019). A challenge dataset and effective models for aspect-based sentiment analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing… find our card
  20. Jose, N., Chakravarthi, B. R., Suryawanshi, S., Sherly, E., and McCrae, J. P. (2020). A survey of current datasets for code-switching research. In 2020 6th International Conference on Advanced Computing & Communication Systems (ICACCS). find
  21. Joshi, A., Prabhu, A., Shrivastava, M., and Varma, V. (2016). Towards sub-word level compositions for sentiment analysis of Hindi-English code mixed text. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2482–2491, Osaka, Japan, D… find
  22. The COL-ING 2016 Organizing Committee. find
  23. Kalchbrenner, N., Grefenstette, E., and Blunsom, P. (2014). A convolutional neural network for modelling sentences. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 655–665, Baltimore, Maryland, June. Association for Computatio… find our card
  24. Kiela, D., Wang, C., and Cho, K. (2018). Dynamic meta-embeddings for improved sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1466–1477, Brussels, Belgium, October-November. Association for Computational Linguistics. find
  25. The Dravidian Languages. Cambridge University Press. Lalitha Devi, S. (2019). Resolving pronouns for a resource-poor language, Malayalam using resource-rich language, Tamil. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 611–618,… find
  26. INCOMA Ltd. Lee, S. and Wang, Z. (2015). Emotion in code-switching texts: Corpus construction and analysis. In Proceedings of the Eighth SIGHAN Workshop on Chinese Language Processing, pages 91–99, Beijing, China, July. Association for Computational Linguistics. find
  27. Mæhlum, P., Barnes, J., Øvrelid, L., and Velldal, E. (2019). Annotating evaluative sentences for sentiment analysis: a dataset for Norwegian. In Proceedings of the 22nd Nordic Conference on Computational Linguistics, pages 121–130, Turku, Finland, September–October. Linköping University Electronic… find
  28. Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013a). Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. find
  29. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. (2013b). Distributed representations of words and phrases and their compositionality. In C. J. C. Burges, et al., editors, Advances in Neural Information Processing Systems 26, pages 3111–3119. find
  30. Curran Associates, Inc. Mohammad, S. (2016). A practical guide to sentiment annotation: Challenges and solutions. In Proceedings of the 7th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 174–179, San Diego, California, June. Association for Computati… find
  31. Mouthami, K., Devi, K. N., and Bhaskaran, V. M. (2013). Sentiment analysis and classification based on textual reviews. In 2013 international conference on Information communication and embedded systems (ICICES), pages 271–276. find
  32. IEEE. Nair, D. S., Jayan, J. P., Sherly, E., et al. (2014). Sentima-sentiment extraction for Malayalam. In 2014 International Conference on Advances in Computing, Communications and Informatics (ICACCI), pages 1719–1723. find
  33. IEEE. Patra, B. G., Das, D., and Das, A. (2018). Sentiment analysis of code-mixed indian languages: An overview of sail code-mixed shared task@ icon-2017. arXiv preprint arXiv:1803.06745. — [arXiv:1803.06745](https://arxiv.org/abs/1803.06745) find
  34. Pennington, J., Socher, R., and Manning, C. D. (2014). Glove: Global vectors for word representation. In EMNLP, volume 14, pages 1532–1543. find
  35. Poria, S., Gelbukh, A., Cambria, E., Yang, P., Hussain, A., and Durrani, T. (2012). Merging SenticNet and WordNet-Affect emotion lists for sentiment analysis. In 2012 IEEE 11th International Conference on Signal Processing, volume 2, pages 1251–1255, Oct. find
  36. Priyadharshini, R., Chakravarthi, B. R., Vegupatti, M., and McCrae, J. P. (2020). Named entity recognition for code-mixed Indian corpus using meta embedding. In 2020 6th International Conference on Advanced Computing & Communication Systems (ICACCS). find
  37. Rani, P., Suryawanshi, S., Goswami, K., Chakravarthi, B. R., Fransen, T., and McCrae, J. P. (2020). A comparative study of different state-of-the-art hate speech detection methods for Hindi-English code-mixed data. In Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying, Mars… find
  38. Ranjan, P., Raja, B., Priyadharshini, R., and Balabantaray, R. C. (2016). A comparative study on code-mixed data of Indian social media vs formal text. In 2016 2nd International Conference on Contemporary Computing and Informatics (IC3I), pages 608–611, Dec. find
  39. Rogers, A., Romanov, A., Rumshisky, A., Volkova, S., Gronas, M., and Gribov, A. (2018). RuSentiment: An enriched sentiment analysis dataset for social media in Russian. In Proceedings of the 27th International Conference on Computational Linguistics, pages 755–763, Santa Fe, New Mexico, USA, August.… find
  40. Rosowsky, A. (2010). ‘writing it in english’: script choices among young multilingual muslims in the uk. Journal of Multilingual and Multicultural Development, 31(2):163– 179. find
  41. Saint-Jacques, B. (1987). The roman alphabet in the japanese writing system. Visible Language, 21(1):88. find
  42. Sarkar, K. and Chakraborty, S. (2015). A sentiment analysis system for indian language tweets. In International Conference on Mining Intelligence and Knowledge Exploration, pages 694–702. find
  43. Springer. Scotton, C. M. (1982). The possibility of code-switching: motivation for maintaining multilingualism. Anthropological linguistics, pages 432–444. find
  44. Se, S., Vinayakumar, R., Kumar, M. A., and Soman, K. (2015). AMRITA-CEN@ SAIL2015: sentiment analysis in Indian languages. In International Conference on Mining Intelligence and Knowledge Exploration, pages 703–710. find
  45. Springer. Se, S., Vinayakumar, R., Kumar, M. A., and Soman, K. (2016). Predicting the sentimental reviews in Tamil movie using machine learning algorithms. Indian Journal of Science and Technology, 9(45). find
  46. Severyn, A., Moschitti, A., Uryupina, O., Plank, B., and Filippova, K. (2014). Opinion mining on YouTube. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1252–1261, Baltimore, Maryland, June. Association for Computational Ling… find
  47. Solorio, T., Blair, E., Maharjan, S., Bethard, S., Diab, M., Ghoneim, M., Hawwari, A., AlGhamdi, F., Hirschberg, J., Chang, A., and Fung, P. (2014). Overview for the first shared task on language identification in code-switched data. In Proceedings of the First Workshop on Computational Approaches t… find our card
  48. Sreelekha, S. and Bhattacharyya, P. (2018). Morphology Injection for English-Malayalam Statistical Machine Translation. In Nicoletta Calzolari (Conference chair), et al., editors, Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan,… find
  49. Suryawanshi, S., Chakravarthi, B. R., Arcan, M., and Buitelaar, P. (2020a). Multimodal meme dataset (Multi-OFF) for identifying offensive content in image and text. In Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying, Marseille, France, May. European Language Resources As… find
  50. Suryawanshi, S., Chakravarthi, B. R., Verma, P., Arcan, M., McCrae, J. P., and Buitelaar, P. (2020b). A dataset for troll classification of Tamil memes. In Proceedings of the 5th Workshop on Indian Language Data Resource and Evaluation (WILDRE-5), Marseille, France, May. European Language Resources… find
  51. Tay, M. W. (1989). Code switching and code mixing as a communicative strategy in multilingual discourse. World Englishes, 8(3):407–417. find
  52. Thottingal, S. (2019). Finite state transducer based morphology analysis for Malayalam language. In Proceedings of the 2nd Workshop on Technologies for MT of Low Resource Languages, pages 1–5, Dublin, Ireland, August. European Association for Machine Translation. find
  53. Valitutti, R. (2004). WordNet-Affect: an affective extension of wordnet. In In Proceedings of the 4th International Conference on Language Resources and Evaluation, pages 1083–1086. find our card
  54. Wiebe, J., Wilson, T., and Cardie, C. (2005). Annotating expressions of opinions and emotions in language. Language Resources and Evaluation, 39(2):165–210, May. find
  55. Zhou, P., Qi, Z., Zheng, S., Xu, J., Bao, H., and Xu, B. (2016). Text classification improved by integrating bidirectional lstm with two-dimensional max pooling. arXiv preprint arXiv:1611.06639. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chakravarthi-2020-sentiment-analysis-dataset-code-mixed/