Elamur Research⌕
☰
home / papers / 2020 / Corpus Creation for Sentiment Analysis in Code-Mixed Tamil-English Text

Corpus Creation for Sentiment Analysis in Code-Mixed Tamil-English Text

Summary

Social media comments in Tamil are often a blend of Tamil and English (Tanglish), usually typed in Roman script and loose with grammar. Tamil has almost no labelled code-mixed sentiment data, and resources built for monolingual text handle such comments poorly. The authors address this by building TamilMixSentiment, a gold-standard corpus of YouTube comments annotated for sentiment. They scraped comments from trailers of 2019 movies. They then filtered out comments that were purely Tamil, purely English, or in other languages, and kept only comments of moderate length. Volunteer native speakers labelled the remaining 15,744 comments through Google Forms, using a staged scheme adapted from Mohammad (2016) that brings in extra annotators when labels conflict. Agreement measured with Krippendorff's α reached 0.6. The authors also trained several classical and neural classifiers as baselines, and logistic regression and random forest performed best. The resource matters because the authors describe it as the first, and the largest general-domain, sentiment dataset for code-mixed Tamil-English. It gives future work on this under-resourced language pair a public benchmark to compare against.

Key points

  • Data collection and filtering: 184,573 comments were collected with the YouTube Comment Scraper. Monolingual comments were removed with comment-level language identification (the langdetect library), and emoticons were stripped. Only comments of 5–15 words were kept, leaving 15,744 Tanglish sentences.
  • Data split: The corpus was split randomly into 11,335 training, 1,260 validation and 3,149 test sentences. The authors used this fixed split rather than k-fold cross-validation.
  • Annotators: Eleven native Tamil speakers took part (9 male, 2 female), with varied education levels and schooling medium (English or Tamil). Guidelines were provided in both English and Tamil.
  • Label scheme: An extra "Other language" label was added to the base scheme. It covers comments written in other languages, such as other Indian languages typed in Roman script.
  • Code-mixing and baselines: The data contains all three code-mixing types: inter-sentential, intra-sentential and tag switching. The baselines covered classical models (SVM, random forest, a decision tree, naive Bayes and logistic regression) as well as neural ones (dynamic meta-embedding, contextualized dynamic meta-embedding, 1DConv-LSTM and BERT). The dataset is released on GitHub (bharathichezhiyan/TamilMixSentiment).

Interesting ideas

  • I1. Escalating annotation: Two people label each item first, a third is added only when they disagree, and two more join if disagreement persists. This saves annotator effort on easy cases while still resolving hard ones, which is a cheap pattern for any volunteer-driven labelling project.
  • I2. Comment-level language filtering: Rather than tagging the language of every word, the authors filter whole comments to keep only mixed ones and discard monolingual text that is already well covered. This avoids costly word-level annotation, which makes it practical for other mixed-language social media data (e.g., Ukrainian-Russian surzhyk).
  • I3. Small forms with identity checks: Batches are capped at 100 sentences, email addresses ensure no one labels the same item twice, and annotators must confirm they understood the scheme before starting. This lightweight quality setup needs no paid crowdsourcing platform.
  • I4. Screening annotators early: Contributions were removed if the annotator responded with unreasonable delays, gave every item the same label, or made too many errors in the first form. These are simple heuristics that transfer easily to other annotation campaigns.
  • I5. Explicit "Other language" label: Giving annotators a category for unexpected languages catches cases the automatic language filter missed, so they do not pollute the sentiment classes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

133 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 130 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{chakravarthi2020corpus,
  title = {{Corpus Creation for Sentiment Analysis in Code-Mixed Tamil-English Text}},
  author = {Chakravarthi, Bharathi Raja and Muralidaran, Vigneshwaran and Priyadharshini, Ruba and McCrae, John P.},
  booktitle = {Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL)},
  year = {2020},
  doi = {10.5281/zenodo.4015252},
  url = {https://doi.org/10.5281/zenodo.4015252}
}

References

50 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agrawal, R., Chenthil Kumar, V., Muralidharan, V., and Sharma, D. (2018). No more beating about the bush : A step towards idiom handling for Indian language NLP. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May. European L… find
  2. AlGhamdi, F., Molina, G., Diab, M., Solorio, T., Hawwari, A., Soto, V., and Hirschberg, J. (2016). Part of speech tagging for code switched data. In Proceedings of the Second Workshop on Computational Approaches to Code Switching, pages 98–107, Austin, Texas, November. Association for Computational… find
  3. A.R., B., Joshi, A., and Bhattacharyya, P. (2012). Cross-lingual sentiment analysis for Indian languages using linked WordNets. In Proceedings of COLING 2012: Posters, pages 73–82, Mumbai, India, December. find
  4. The COLING 2012 Organizing Committee. find
  5. Barman, U., Das, A., Wagner, J., and Foster, J. (2014). Code mixing: A challenge for language identification in the language of social media. In Proceedings of the First Workshop on Computational Approaches to Code Switching, pages 13–23, Doha, Qatar, October. Association for Computational Linguisti… find
  6. Chakravarthi, B. R., Arcan, M., and McCrae, J. P. (2018). Improving wordnets for under-resourced languages using machine translation. In Proceedings of the 9th Global WordNet Conference (GWC 2018), page 78. find
  7. Chakravarthi, B. R., Arcan, M., and McCrae, J. P. (2019a). Comparison of different orthographies for machine translation of under-resourced dravidian languages. In 2nd Conference on Language, Data and Knowledge (LDK 2019). Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik. find
  8. Chakravarthi, B. R., Arcan, M., and McCrae, J. P. (2019b). WordNet gloss translation for under-resourced languages using multilingual neural machine translation. In Proceedings of the Second Workshop on Multilingualism at the Intersection of Knowledge Bases and Machine Translation, pages 1–7, Dublin… find
  9. Chakravarthi, B. R., Priyadharshini, R., Stearns, B., Jayapal, A., S, S., Arcan, M., Zarrouk, M., and McCrae, J. P. (2019c). Multilingual multimodal machine translation for Dravidian languages utilizing phonetic transcription. In Proceedings of the 2nd Workshop on Technologies for MT of Low Resource… find
  10. Chakravarthi, B. R., Jose, N., Suryawanshi, S., Sherly, E., and McCrae, J. P. (2020). A sentiment analysis dataset for code-mixed Malayalam-English. In Proceedings of the 1st Joint Workshop of SLTU (Spoken Language Technologies for Under-resourced languages) and CCURL (Collaboration and Computing fo… find our card
  11. Chanda, A., Das, D., and Mazumdar, C. (2016). Unraveling the English-Bengali code-mixing phenomenon. In Proceedings of the Second Workshop on Computational Approaches to Code Switching, pages 80–89, Austin, Texas, November. Association for Computational Linguistics. find
  12. Cieliebak, M., Deriu, J. M., Egger, D., and Uzdilli, F. (2017). A Twitter corpus and benchmark resources for German sentiment analysis. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, pages 45–51, Valencia, Spain, April. Association for Computation… find
  13. Cohan, A., Beltagy, I., King, D., Dalvi, B., and Weld, D. (2019). Pretrained language models for sequential sentence classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EM… find
  14. Das, A. and Bandyopadhyay, S. (2010). SentiWordNet for Indian languages. In Proceedings of the Eighth Workshop on Asian Language Resouces, pages 56–63, Beijing, China, August. find
  15. Coling 2010 Organizing Committee. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. find
  16. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis,… find
  17. Hu, M. and Liu, B. (2004). Mining and summarizing customer reviews. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’04, page 168–177, New York, NY, USA. Association for Computing Machinery. find
  18. Jiang, Q., Chen, L., Xu, R., Ao, X., and Yang, M. (2019). A challenge dataset and effective models for aspect-based sentiment analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing… find our card
  19. Jose, N., Chakravarthi, B. R., Suryawanshi, S., Sherly, E., and McCrae, J. P. (2020). A survey of current datasets for code-switching research. In 2020 6th International Conference on Advanced Computing & Communication Systems (ICACCS). find
  20. Joshi, A., Prabhu, A., Shrivastava, M., and Varma, V. (2016). Towards sub-word level compositions for sentiment analysis of Hindi-English code mixed text. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2482–2491, Osaka, Japan, D… find
  21. The COL-ING 2016 Organizing Committee. find
  22. Kannan, A., Mohanty, G., and Mamidi, R. (2016). Towards building a SentiWordNet for Tamil. In Proceedings of the 13th International Conference on Natural Language Processing, pages 30–35, Varanasi, India, December. NLP Association of India. find
  23. Kiela, D., Wang, C., and Cho, K. (2018). Dynamic meta-embeddings for improved sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1466–1477, Brussels, Belgium, October-November. Association for Computational Linguistics. find
  24. Krippendorff, K. (1970). Estimating the reliability, systematic error and random error of interval data. Educational and Psychological Measurement, 30(1):61–70. find
  25. Krishnasamy, K. (2015). Code mixing among Tamil-English bilingual children. International Journal of Social Science and Humanity, 5(9):788. find
  26. Lee, S. and Wang, Z. (2015). Emotion in code-switching texts: Corpus construction and analysis. In Proceedings of the Eighth SIGHAN Workshop on Chinese Language Processing, pages 91–99, Beijing, China, July. Association for Computational Linguistics. find
  27. Ma, X., Xu, P., Wang, Z., Nallapati, R., and Xiang, B. (2019). Domain adaptation with BERT-based domain classification and data selection. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 76–83, Hong Kong, China, November. Association for Compu… find
  28. Mæhlum, P., Barnes, J., Øvrelid, L., and Velldal, E. (2019). Annotating evaluative sentences for sentiment analysis: a dataset for Norwegian. In Proceedings of the 22nd Nordic Conference on Computational Linguistics, pages 121–130, Turku, Finland, September–October. Linköping University Electronic… find
  29. Mohammad, S. (2016). A practical guide to sentiment annotation: Challenges and solutions. In Proceedings of the 7th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 174–179, San Diego, California, June. Association for Computational Linguistics. find
  30. Padmamala, R. and Prema, V. (2017). Sentiment analysis of online Tamil contents using recursive neural network models approach for Tamil language. In 2017 IEEE International Conference on Smart Technologies and Management for Computing, Communication, Controls, Energy and Materials (ICSTM), pages 28… find
  31. Patra, B. G., Das, D., and Das, A. (2018). Sentiment analysis of code-mixed indian languages: An overview of sail code-mixed shared task@ icon-2017. arXiv preprint arXiv:1803.06745. — [arXiv:1803.06745](https://arxiv.org/abs/1803.06745) find
  32. Phani, S., Lahiri, S., and Biswas, A. (2016). Sentiment analysis of Tweets in three Indian languages. In Proceedings of the 6th Workshop on South and Southeast Asian Natural Language Processing (WSSANLP2016), pages 93–102, Osaka, Japan, December. find
  33. The COLING 2016 Organizing Committee. find
  34. Prasad, S. S., Kumar, J., Prabhakar, D. K., and Tripathi, S. (2016). Sentiment mining: An approach for Bengali and Tamil tweets. In 2016 Ninth International Conference on Contemporary Computing (IC3), pages 1–4, Aug. find
  35. Pratapa, A., Bhat, G., Choudhury, M., Sitaram, S., Dandapat, S., and Bali, K. (2018a). Language modeling for code-mixing: The role of linguistic theory based synthetic data. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1543… find our card
  36. Pratapa, A., Choudhury, M., and Sitaram, S. (2018b). Word embeddings for code-mixed language processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3067– 3072, Brussels, Belgium, October-November. Association for Computational Linguistics. find
  37. Priyadharshini, R., Chakravarthi, B. R., Vegupatti, M., and McCrae, J. P. (2020). Named entity recognition for code-mixed Indian corpus using meta embedding. In 2020 6th International Conference on Advanced Computing & Communication Systems (ICACCS). find
  38. Rani, P., Suryawanshi, S., Goswami, K., Chakravarthi, B. R., Fransen, T., and McCrae, J. P. (2020). A comparative study of different state-of-the-art hate speech detection methods for Hindi-English code-mixed data. In Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying, Mars… find
  39. Ranjan, P., Raja, B., Priyadharshini, R., and Balabantaray, R. C. (2016). A comparative study on code-mixed data of Indian social media vs formal text. In 2016 2nd International Conference on Contemporary Computing and Informatics (IC3I), pages 608–611. find
  40. Remmiya Devi, G., Veena, P., Anand Kumar, M., and Soman, K. (2016). Amrita-cen@ fire 2016: Code-mix entity extraction for Hindi-English and Tamil-English tweets. In CEUR workshop proceedings, volume 1737, pages 304–308. find
  41. Rogers, A., Romanov, A., Rumshisky, A., Volkova, S., Gronas, M., and Gribov, A. (2018). RuSentiment: An enriched sentiment analysis dataset for social media in Russian. In Proceedings of the 27th International Conference on Computational Linguistics, pages 755–763, Santa Fe, New Mexico, USA, August.… find
  42. Sitaram, D., Murthy, S., Ray, D., Sharma, D., and Dhar, K. (2015). Sentiment analysis of mixed language employing hindi-english code switching. In 2015 International Conference on Machine Learning and Cybernetics (ICMLC), volume 1, pages 271–276, July. find
  43. Solorio, T., Blair, E., Maharjan, S., Bethard, S., Diab, M., Ghoneim, M., Hawwari, A., AlGhamdi, F., Hirschberg, J., Chang, A., and Fung, P. (2014). Overview for the first shared task on language identification in code-switched data. In Proceedings of the First Workshop on Computational Approaches t… find our card
  44. Suryawanshi, S., Chakravarthi, B. R., Arcan, M., and Buitelaar, P. (2020a). Multimodal meme dataset (Multi-OFF) for identifying offensive content in image and text. In Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying, Marseille, France, May. European Language Resources As… find
  45. Suryawanshi, S., Chakravarthi, B. R., Verma, P., Arcan, M., McCrae, J. P., and Buitelaar, P. (2020b). A dataset for troll classification of Tamil memes. In Proceedings of the 5th Workshop on Indian Language Data Resource and Evaluation (WILDRE-5), Marseille, France, May. European Language Resources… find
  46. Tayyar Madabushi, H., Kochkina, E., and Castelle, M. (2019). Cost-sensitive BERT for generalisable sentence classification on imbalanced data. In Proceedings of the Second Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda, pages 125–134, Hong Ko… find
  47. Wiebe, J., Wilson, T., and Cardie, C. (2005). Annotating expressions of opinions and emotions in language. Language Resources and Evaluation, 39(2):165–210, May. find
  48. Winata, G. I., Lin, Z., and Fung, P. (2019a). Learning multilingual meta-embeddings for code-switching named entity recognition. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 181–186, Florence, Italy, August. Association for Computational Linguistics. find
  49. Winata, G. I., Lin, Z., Shin, J., Liu, Z., and Fung, P. (2019b). Hierarchical meta-embeddings for code-switching named entity recognition. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processi… find
  50. Yang, Y. and Eisenstein, J. (2017). Overcoming language variation in sentiment analysis with social attention. Transactions of the Association for Computational Linguistics, 5:295–307. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chakravarthi-2020-corpus-creation-sentiment-analysis-code/