Elamur Research⌕
☰
home / papers / 2021 / Findings of the Shared Task on Offensive Language Identification in Tamil, Malayalam, and Kannada

Findings of the Shared Task on Offensive Language Identification in Tamil, Malayalam, and Kannada

Paper · 2021 · Bharathi Raja Chakravarthi, Ruba Priyadharshini, Navya Jose et al.

Summary

This overview paper reports on a shared task on detecting offensive content in social media comments written in three under-resourced Dravidian languages: Tamil, Malayalam and Kannada. Such comments are frequently code-mixed with English and are often typed in Latin rather than native script. The organisers collected YouTube comments posted under 2019 film trailers, filtered out unrelated languages with the langdetect library, and annotated each comment with one of six labels. These labels are a single-level adaptation of the hierarchical scheme of Zampieri et al. (2019), with an added category for comments in the wrong language. Teams trained on released train and validation splits and were scored on a hidden test set using weighted average F1. Most submissions fine-tuned multilingual transformers such as mBERT, XLM-RoBERTa, IndicBERT and DistilBERT, used ULMFiT or AWD-LSTM, or built ensembles of these. Top Malayalam systems reached about 0.97 weighted F1, while Tamil and Kannada stayed near 0.75. The publicly released corpus serves as a benchmark for moderation research in these languages.

Key points

  • Corpus size: Tamil has 43,919 comments, Malayalam 20,010 and Kannada 7,772, with vocabularies of 94,772, 40,729 and 20,796 respectively.
  • The six classes are:
    • Not Offensive
    • Offensive Untargeted
    • Offensive Targeted Individual
    • Offensive Targeted Group
    • Offensive Targeted Other
    • Not in intended language (Malayalam has no "Other" class instances)
  • The data is strongly imbalanced. Non-offensive comments make up about 71% of Tamil and the large majority of Malayalam.
  • 119 participants registered. Because of CodaLab problems, predictions were submitted through a Google form and scored manually.
  • Leading teams:
    • hate-alert placed first in Tamil and Malayalam and second in Kannada.
    • SJ-AJ (mBERT + XLM-RoBERTa ensemble with task-adaptive pre-training) won Kannada.
    • indicnlp@kgp (AWD-LSTM + BERT + RoBERTa ensemble) placed near the top in all three languages.

Interesting ideas

  • I1. Explicit wrong-language class. A dedicated "Not in intended language" label lets noisy scraped data stay in the dataset instead of being silently discarded. The classifier also learns to flag out-of-scope input, which suits any multilingual social media corpus.
  • I2. Flattening a hierarchical scheme. The offensive/targeted/target-type hierarchy was merged into one flat set of labels. This simplifies annotation while keeping target information that can be recovered later.
  • I3. Task-adaptive pre-training. Continuing masked language modelling on the task's own unlabeled comments before fine-tuning helped a multilingual model handle informal code-mixed text. This is a cheap gain for any low-resource domain.
  • I4. Cross-lingual pooling with imbalance handling. Some teams trained one model on the combined data of all three related languages, adding class weights, class merging or a class-balancing loss. Related low-resource languages can share signal this way, and rare offensive classes are not drowned out.
  • I5. Character plus word features and transliteration. Character sequences combined with words, or transliteration into a common script, reduce sparsity caused by mixed scripts and spelling variation. This is relevant wherever users type a language in non-native script.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksSocial Sciences · 744 worksCommunication · 44 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 608 worksEvaluation & benchmarks · 685 works

Cite

Cite as Bharathi Raja Chakravarthi, Ruba Priyadharshini, Navya Jose, Anand Kumar M, Thomas Mandl, Prasanna Kumar Kumaresan, Rahul Ponnusamy, Hariharan R L, John Philip McCrae and Elizabeth Sherly. 2021. Findings of the Shared Task on Offensive Language Identification in Tamil, Malayalam, and Kannada. Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{chakravarthi2021findings,
  title = {{Findings of the Shared Task on Offensive Language Identification in Tamil, Malayalam, and Kannada}},
  author = {Bharathi Raja Chakravarthi, Ruba Priyadharshini, Navya Jose, Anand Kumar M, Thomas Mandl, Prasanna Kumar Kumaresan, Rahul Ponnusamy, Hariharan R L, John Philip McCrae and Elizabeth Sherly},
  booktitle = {Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages},
  year = {2021},
  url = {https://aclanthology.org/2021.dravidianlangtech-1.17/}
}

References

60 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Devasagayam Albert et al. 1985. Tolkāppiyam Phonology and Morphology: An English Translation, volume 115. International Institute of Tamil Studies. find
  2. JudithJeyafreeda Andrew. 2021. JudithJeyafreedaAndrew@DravidianLangTech-EACL2021:Offensive language detection for Dravidian Code-mixed YouTube comments. In Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages. Association for Computational Linguistics. find
  3. Mikhail Sergeevich Andronov. 1996. A grammar of the Malayalam language in historical treatment, volume 1. Otto Harrassowitz Verlag. find
  4. Vasudev Awatramani. 2021. No Offense@DravidianLangTech-EACL2021: Offensive Tamil Identification and beyond the performance . In Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages. Association for Computational Linguistics. find
  5. Bharathi B and Agnusimmaculate Silvia A. 2021. SS-NCSE NLP@DravidianLangTech-EACL2021: Offensive Language Identification on Multilingual Code Mixing Text. In Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages. Association for Computational Linguistics. find
  6. Fazlourrahman Balouchzahi, Aparna B K, and H L Shashirekha. 2021. MUCS@DravidianLangTech-EACL2021:COOLI-Code-Mixing Offensive Language Identification. In Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages. Association for Computational Linguistics. find
  7. Darina Benikova, Michael Wojatzki, and Torsten Zesch. 2018. What Does This Imply? Examining the Impact of Implicitness on the Perception of Hate Speech. In Language Technologies for the Challenges of the Digital Age, pages 171–179, Cham. Springer International Publishing. find
  8. Stuart H Blackburn. 2006. Print, folklore, and nationalism in colonial South India. Orient Blackswan. find
  9. Bharathi Raja Chakravarthi. 2020. Leveraging orthographic information to improve machine translation of under-resourced languages. Ph.D. thesis, NUI Galway. find
  10. Bharathi Raja Chakravarthi, Mihael Arcan, and John P. McCrae. 2019. Comparison of Different Orthographies for Machine Translation of Under-Resourced Dravidian Languages. In 2nd Conference on Language, Data and Knowledge (LDK 2019), volume 70 of OpenAccess Series in Informatics (OA-SIcs), pages 6:1–6… find
  11. Bharathi Raja Chakravarthi, Navya Jose, Shardul Suryawanshi, Elizabeth Sherly, and John Philip Mc-Crae. 2020a. A sentiment analysis dataset for code-mixed Malayalam-English. In Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboratio…
  12. Bharathi Raja Chakravarthi, M Anand Kumar, John Philip McCrae, Premjith B, Soman KP, and Thomas Mandl. 2020b. Overview of the track on ”HASOC-Offensive Language Identification-DravidianCodeMix”. In Proceedings of the 12th Forum for Information Retrieval Evaluation, FIRE ’20. find
  13. Bharathi Raja Chakravarthi, Vigneshwaran Muralidaran, Ruba Priyadharshini, and John Philip Mc-Crae. 2020c. Corpus creation for sentiment analysis in code-mixed Tamil-English text. In Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collab…
  14. Bharathi Raja Chakravarthi, Ruba Priyadharshini, Vigneshwaran Muralidaran, Shardul Suryawanshi, Navya Jose, Elizabeth Sherly, and John P. McCrae. 2020d. Overview of the track on sentiment analysis for dravidian languages in code-mixed text. In Forum for Information Retrieval Evaluation, FIRE 2020, p… find
  15. Shi Chen and Bing Kong. 2021. cs@DravidianLangTech-EACL2021: Offensive Language Identification Based On Multilingual BERT Model. In Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages. Association for Computational Linguistics. find
  16. Bhargav Dave, Shripad Bhat, and Prasenjit Majumder. 2021. IRNLPDAIICT@DravidianLangTech-EACL2021:Offensive Language identification in Dravidian Languages using TF-IDF Char N-grams and MuRIL . In Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages. Associatio… find
  17. Suman Dowlagar and Radhika Mamidi. 2021. OFFLangOne@DravidianLangTech-EACL2021: Transformers with the Class Balanced Loss for Offensive Language Identification in Dravidian Code-Mixed text . In Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages. Association… find
  18. Simeon Edosomwan, Sitalaskshmi Kalangot Prakasan, Doriane Kouame, Jonelle Watson, and Tom Seymour. 2011. The history of social media and its impact on business. Journal of Applied Management and entrepreneurship, 16(3):79–91. find
  19. Avishek Garain, Atanu Mandal, and Sudip Kumar Naskar. 2021. JUNLP@DravidianLangTech-EACL2021: Offensive Language Identification in Dravidian Langauges . In Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages. Association for Computational Linguistics. find
  20. Karimpumannil Mathai George. 1972. Western influence on Malayalam language and literature. Sahitya Akademi. find
  21. Nikhil Ghanghor, Bharathi Raja Chakravarthi, Ruba Priyadharshini, Sajeetha Thavareesan, and Parameswari Krishnamurthy. 2021. IIITK@DravidianLangTech-EACL2021: Offensive Language Identification and Meme Classification in Tamil, Malayalam and Kannada . In Proceedings of the First Workshop on Speech an… find
  22. Adeep Hande, Ruba Priyadharshini, and Bharathi Raja Chakravarthi. 2020. KanCMD: Kannada CodeMixed dataset for sentiment analysis and offensive language detection. In Proceedings of the Third Workshop on Computational Modeling of People’s Opinions, Personality, and Emotion’s in Social Media, pages 54… find
  23. Bo Huang and Yang Bai. 2021. HUB@DravidianLangTech-EACL2021: Identify and Classify Offensive Text in Multilingual Code Mixing in Social Media. In Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages. Association for Computational Linguistics. find
  24. Sai Muralidhar Jayanthi and Akshat Gupta. 2021. SJAJ@DravidianLangTech-EACL2021: Task-Adaptive Pre-Training of Multilingual BERT models for Offensive Language Identification. In Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages. Association for Computation… find
  25. Navya Jose, Bharathi Raja Chakravarthi, Shardul Suryawanshi, Elizabeth Sherly, and John P. McCrae. 2020. A Survey of Current Datasets for Code-Switching Research. In 2020 6th International Conference on Advanced Computing and Communication Systems (ICACCS). find
  26. Sreelakshmi K, Premjith B, and Soman KP. 2021. AmritaCENNLP@DravidianLangTech-EACL2021: Deep Learning-based Offensive Language Identification in Malayalam, Tamil and Kannada . In Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages. Association for Computatio… find
  27. Kushal Kedia and Abhilash Nandy. 2021. indicnlp@kgp@DravidianLangTech-EACL2021: Offensive Language Identification in Dravidian Languages . In Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages. Association for Computational Linguistics. find
  28. Teo Keipi, Matti Näsi, Atte Oksanen, and Pekka Räsänen. 2016. Online hate and harmful content: Cross-national perspectives. Taylor & Francis. find
  29. Vishnupriya Kolipakam, Fiona M Jordan, Michael Dunn, Simon J Greenhill, Remco Bouckaert, Russell D Gray, and Annemarie Verkerk. 2018. A bayesian phylogenetic study of the dravidian language family. Royal Society open science, 5(3):171504. find
  30. Bhadriraju Krishnamurti. 2003. The Dravidian languages. Cambridge University Press. find
  31. Ritesh Kumar, Atul Kr. Ojha, Shervin Malmasi, and Marcos Zampieri. 2018. Benchmarking aggression identification in social media. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018), pages 1–11, Santa Fe, New Mexico, USA. Association for Computational Linguistic…
  32. Zichao Li. 2021. Codewithzichao@DravidianLangTech-EACL2021: Exploring Multilingual Transformers for Offensive Language Identification on Code Mixing Text . In Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages. Association for Computational Linguistics. of… find
  33. Ishani Maitra and Mary Kate McGowan. 2012. Speech and harm: Controversies over free speech. Oxford University Press on Demand. find
  34. Debjoy Saha, Naman Paharia, Debajit Chakraborty, Punyajoy Saha, and Animesh Mukherjee. 2021. Hate-Alert@DravidianLangTech-EACL2021: Ensembling strategies for Transformer-based Offensive language Detection . In Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Langua… find
  35. Thomas Mandl, Sandip Modha, Anand Kumar M, and Bharathi Raja Chakravarthi. 2020. Overview of the HASOC Track at FIRE 2020: Hate Speech and Offensive Language Identification in Tamil, Malayalam, Hindi, English and German. In Forum for Information Retrieval Evaluation, FIRE 2020, page 29–32, New York,… find
  36. Srinath Nair and Dolton Fernandes. 2021. professionals@DravidianLangTech-EACL2021. In Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages. Association for Computational Linguistics. find
  37. Endang Wahyu Pamungkas, Valerio Basile, and Viviana Patti. 2020. Do You Really Want to Hurt Me? Predicting Abusive Swearing in Social Media. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 6237–6246, Marseille, France. European Language Resources Association. find
  38. Desmond Upton Patton, Robert D. Eschmann, and Dirk A. Butler. 2013. Internet banging: New trends in social media, gang violence, masculinity and hip hop. Computers in Human Behavior, 29(5):A54 – A59. find
  39. MS Purnalingam Pillai. 1904. A Primer of Tamil Literature. Ananda Press. find
  40. Ruba Priyadharshini, Bharathi Raja Chakravarthi, Mani Vegupatti, and John P. McCrae. 2020. Named entity recognition for code-mixed Indian corpus using meta embedding. In 2020 6th International Conference on Advanced Computing and Communication Systems (ICACCS). find
  41. Qinyu Que, Gang Wang, and Shuangjun Jia. 2021. Simon @ DravidianLangTech-EACL2021: Detecting Offensive Content in Kannada Language . In Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages. Association for Computational Linguistics. find
  42. S Rajendran. 2004. Strategies in the formation of compound nouns in tamil. Languages Of India, 4. find
  43. HG Rawlinson. 1930. The kadamba kula—a history of ancient and mediæval karnataka.(studies in indian history of the indian historical research institute, st. xavier’s college, bombay, no. 5). find
  44. Sara Renjit and Sumam Mary Idicula. 2021. CUSATNLP@DravidianLangTech-EACL2021:Language Agnostic Classification of Offensive Content in Tweets. In Proceedings Richard Salomon. 1998. Indian epigraphy: a guide to the study of inscriptions in Sanskrit, Prakrit, and the other Indo-Aryan languages. Oxford… find
  45. A. C. Sekhar. 1951. [Evolution of Malayalam]. Bulletin of the Deccan College Research Institute, 12(1/2):1–216. find
  46. Omar Sharif, Eftekhar Hossain, and Mohammed Moshiul Hoque. 2021. NLP-CUET@DravidianLangTech-EACL2021: Offensive Language Detection from Multilingual Code-Mixed Text using Transformers. In Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages. Association for C… find
  47. R Sivanantham and M Seran. 2019. Keeladi: An Urban Settlement of Sangam Age on the Banks of River Vaigai. India: Department of Archaeology, Government of Tamil Nadu, Chennai. find
  48. BGL Swamy. 1975. The Date of Tolkappiyam: A Retrospect. Annals of Oriental Research (Madras), Silver Jubilee, 292:317. find
  49. Takanobu Takahashi. 1995. Tamil love poetry and poetics, volume 9. Brill. find
  50. Sajeetha Thavareesan and Sinnathamby Mahesan. 2019. Sentiment Analysis in Tamil Texts: A Study on Machine Learning Techniques and Feature Representation. In 2019 14th Conference on Industrial and Information Systems (ICIIS), pages 320–325. find
  51. Sajeetha Thavareesan and Sinnathamby Mahesan. 2020a. Sentiment Lexicon Expansion using Word2vec and fastText for Sentiment Prediction in Tamil texts. In 2020 Moratuwa Engineering Research Conference (MERCon), pages 272–276. find
  52. Sajeetha Thavareesan and Sinnathamby Mahesan. 2020b. Word embedding-based Part of Speech tagging in Tamil texts. In 2020 IEEE 15th International Conference on Industrial and Information Systems (ICIIS), pages 478–482. find
  53. Debapriya Tula, Prathyush Potluri, Shreyas MS, Sumanth Doddapaneni, Pranjal Sahu, Rohan Sukumaran, and Parth Patwa. 2021. Bitions@DravidianLangTech-EACL2021: Ensemble of Multilingual Language Models with Pseudo Labeling for Offense Detection in Dravidian Languages. In Proceedings of the First Worksh… find
  54. Charangan Vasantharajan and Uthayasanker Thayasivam. 2021. Hypers@DravidianLangTech-EACL2021: Offensive language identification in Dravidian code-mixed YouTube Comments and Posts. In Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages. Association for Comput… find
  55. Maoqin Yang. 2021. Maoqin @ DravidianLangTech-EACL2021: The Application of Transformer-Based Model . In Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages. Association for Computational Linguistics. find
  56. Konthala Yasaswini, Karthik Puranik, Adeep Hande, Ruba Priyadharshini, Sajeetha Thavareesan, and Bharathi Raja Chakravarthi. 2021. IIITT@DravidianLangTech-EACL2021: Transfer Learning for Offensive Language Detection in Dravidian Languages . In Proceedings of the First Workshop on Speech and Language… find
  57. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. Predicting the type and target of offensive posts in social media. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Languag…
  58. Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Çağrı Çöltekin. 2020. SemEval-2020 task 12: Multilingual offensive language identification in social media (OffensEval 2020). In Proceedings of the Fourteenth Work… find
  59. Yingjia Zhao. 2021. ZYJ123@DravidianLangTech-EACL2021: Offensive Language Identification based on XLM-RoBERTa with DPCNN. In Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages. Association for Computational Linguistics. find
  60. Kamil V Zvelebil. 1991. Comments on the Tolkappiyam Theory of Literature. Archı́v Orientálnı́, 59:345–359. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chakravarthi-2021-findings-shared-task-offensive-language/