Elamur Research⌕
☰
home / papers / 2021 / Findings of the Shared Task on Hope Speech Detection for Equality, Diversity, and Inclusion

Findings of the Shared Task on Hope Speech Detection for Equality, Diversity, and Inclusion

Summary

This paper reports on the first shared task on hope speech, held at the LT-EDI-2021 workshop (EACL 2021). The organisers argue that social-media NLP has focused on suppressing harmful content, which can restrict how some groups express themselves. They propose detecting and promoting supportive, encouraging, future-oriented messages instead. Participants received the HopeEDI corpus, which contains YouTube comments in English and in code-mixed Tamil and Malayalam. The comments were gathered around EDI-related topics such as LGBTIQ issues, COVID-19, women in STEM and Black Lives Matter. Each comment had to be labelled as hope speech, non-hope speech, or not in the target language. Systems were ranked by weighted F1 on a held-out test set through CodaLab. About thirty teams submitted per language, and the best entries mostly fine-tuned XLM-RoBERTa (or RoBERTa/BERT for English). Top scores were 0.93 for English, 0.85 for Malayalam and only 0.61 for Tamil. The authors link the Tamil gap to code-mixing and a more balanced, truly three-way label distribution. The task leaves an open benchmark for positive-content detection in under-resourced languages.

Key points

  • HopeEDI has 59,354 comments: 28,451 English, 20,198 Tamil and 10,705 Malayalam. Data were split 80/10/10 into train, development and test sets.
  • Each comment was labelled by at least three annotators who were briefed on EDI. Krippendorff's alpha was 0.63 (English), 0.76 (Tamil) and 0.85 (Malayalam).
  • Classes are skewed in English and Malayalam:
    • English has 2,484 hope vs 25,940 non-hope, with only 27 other-language comments.
    • Tamil is closer to balanced, with 7,899 hope, 9,816 non-hope and 2,483 not-Tamil comments.
  • 137 participants registered, and roughly 30–31 teams per language submitted final runs. The official metric was weighted F1, with sklearn's classification report recommended for checking.
  • Score ranges were 0.61–0.93 for English, 0.49–0.85 for Malayalam and 0.37–0.61 for Tamil. Baselines were Tf-Idf features with standard sklearn classifiers (SVM, KNN, decision tree, logistic regression, multinomial Naive Bayes).

Interesting ideas

  • I1. Positive framing of a moderation problem. Instead of only flagging harmful content, the task identifies content worth amplifying. The same reframing could work for other moderation or recommendation tasks where pure filtering risks silencing dialects or minority voices.
  • I2. An explicit "not target language" class. Off-language comments get their own label rather than being silently dropped. This makes language mixing a measurable part of the problem, which is useful for any corpus with heavy code-mixing or script switching.
  • I3. Language ID first, then classification. Some teams first predicted the language of a comment and only then the content label. This is a simple modular design for mixed-language data, where routing errors can be diagnosed separately from content errors.
  • I4. Synthetic code-mixed training data. One team generated artificial code-mixed text to pre-train a RoBERTa model from scratch for Tamil. This could help any low-resource setting where natural mixed-language text is scarce.
  • I5. Explaining score gaps through label distribution. The organisers note that English and Malayalam behave almost like binary tasks, while Tamil is genuinely three-way. Comparing class distributions before comparing scores across languages helps avoid misleading conclusions about model quality.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · Society & ethics

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{chakravarthi2021findings,
  title = {{Findings of the Shared Task on Hope Speech Detection for Equality, Diversity, and Inclusion}},
  author = {Chakravarthi, Bharathi Raja and Muralidaran, Vigneshwaran},
  booktitle = {Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion},
  year = {2021},
  url = {https://aclanthology.org/2021.ltedi-1.8/}
}

References

41 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Vasudev Awatramani. 2021. Hopeful NLP@LT-EDI-EACL2021: Finding Hope in YouTube Comment Section. In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, Online. Association for Computational Linguistics. find
  2. Fazlourrahman Balouchzahi, Aparna B K, and H L Shashirekha. 2021. MUCS@LT-EDI-Suman Dowlagar and Radhika Mamidi. 2021. EDIOne@LT-EDI-EACL2021: Pre-trained Transformers with Convolutional Neural Networks for Hope Speech Detection. In Proceedings of the First Workshop on Language Technology for Equali… find
  3. EACL2021:CoHope-Hope Speech Detection for Equality, Diversity, and Inclusion in Code-Mixed Texts. In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, Online. Association for Computational Linguistics. find
  4. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. ” O’Reilly Media, Inc.”. find
  5. Nikhil Kumar Ghanghor, Parameswari Krishnamurthy, Sajeetha Thavareesan, Ruba Priyadharshini, and Bharathi Raja Chakravarthi. 2021a. IIITK@DravidianLangTech-EACL2021: Offensive Language Identification and Meme Classification in Tamil, Malayalam and Kannada. In Proceedings of the First Workshop on Spe… find
  6. Bharathi Raja Chakravarthi. 2020a. HopeEDI: A multilingual hope speech detection dataset for equality, diversity, and inclusion. In Proceedings of the Third Workshop on Computational Modeling of People’s Opinions, Personality, and Emotion’s in Social Media, pages 41–53, Barcelona, Spain (Online). As… find our card
  7. Nikhil Kumar Ghanghor, Rahul Ponnusamy, Prasanna Kumar Kumaresan, Ruba Priyadharshini, Sajeetha Thavareesan, and Bharathi Raja Chakravarthi. 2021b. IIITK@LT-EDI-EACL2021: Hope Speech Detection for Equality, Diversity, and Inclusion in Tamil, Malayalam and English. In Proceedings of the First Worksho… find
  8. Bharathi Raja Chakravarthi. 2020b. Leveraging orthographic information to improve machine translation of under-resourced languages. Ph.D. thesis, NUI Galway. find
  9. Bharathi Raja Chakravarthi, Navya Jose, Shardul Suryawanshi, Elizabeth Sherly, and John P Mc-Crae. 2020. A sentiment analysis dataset for code-mixed Malayalam-English. In Proceedings of the 1st Joint Workshop of SLTU (Spoken Language Technologies for Under-resourced languages) and CCURL (Collaborati… find our card
  10. Sunil Gundapu and Radhika Mamidi. 2021. Autobots@LT-EDI-EACL2021: All Lives Matter! Hope Speech Detection with BERT Transformer Model. In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, Online. Association for Computational Linguistics. find
  11. Siddhanth U Hegde, Adeep Hande, Ruba Priyadharshini, Sajeetha Thavareesan, and Bharathi Raja Chakravarthi. 2021. UVCE-IIITT@DravidianLangTech-EACL2021: Tamil Troll Meme Classification: You need to Pay more Attention. In Proceedings of the First Workshop on Speech and Language Technologies for Dravid… find
  12. Bharathi Raja Chakravarthi and Vigneshwaran Muralidaran. 2021. Findings of the shared task on Hope Speech Detection for Equality, Diversity, and Inclusion. In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion. Association for Computational Linguistics. find
  13. Shi Chen and Bing Kong. 2021. cs-english@LT-EDI-EACL2021: Hope Speech Detection Based On Fine-tuning AlBERT Model. In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, Online. Association for Computational Linguistics. find
  14. Eftekhar Hossain, Omar Sharif, and Mohammed Moshiul Hoque. 2021. NLP-CUET@LT-EDI-EACL2021: Multilingual Code-Mixed Hope Speech Detection using Cross-lingual Representation Learner . In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, Online. Association… find
  15. Dhivya Chinnappa. 2021. Multilingual Hope Speech Detection for Code-mixed and Transliterated Texts. In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, Online. Association for Computational Linguistics. find
  16. Bo Huang and Yang Bai. 2021. TEAM HUB@LT-EDI-EACL2021: Hope Speech Detection Based On Pre-trained Language Model. In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, Online. Association for Computational Linguistics. find
  17. Bhargav Dave, Shripad Bhat, and Prasenjit Majumder. 2021. IRNLP-DAIICT@LT-EDI-EACL2021: Hope Speech detection in Code Mixed text using TF-IDF Char N-grams and MuRIL. In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, Online. Association for Computation… find
  18. Vishnupriya Kolipakam, Fiona M Jordan, Michael Dunn, Simon J Greenhill, Remco Bouckaert, Russell D Gray, and Annemarie Verkerk. 2018. find
  19. A Thara S, Ravi Teja Tasubilli, and Kothamasu Sai Rahul. 2021b. Amrita@LT-EDI-EACL2021: Hope Speech Detection on Multilingual Text. In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, Online. Association for Computational Linguistics. find
  20. Bayesian phylogenetic study of the Dravidian language family. Royal Society open science, 5(3):171504. find
  21. Bhadriraju Krishnamurti. 2003. The Dravidian languages. Cambridge University Press. find
  22. Ritesh Kumar, Atul Kr. Ojha, Shervin Malmasi, and Marcos Zampieri. 2018. Benchmarking aggression identification in social media. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018), pages 1–11, Santa Fe, New Mexico, USA. Association for Computational Linguistic… find our card
  23. Sunil Saumya and Ankit Kumar Mishra. 2021. IIIT-DWD@LT-EDI-EACL2021: Hope Speech Detection in YouTube multilingual comments. In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, Online. Association for Computational Linguistics. find
  24. Megha Sharma and Gaurav Arora. 2021. Spartans@LT-EDI-EACL2021: Inclusive Speech Detection using Pretrained Language Models. In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, Online. Association for Computational Linguistics. find
  25. Junaida M K and Ajees A P. 2021. KU-NLP@LT-EDI-EACL2021: A Multilingual Hope Speech Detection for Equality, Diversity, and Inclusion using Context Aware Embeddings. In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, Online. Association for Computationa… find
  26. Ramya Srinivasan. 2019. Reinventing tamil script. find
  27. Iravatham Mahadevan. 2003. Early tamil epigraphy. From the earliest times to the sixth century ad. find
  28. Sajeetha Thavareesan and Sinnathamby Mahesan. 2019. Sentiment Analysis in Tamil Texts: A Study on Machine Learning Techniques and Feature Representation. In 2019 14th Conference on Industrial and Information Systems (ICIIS), pages 320–325. find
  29. Khyati Mahajan, Erfan Al-Hossami, and Samira Shaikh. 2021. TeamUNCC@LT-EDI-EACL2021: Hope Speech Detection using Transfer Learning with Transformers. In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, Online. Association for Computational Linguistics. find
  30. Sajeetha Thavareesan and Sinnathamby Mahesan. 2020a. Sentiment Lexicon Expansion using Word2vec and fastText for Sentiment Prediction in Tamil texts. In 2020 Moratuwa Engineering Research Conference (MERCon), pages 272–276. find
  31. Thomas Mandl, Sandip Modha, Anand Kumar M, and Bharathi Raja Chakravarthi. 2020. Overview of the HASOC Track at FIRE 2020: Hate Speech and Offensive Language Identification in Tamil, Malayalam, Hindi, English and German. In Forum for Information Retrieval Evaluation, FIRE 2020, page 29–32, New York,… find
  32. Sajeetha Thavareesan and Sinnathamby Mahesan. 2020b. Word embedding-based Part of Speech tagging in Tamil texts. In 2020 IEEE 15th International Conference on Industrial and Information Systems (ICIIS), pages 478–482. find
  33. Ishan Sanjeev Upadhyay, Nikhil E, Anshul Wadhawan, and Radhika Mamidi. 2021. Hopeful Men@LT-EDI-EACL2021: Hope Speech Detection Using Indic Transliteration and Transformers. In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, Online. Association for Com… find
  34. Karthik Puranik, Adeep Hande, Ruba Priyadharshini, Sajeetha Thavareesan, and Bharathi Raja Chakravarthi. 2021. IIITT@LT-EDI-EACL2021-Hope Speech Detection: There is always hope in Transformers. In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, Online.… find
  35. Konthala Yasaswini, Karthik Puranik, Adeep Hande, Ruba Priyadharshini, Sajeetha Thavareesan, and Bharathi Raja Chakravarthi. 2021. IIITT@DravidianLangTech-EACL2021: Transfer Learning for Offensive Language Detection in Dravidian Languages. In Proceedings of the First Workshop on Speech and Language… find
  36. Qinyu Que. 2021. Simon @ LT-EDI-EACL2021: Detecting Hope Speech with BERT. In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, Online. Association for Computational Linguistics. find
  37. Arunima S, Akshay Ramakrishnan, Avantika Balaji, Thenmozhi D, and Senthil Kumar B. 2021a. ssn-diBERTsity@LT-EDI-EACL2021:Hope Speech Detection on multilingual YouTube comments via transformer based approach. In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Incl… find
  38. Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Çağrı Çöltekin. 2020. SemEval-2020 task 12: Multilingual offensive language identification in social media (OffensEval 2020). In Proceedings of the Fourteenth Work… find
  39. Yingjia Zhao. 2021. ZYJ@LT-EDI-EACL2021:XLM-RoBERTa-Based Model with Attention for Hope Speech Detection. In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, Online. Association for Computational Linguistics. find
  40. Siyao Zhou. 2021. Zeus@LT-EDI-EACL2021: Hope speech detection based on Pre-training Mode. In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, Online. Association for Computational Linguistics. find
  41. Stefan Ziehe, Franziska Pannach, and Aravind Krishnan. 2021. cs-english@GCDH@LT-EDI-EACL2021: XLM-RoBERTa for Hope Speech Detection in English, Malayalam, and Tamil. In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, Online. Association for Computation… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chakravarthi-2021-findings-shared-task-hope-speech/