Elamur Research⌕
☰
home / papers / 2020 / HopeEDI: A Multilingual Hope Speech Detection Dataset for Equality, Diversity, and Inclusion

HopeEDI: A Multilingual Hope Speech Detection Dataset for Equality, Diversity, and Inclusion

Summary

Most work on moderating social media targets harmful content (hate, abuse, offensive language) and removes it. The author argues this can restrict free expression and carry dataset biases that hurt minorities. The paper instead proposes detecting and promoting hope speech: supportive, encouraging and inclusive comments, especially those related to equality, diversity and inclusion (EDI) for groups such as LGBTIQ people, women in STEM, racial minorities and people with disabilities.

The resulting resource, HopeEDI, contains YouTube comments in English, code-switched Tamil and code-switched Malayalam. Each comment was manually labelled as hope speech, not hope speech, or other language, following detailed annotator guidelines. Agreement was measured with Krippendorff's alpha.

Baselines are classical sklearn classifiers on Tf-Idf features, evaluated with macro-averaged precision, recall and F1. All baselines perform modestly, mainly because the classes are imbalanced, so the task remains open.

The dataset matters because it is presented as the first public, multilingual hope speech corpus for EDI. It also covers two under-resourced, code-mixed languages.

Key points

  • HopeEDI sizes: 28,451 English, 20,198 Tamil and 10,705 Malayalam comments, all from YouTube.
  • Labels are Hope Speech, Not-Hope Speech and Other language.
    • The Other language label is kept because the Tamil and Malayalam comments are code-mixed and often written in Latin script.
  • The guidelines treat non-hope speech as distinct from hate speech: a neutral or sceptical remark can be non-hope without being hateful.
  • Baselines use Tf-Idf features with these models:
    • Multinomial Naive Bayes (alpha = 0.7).
    • KNN, SVM, decision tree and logistic regression, tuned by grid search.
    • Models were trained on the train split, tuned on the dev split and scored on a held-out test split.
  • Baseline results (macro F1):
    • SVM was weakest: 0.32 (English), 0.21 (Tamil), 0.28 (Malayalam).
    • Decision tree was best for English (0.46) and Malayalam (0.56).
    • Logistic regression was best for Tamil (0.55).
    • The English test set is heavily skewed: 250 hope vs. 2,593 non-hope comments.

Interesting ideas

  • I1. Detect the positive class instead of only filtering the negative one. Surfacing supportive content rather than deleting hostile content avoids some censorship concerns. This framing could inspire positive-content datasets for other communities or crisis contexts.
  • I2. Write the label definition as a checklist of concrete conditions with short examples. Examples include peace-seeking statements, survival stories and calls for fairness. Spelling the category out this way makes a fuzzy concept like "hope" annotatable and is a good template for any subjective labelling task.
  • I3. Explicitly separate "not positive" from "hateful". The guidelines give examples of neutral non-hope comments. This prevents annotators and models from collapsing the task into a sentiment or toxicity proxy, a useful safeguard for any opposite-of-toxicity classifier.
  • I4. Keep an "Other language" label for code-mixed, non-native-script data. Automatic language identification alone was not enough here, so annotators flagged leftover off-target comments. Any corpus of transliterated or mixed-language social media text could benefit from the same option.
  • I5. Report macro-averaged and per-class scores alongside weighted ones. On imbalanced data, accuracy and weighted scores hide failures on the minority class. For example, SVM looks acceptable on weighted metrics but never predicts hope speech, which only per-class reporting exposes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

115 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 113 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{chakravarthi2020hopeedi,
  title = {{HopeEDI: A Multilingual Hope Speech Detection Dataset for Equality, Diversity, and Inclusion}},
  author = {Chakravarthi, Bharathi Raja},
  booktitle = {Proceedings of the Third Workshop on Computational Modeling of People's Opinions, Personality, and Emotion's in Social Media},
  year = {2020},
  url = {https://aclanthology.org/2020.peoples-1.5/}
}

References

57 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Edgar Altszyler, Ariel J. Berenstein, David Milne, Rafael A. Calvo, and Diego Fernandez Slezak. 2018. Using contextual information for automatic triage of posts in a peer-support forum. In Proceedings of the Fifth Workshop on Computational Linguistics and Clinical Psychology: From Keyboard to Clinic… find
  2. Kalika Bali, Jatin Sharma, Monojit Choudhury, and Yogarshi Vyas. 2014. “I am borrowing ya mixing ?” an analysis of English-Hindi code mixing in Facebook. In Proceedings of the First Workshop on Computational Approaches to Code Switching, pages 116–126, Doha, Qatar, October. Association for Computati… find
  3. Shubhanker Banerjee, Bharathi Raja Chakravarthi, and John Philip McCrae. 2020. Comparison of pretrained embeddings to identify hate speech in Indian code-mixed text. In 2nd IEEE International Conference on Advances in Computing, Communication Control and Networking –ICACCCN (ICAC3N-20). find
  4. Utsab Barman, Joachim Wagner, Grzegorz Chrupała, and Jennifer Foster. 2014. DCU-UVT: Word-level language classification with code-mixed data. In Proceedings of the First Workshop on Computational Approaches to Code Switching, pages 127–132, Doha, Qatar, October. Association for Computational Linguis… find
  5. Pete Burnap, Gualtiero Colombo, Rosie Amery, Andrei Hodorog, and Jonathan Scourfield. 2017. Multi-class machine classification of suicide-related communication on twitter. Online Social Networks and Media, 2:32 – 44. find
  6. Bharathi Raja Chakravarthi, Mihael Arcan, and John P. McCrae. 2019. WordNet gloss translation for under-resourced languages using multilingual neural machine translation. In Proceedings of the Second Workshop on Multilingualism at the Intersection of Knowledge Bases and Machine Translation, pages 1–… find
  7. Bharathi Raja Chakravarthi, Navya Jose, Shardul Suryawanshi, Elizabeth Sherly, and John Philip McCrae. 2020a. A sentiment analysis dataset for code-mixed Malayalam-English. In Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration… find our card
  8. Bharathi Raja Chakravarthi, Vigneshwaran Muralidaran, Ruba Priyadharshini, and John Philip McCrae. 2020b. Corpus creation for sentiment analysis in code-mixed Tamil-English text. In Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collabo… find our card
  9. Bharathi Raja Chakravarthi, Navaneethan Rajasekaran, Mihael Arcan, Kevin McGuinness, Noel E.O’Connor, and John P McCrae. 2020c. Bilingual lexicon induction across orthographically-distinct under-resourced Dravidian languages. In Proceedings of the Seventh Workshop on NLP for Similar Languages, Varie… find
  10. Bharathi Raja Chakravarthi. 2020. Leveraging orthographic information to improve machine translation of under-resourced languages. Ph.D. thesis, NUI Galway. find
  11. Edward C. Chang. 1998. Hope, problem-solving ability, and coping in a college student population: Some implications for theory and practice. Journal of Clinical Psychology, 54(7):953–962. find
  12. Yi-Ling Chung, Elizaveta Kuzmenko, Serra Sinem Tekiroglu, and Marco Guerini. 2019. CONAN - COunter NArratives through nichesourcing: a multilingual dataset of responses to fight online hate speech. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2819… find our card
  13. Jae Eun Chung. 2013. Social networking in online support groups for health: How online social networking benefits patients. Journal of Health Communication, 19(6):639–659, April. find
  14. Isobelle Clarke and Jack Grieve. 2017. Dimensions of abusive language on twitter. In Proceedings of the First Workshop on Abusive Language Online, pages 1–10, Vancouver, BC, Canada, August. Association for Computational Linguistics. find
  15. Rob Cover. 2013. Queer youth resilience: Critiquing the discourse of hope and hopelessness in lgbt suicide representation. M/C Journal, 16(5). find
  16. Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. 2019. Racial bias in hate speech and abusive language detection datasets. In Proceedings of the Third Workshop on Abusive Language Online, pages 25–35, Florence, Italy, August. Association for Computational Linguistics. find our card
  17. Kris Gowen, Matthew Deschaine, Darcy Gruttadara, and Dana Markey. 2012. Young adults with mental health conditions and social networking websites: Seeking tools to build community. Psychiatric Rehabilitation Journal, 35(3):245–250. find
  18. Deepak Gupta, Pabitra Lenka, Asif Ekbal, and Pushpak Bhattacharyya. 2018. Uncovering code-mixed challenges: A framework for linguistically driven question generation and neural based question answering. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 119–130,… find
  19. Adeep Hande, Ruba Priyadharshini, and Bharathi Raja Chakravarthi. 2020. KanCMD: Kannada codemixed dataset for sentiment analysis and offensive language detection. In Proceedings of the Third Workshop on Computational Modeling of People’s Opinions, Personality, and Emotions in Social Media, Barcelona… find
  20. Henning Herrestad and Stian Biong. 2010. Relational hopes: A study of the lived experience of hope in some patients hospitalized for intentional self-harm. International Journal of Qualitative Studies on Health and Well-being, 5(1):4651. PMID: 20640026. find
  21. Navya Jose, Bharathi Raja Chakravarthi, Shardul Suryawanshi, Elizabeth Sherly, and John P. McCrae. 2020. A survey of current datasets for code-switching research. In 2020 6th International Conference on Advanced Computing and Communication Systems (ICACCS). find
  22. Vanessa Kitzie. 2018. ”i pretended to be a boy on the internet”: Navigating affordances and constraints of social networking sites and search engines for lgbtq+ identity work. First Monday, 23(7), Jul. find
  23. Klaus Krippendorff. 1970. Estimating the reliability, systematic error and random error of interval data. Educational and Psychological Measurement, 30(1):61–70. find
  24. Klaus Krippendorff. 2011. Computing krippendorff’s alpha-reliability. find
  25. Amar Krishna, Joseph Zambreno, and Sandeep Krishnan. 2013. Polarity Trend Analysis of Public Sentiment on YouTube. In Proceedings of the 19th International Conference on Management of Data, COMAD ’13, page 125–128, Mumbai, Maharashtra, IND. Computer Society of India. find
  26. Younghun Lee, Seunghyun Yoon, and Kyomin Jung. 2018. Comparative studies of detecting abusive language on twitter. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2), pages 101–106, Brussels, Belgium, October. Association for Computational Linguistics. find
  27. Edison Marrese-Taylor, Jorge Balazs, and Yutaka Matsuo. 2017. Mining fine-grained opinions on closed captions of YouTube videos with an attention-RNN. In Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 102–111, Copenhagen, Denma… find
  28. Binny Mathew, Punyajoy Saha, Hardik Tharad, Subham Rajgaria, Prajwal Singhania, Suman Kalyan Maity, Pawan Goyal, and Animesh Mukherjee. 2019. Thou shalt not hate: Countering online hate speech. Proceedings of the International AAAI Conference on Web and Social Media, 13(01):369–380, Jul. find
  29. Harvey Milk. 1997. The hope speech. We are everywhere: A historical sourcebook of gay and lesbian politics, pages 51–53. find
  30. David N. Milne, Glen Pink, Ben Hachey, and Rafael A. Calvo. 2016. find
  31. CLPsych 2016 shared task: Triaging content in online peer-support forums. In Proceedings of the Third Workshop on Computational Linguistics and Clinical Psychology, pages 118–127, San Diego, CA, USA, June. Association for Computational Linguistics. find
  32. Skanda Muralidhar, Laurent Nguyen, and Daniel Gatica-Perez. 2018. Words worth: Verbal content and hirability impressions in YouTube video resumes. In Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 322–327, Brussels, Belgium, Oc… find
  33. Cicero Nogueira dos Santos, Igor Melnyk, and Inkit Padhi. 2018. Fighting offensive language on social media with unsupervised text style transfer. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 189–194, Melbourne, Australia,… find our card
  34. Nedjma Ousidhoum, Zizheng Lin, Hongming Zhang, Yangqiu Song, and Dit-Yan Yeung. 2019. Multilingual and multi-aspect hate speech analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processin… find our card
  35. Shriphani Palakodety, Ashiqur R KhudaBukhsh, and Jaime G Carbonell. 2020a. Hope speech detection: A computational analysis of the voice of peace. In Proceedings of the 24th European Conference on Artificial Intelligence - ECAI 2020. find
  36. Shriphani Palakodety, Ashiqur R KhudaBukhsh, and Jaime G Carbonell. 2020b. Voice for the voiceless: Active sampling to detect comments supporting the rohingyas. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 454–462. find
  37. Marcelo O. R. Prates, Pedro H. Avelar, and Luı́s C. Lamb. 2020. Assessing gender bias in machine translation: a case study with google translate. Neural Computing and Applications, 32(10):6363–6381, May. find
  38. Ruba Priyadharshini, Bharathi Raja Chakravarthi, Mani Vegupatti, and John P. McCrae. 2020. Named entity recognition for code-mixed Indian corpus using meta embedding. In 2020 6th International Conference on Advanced Computing and Communication Systems (ICACCS). find
  39. Laura Robinson, Jeremy Schulz, Grant Blank, Massimo Ragnedda, Hiroshi Ono, Bernie Hogan, Gustavo S. Mesch, Shelia R. Cotten, Susan B. Kretchmer, Timothy M. Hale, Tomasz Drabowicz, Pu Yan, Barry Wellman, Molly-Gloria Harper, Anabel Quan-Haase, Hopeton S. Dunn, Antonio A. Casilli, Paola Tubaro, Rod Ca… find
  40. Anna Schmidt and Michael Wiegand. 2017. A survey on hate speech detection using natural language processing. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, pages 1–10, Valencia, Spain, April. Association for Computational Linguistics. find our card
  41. Anne-Marie Di Sciullo, Pieter Muysken, and Rajendra Singh. 1986. Government and code-mixing. Journal of Linguistics, 22(1):1–24. find
  42. Aliaksei Severyn, Alessandro Moschitti, Olga Uryupina, Barbara Plank, and Katja Filippova. 2014. Opinion mining on YouTube. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1252–1261, Baltimore, Maryland, June. Association for… find
  43. Gudbjartur Ingi Sigurbergsson and Leon Derczynski. 2020. Offensive language and hate speech detection for Danish. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 3498–3508, Marseille, France, May. European Language Resources Association. find our card
  44. Charles R Snyder, Kevin L Rand, and David R Sigmon. 2002. Hope theory: A member of the positive psychology family. find
  45. Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating gender bias in natural language processing: Literature review. In Proceedings of the 57th Annual Meeting of the Association for Computati… find our card
  46. Rachael Tatman. 2017. Gender and dialect bias in YouTube’s automatic captions. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing, pages 53–59, Valencia, Spain, April. Association for Computational Linguistics. find our card
  47. Serra Sinem Tekiroğlu, Yi-Ling Chung, and Marco Guerini. 2020. Generating counter narratives against online hate speech: Data and strategies. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1177–1190, Online, July. Association for Computational Ling… find
  48. Giuliano Tortoreto, Evgeny Stepanov, Alessandra Cervone, Mateusz Dubiel, and Giuseppe Riccardi. 2019. Affective behaviour analysis of on-line user interactions: Are on-line support groups more therapeutic than twitter? In Proceedings of the Fourth Social Media Mining for Health Applications (\#SMM4H)… find
  49. Eva Vanmassenhove, Christian Hardmeier, and Andy Way. 2018. Getting gender right in neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3003–3008, Brussels, Belgium, October-November. Association for Computational Linguistics. find our card
  50. Zijian Wang and David Jurgens. 2018. It’s going to be okay: Measuring access to support in online communities. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 33–45, Brussels, Belgium, October-November. Association for Computational Linguistics. find
  51. Zeerak Waseem, Thomas Davidson, Dana Warmsley, and Ingmar Weber. 2017. Understanding abuse: A typology of abusive language detection subtasks. In Proceedings of the First Workshop on Abusive Language Online, pages 78–84, Vancouver, BC, Canada, August. Association for Computational Linguistics. find our card
  52. Michael Wiegand, Josef Ruppenhofer, and Thomas Kleinbauer. 2019. Detection of Abusive Language: the Problem of Biased Datasets. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short… find our card
  53. Mengzhou Xia, Anjalie Field, and Yulia Tsvetkov. 2020. Demoting racial bias in hate speech detection. In Proceedings of the Eighth International Workshop on Natural Language Processing for Social Media, pages 7–14, Online, July. Association for Computational Linguistics. find
  54. Andrew Yates, Arman Cohan, and Nazli Goharian. 2017. Depression and self-harm risk assessment in online forums. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2968–2978, Copenhagen, Denmark, September. Association for Computational Linguistics. find our card
  55. Carolyn M. Youssef and Fred Luthans. 2007. Positive organizational behavior in the workplace: The impact of hope, optimism, and resilience. Journal of Management, 33(5):774–800. find
  56. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019a. Predicting the type and target of offensive posts in social media. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua… find our card
  57. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019b. SemEval-2019 task 6: Identifying and categorizing offensive language in social media (OffensEval). In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 75–86, Minneapolis… find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chakravarthi-2020-hopeedi-multilingual-hope-speech-detection/