Elamur Research⌕
☰
home / papers / 2022 / CURAJ_IIITDWD@LT-EDI-ACL 2022: Hope Speech Detection in English YouTube Comments using Deep Learning Techniques

CURAJ_IIITDWD@LT-EDI-ACL 2022: Hope Speech Detection in English YouTube Comments using Deep Learning Techniques

Paper · 2022 · Vanshita Jha, Ankit Mishra and Sunil Saumya

Summary

The paper describes a system submitted to the English track of the LT-EDI-ACL 2022 competition, whose goal was to identify hope speech in support of equality, diversity, and inclusion. The task is binary: decide whether a YouTube comment is encouraging and supportive ("hope speech") or not. The English data is strongly imbalanced, with fewer than one comment in ten labelled as hope. The authors compared several compact neural classifiers: a three-layer dense network (DNN), a single-layer CNN, Bi-LSTM and GRU, and stacked LSTM-CNN and LSTM-LSTM. All models shared the same front end, which lowercases the text, strips noise and stopwords, maps words to indices, pads each comment to ten tokens, and feeds it to a trainable 300-dimensional embedding. Only the feature extractor changed between models. The plain DNN did best, with a macro F1 of 0.67 on both the development and test sets. The work is a modest baseline for a socially motivated classification task. It also shows how large the gap between weighted and macro scores can be when one class is rare.

Key points

  • English data sizes: training has 22,740 comments (1,962 hope, 20,778 non-hope), development has 2,841 (272 hope), and test has 2,843 (250 hope).
  • Preprocessing steps:
    • lowercasing;
    • removal of numbers, punctuation, symbols, emojis and links;
    • removal of stopwords with nltk;
    • index encoding and padding or cutting to a length of 10. The vocabulary has about 20,255 entries.
  • Embeddings with random initialization worked better than pretrained Word2Vec or GloVe weights, so only the random setup is reported.
  • Best model: a DNN with three layers (ReLU inside, sigmoid output), built in Keras and sklearn. It reached macro F1 0.67 and weighted F1 0.89 on both dev and test. The other architectures scored macro F1 0.65–0.67.
  • Per-class F1 is about 0.94 for one class and about 0.39–0.40 for the other. The table labels this low score "Non-Hope", but because hope is the minority class, the labels look swapped.

Interesting ideas

  • I1. The paper reports both weighted and macro averages, and they differ a lot (0.89 vs 0.67). The weighted score hides that the rare class is handled poorly, which is a reminder to always report per-class and macro figures on skewed data.
  • I2. The comparison is controlled: one fixed pipeline, and only the feature-extraction block changes. This makes it clear that architecture choice mattered little for this task (all within about 0.02 F1).
  • I3. The authors checked whether the development score held on the released test labels (0.67 on both). This is a simple test for overfitting to the dev set.
  • I4. The authors themselves list cutting comments to ten tokens and removing stopwords and emojis as choices to revisit. Discarding function words and affective symbols can erase exactly the cues that signal encouragement.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluations of rare phenomena, such as irony or a specific speech act, should report the minority class separately and in macro form, so that a high majority-class score does not hide failure.
  • I2 (learn): When testing parser or tagger variants, keep the input, tokenization and scoring fixed and swap one component at a time, so that differences can be attributed cleanly.
  • I3 (learn): Keep a held-out set that is scored only after design decisions are frozen, as a guard against tuning repair rules or teacher thresholds to the dev data.
  • I4 (frames): Speech-act and attitude detection should keep negation, modals, pronouns and emoji tokens, and attach them to the UD tree rather than filtering them out early.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Indian Institute of Information Technology DharwadDharwad · India4 works · 6 citations🎓Goa UniversityTaleigao · India3 works · 4 citations

Also on the byline: Central University of Rajasthan.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksPsychology · 1,552 worksSocial Psychology · 306 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Vanshita Jha, Ankit Mishra and Sunil Saumya. 2022. CURAJ_IIITDWD@LT-EDI-ACL 2022: Hope Speech Detection in English YouTube Comments using Deep Learning Techniques. Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.ltedi-1.25

@inproceedings{jha2022curaj,
  title = {{CURAJ\_IIITDWD@LT-EDI-ACL 2022: Hope Speech Detection in English YouTube Comments using Deep Learning Techniques}},
  author = {Vanshita Jha, Ankit Mishra and Sunil Saumya},
  booktitle = {Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion},
  year = {2022},
  doi = {10.18653/v1/2022.ltedi-1.25},
  url = {https://doi.org/10.18653/v1/2022.ltedi-1.25}
}

References

23 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. B Bharathi, Bharathi Raja Chakravarthi, Subalalitha Chinnaudayar Navaneethakrishnan, N Sripriya, Arunaggiri Pandian, and Swetha Valli. 2022. Findings of the shared task on Speech Recognition for Vulnerable Individuals in Tamil. In Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion. Association for Computational Linguistics. ↗
  2. Shankar Biradar, Sunil Saumya, and Arun Chauhan. 2021. Hate or non-hate: Translation based hate speech identification in code-mixed hinglish data set. In 2021 IEEE International Conference on Big Data (Big Data), pages 2470–2475. IEEE. find
  3. Bharathi Raja Chakravarthi. 2020. HopeEDI: A multilingual hope speech detection dataset for equality, diversity, and inclusion. In Proceedings of the Third Workshop on Computational Modeling of People’s Opinions, Personality, and Emotion’s in Social Media, pages 41–53, Barcelona, Spain (Online). Association for Computational Linguistics.
  4. Bharathi Raja Chakravarthi and Vigneshwaran Muralidaran. 2021. Findings of the shared task on hope speech detection for equality, diversity, and inclusion. In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, pages 61–72, Kyiv. Association for Computational Linguistics.
  5. Bharathi Raja Chakravarthi, Vigneshwaran Muralidaran, Ruba Priyadharshini, Subalalitha Chinnaudayar Navaneethakrishnan, John Phillip McCrae, Miguel Ángel García-Cumbreras, Salud María Jiménez-Zafra, Rafael Valencia-García, Prasanna Kumar Kumaresan, Rahul Ponnusamy, Daniel García-Baena, and José Antonio García-Díaz. 2022a. Findings of the shared task on Hope Speech Detection for Equality, Diversity, and Inclusion. In Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion. Association for Computational Linguistics.
  6. Bharathi Raja Chakravarthi, Vigneshwaran Muralidaran, Ruba Priyadharshini, and John Philip McCrae. 2020. Corpus creation for sentiment analysis in code-mixed Tamil-English text. In Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Computing for Under-Resourced Languages (CCURL), pages 202–210, Marseille, France. European Language Resources association. ↗
  7. Bharathi Raja Chakravarthi, Ruba Priyadharshini, Thenmozhi Durairaj, John Phillip McCrae, Paul Buitaleer, Prasanna Kumar Kumaresan, and Rahul Ponnusamy. 2022b. Findings of the shared task on Homophobia Transphobia Detection in Social Media Comments. In Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion. Association for Computational Linguistics. find
  8. Bharathi Raja Chakravarthi, Ruba Priyadharshini, Rahul Ponnusamy, Prasanna Kumar Kumaresan, Kayalvizhi Sampath, Durairaj Thenmozhi, Sathiyaraj Thangasamy, Rajendran Nallathambi, and John Phillip McCrae. 2021. Dataset for identification of homophobia and transophobia in multilingual YouTube comments. arXiv preprint arXiv:2109.00227. ↗
  9. Shi Chen and Bing Kong. 2021. cs_english@ LT-EDI-EACL2021: Hope Speech Detection Based On Fine-tuning ALBERT Model. In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, pages 128–131. find
  10. Bhargav Dave, Shripad Bhat, and Prasenjit Majumder. 2021. IRNLP_DAIICT@ LT-EDI-EACL2021: hope speech detection in code mixed text using TF-IDF char n-grams and MuRIL. In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, pages 114–117. find
  11. Bo Huang and Yang Bai. 2021. TEAM HUB@ LT-EDI-EACL2021: hope speech detection based on pre-trained language model. In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, pages 122–127. find
  12. Abhinav Kumar, Sunil Saumya, and Jyoti Prakash Singh. 2020. NITP-AI-NLP@ HASOC-Dravidian-CodeMix-FIRE2020: A Machine Learning Approach to Identify Offensive Languages from Dravidian Code-Mixed Text. In FIRE (Working Notes), pages 384–390. find
  13. Prasanna Kumar Kumaresan, Ratnasingam Sakuntharaj, Sajeetha Thavareesan, Subalalitha Navaneethakrishnan, Anand Kumar Madasamy, Bharathi Raja Chakravarthi, and John P McCrae. 2021. Findings of shared task on offensive language identification in Tamil and Malayalam. In Forum for Information Retrieval Evaluation, pages 16–18. find
  14. Khyati Mahajan, Erfan Al-Hossami, and Samira Shaikh. 2021. Teamuncc@ lt-edi-eacl2021: Hope speech detection using transfer learning with transformers. In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, pages 136–142. find
  15. Ruba Priyadharshini, Bharathi Raja Chakravarthi, Subalalitha Chinnaudayar Navaneethakrishnan, Thenmozhi Durairaj, Malliga Subramanian, Kogilavani Shanmugavadivel, Siddhanth U Hegde, and Prasanna Kumar Kumaresan. 2022. Findings of the shared task on Abusive Comment Detection in Tamil. In Proceedings of the Second Workshop on Speech and Language Technologies for Dravidian Languages. Association for Computational Linguistics. find
  16. Ruba Priyadharshini, Bharathi Raja Chakravarthi, Sajeetha Thavareesan, Dhivya Chinnappa, Durairaj Thenmozhi, and Rahul Ponnusamy. 2021. Overview of the DravidianCodeMix 2021 shared task on sentiment detection in Tamil, Malayalam, and Kannada. In Forum for Information Retrieval Evaluation, pages 4–6. find
  17. Karthik Puranik, Adeep Hande, Ruba Priyadharshini, Sajeetha Thavareesan, and Bharathi Raja Chakravarthi. 2021. IIITT@ LT-EDI-EACL2021-hope speech detection: there is always hope in transformers. arXiv preprint arXiv:2104.09066. ↗
  18. Manikandan Ravikiran, Bharathi Raja Chakravarthi, Anand Kumar Madasamy, Sangeetha Sivanesan, Ratnavel Rajalakshmi, Sajeetha Thavareesan, Rahul Ponnusamy, and Shankar Mahadevan. 2022. Findings of the shared task on Offensive Span Identification in code-mixed Tamil-English comments. In Proceedings of the Second Workshop on Speech and Language Technologies for Dravidian Languages. Association for Computational Linguistics. find
  19. Anbukkarasi Sampath, Thenmozhi Durairaj, Bharathi Raja Chakravarthi, Ruba Priyadharshini, Subalalitha Chinnaudayar Navaneethakrishnan, Kogilavani Shanmugavadivel, Sajeetha Thavareesan, Sathiyaraj Thangasamy, Parameswari Krishnamurthy, Adeep Hande, Sean Benhur, Kishor Kumar Ponnusamy, and Santhiya Pandiyan. 2022. Findings of the shared task on Emotion Analysis in Tamil. In Proceedings of the Second Workshop on Speech and Language Technologies for Dravidian Languages. Association for Computational Linguistics. ↗
  20. Sunil Saumya, Abhinav Kumar, and Jyoti Prakash Singh. 2021. Offensive language identification in dravidian code mixed social media text. In Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages, pages 36–45. find
  21. Sunil Saumya and Ankit Kumar Mishra. 2021. IIIT_DWD@ LT-EDI-EACL2021: hope speech detection in YouTube multilingual comments. In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, pages 107–113. find
  22. Yingjia Zhao and Xin Tao. 2021. ZYJ@ LT-EDI-EACL2021: XLM-RoBERTa-based model with attention for hope speech detection. In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, pages 118–121. find
  23. Stefan Ziehe, Franziska Pannach, and Aravind Krishnan. 2021. GCDH@ LT-EDI-EACL2021: XLM-RoBERTa for hope speech detection in English, Malayalam, and Tamil. In Proceedings of the First Workshop on Language Technology for Equality, Diversity and Inclusion, pages 132–135. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jha-2022-curaj-iiitdwd-lt-edi-acl/