Elamur Research⌕
☰
home / papers / 2024 / dkit@LT-EDI-2024: Detecting Homophobia and Transphobia in English Social Media Comments

dkit@LT-EDI-2024: Detecting Homophobia and Transphobia in English Social Media Comments

Paper · 2024 · Sargam Yadav, Abhishek Kaushik and Kevin McDaid

Summary

The paper describes the Dundalk Institute of Technology entry to the LT-EDI-2024 shared task on spotting homophobic and transphobic language in English YouTube comments. Each comment gets one of three labels: Non-anti-LGBT+ content, Homophobia, or Transphobia. The data is extremely skewed: of 3,164 training comments, only 7 are transphobic. The authors merged the train and dev sets and made a stratified 70/30 split. They then compared three kinds of systems:

  • classic classifiers (LR, Naive Bayes variants, SVMs, DT, RF), each fed TF-IDF, count, or custom Word2Vec features and tuned by grid search;
  • a small dense neural network on TF-IDF features, trained with class weights;
  • a fine-tuned bert-base-uncased model.

BERT scored highest, with macro-F1 0.4963, which placed first in the task. Several models reached accuracy above 0.9 while their macro-F1 stayed low, showing how much the majority class inflates accuracy. The work offers a simple, reproducible baseline for a socially important moderation problem. It also shows how hard rare hate categories remain when they have almost no training examples.

Key points

  • The English training set has 3,164 comments: 2,978 non-anti-LGBT+, 179 homophobia, and 7 transphobia. The dev and test sets hold 792 and 990 comments, though the paper's text and its table disagree on which set is which size.
  • Text features were TF-IDF and count vectors, each capped at 2,000 features, plus a skip-gram Word2Vec model trained on the task data with Gensim (300 dimensions, window 10).
  • The BERT setup was bert-base-uncased with max length 128, 3 epochs, learning rate 3e-5, and batch size 2. It reached accuracy 0.9556 and macro-F1 0.4963. The model is released on Hugging Face as sam34738/BERT_homo.
  • The runner-up was the TF-IDF + DNN model (three dense layers of 128, 64 and 32 units with dropout, plus class weights), with macro-F1 0.4295. The best classic model was count vectors + DT, at 0.3731.
  • Word2Vec + multinomial NB reached 0.9418 accuracy but only 0.3233 macro-F1, which amounts to predicting the majority class.

Interesting ideas

  • I1. With a heavily skewed label set, high accuracy can hide a model that never predicts the minority classes, so macro-F1 should be the selection metric.
  • I2. The authors pooled the official train and dev data and re-split it with stratification, so the rare classes appear in both the tuning and validation portions.
  • I3. Rescaling Word2Vec vectors to the [0,1] range lets them be used by classifiers that need non-negative inputs, such as multinomial Naive Bayes.
  • I4. The authors keep only the top two models per feature type, which gives a compact comparison table across representation families rather than a single winner.
  • I5. The limitations section notes that informal social-media English departs from the formal text that pretrained tools are built on.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Report per-class or macro scores with a "majority-label" negative control in evaluations, so that rare readings such as irony or rare frames are not masked by the common case.
  • I2 (learn): When building silver training sets for rare constructions, stratify the held-out split by construction type so that every rare type is represented in evaluation.
  • I3 (tokens): Normalizing feature ranges before feeding embeddings into a component that assumes non-negative counts avoids silent failures in mixed perceptron/statistical taggers.
  • I4 (tree): Comparing witness parsers grouped by input representation (raw tokens vs. tagged vs. embeddings) would show whether gains come from the parser or from its features.
  • I5 (tokens): Add an informal-register test slice (comments, dropped punctuation, odd spelling) to check how the tokenizer and tagger degrade before errors reach the tree and frame stages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: Dundalk Institute of Technology.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Sargam Yadav, Abhishek Kaushik and Kevin McDaid. 2024. dkit@LT-EDI-2024: Detecting Homophobia and Transphobia in English Social Media Comments. Proceedings of the Fourth Workshop on Language Technology for Equality, Diversity, Inclusion.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.ltedi-1.36

@inproceedings{yadav2024dkit,
  title = {{dkit@LT-EDI-2024: Detecting Homophobia and Transphobia in English Social Media Comments}},
  author = {Sargam Yadav, Abhishek Kaushik and Kevin McDaid},
  booktitle = {Proceedings of the Fourth Workshop on Language Technology for Equality, Diversity, Inclusion},
  year = {2024},
  doi = {10.18653/v1/2024.ltedi-1.36},
  url = {https://doi.org/10.18653/v1/2024.ltedi-1.36}
}

References

35 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bharathi Raja Chakravarthi. 2023. Detection of homophobia and transphobia in YouTube comments. International Journal of Data Science and Analytics, pages 1–20. ↗
  2. Bharathi Raja Chakravarthi, Adeep Hande, Rahul Ponnusamy, Prasanna Kumar Kumaresan, and Ruba Priyadharshini. 2022. How can we detect Homophobia and Transphobia? Experiments in a multilingual code-mixed setting for social media governance. International Journal of Information Management Data Insights, 2(2):100119. find
  3. Bharathi Raja Chakravarthi, Rahul Ponnusamy, Malliga Subramanian, Paul Buitelaar, Miguel Ángel García-Cumbreras, Salud María Jiménez-Zafra, José Antonio García-Díaz, Rafael Valencia-García, and Nitesh Jindal. 2023. Overview of Second Shared Task on Homophobia and Transphobia Detection in English, Spanish, Hindi, Tamil, and Malayalam. In Proceedings of the Third Workshop on Language Technology for Equality, Diversity and Inclusion, Varna, Bulgaria. Recent Advances in Natural Language Processing. find
  4. Justin Cheng, Michael Bernstein, Cristian Danescu-Niculescu-Mizil, and Jure Leskovec. 2017. Anyone can become a troll: Causes of trolling behavior in online discussions. In Proceedings of the 2017 ACM conference on computer supported cooperative work and social computing, pages 1217–1230. ↗
  5. Subalalitha Chinnaudayar Navaneethakrishnan, Bharathi Raja Chakravarthi, Kogilavani Shanmugavadivel, Malliga Subramanian, Prasanna Kumar Kumaresan, Bharathi, Lavanya Sambath Kumar, and Rahul Ponnusamy. 2022. Findings of shared task on Sentiment Analysis and Homophobia Detection of YouTube Comments in Code-Mixed Dravidian Languages. In Proceedings of the 14th Annual Meeting of the Forum for Information Retrieval Evaluation, pages 18–21. find
  6. Suraj Kumar Donthula and Abhishek Kaushik. 2019. Man is what he eats: A research on Hinglish sentiments of YouTube cookery channels using deep learning. International Journal of Recent Technology and Engineering (IJRTE), 8(2S11):930–937. find
  7. Elisabetta Fersini, Debora Nozza, Paolo Rosso, et al. 2020. AMI@ EVALITA2020: Automatic misogyny identification. In Proceedings of the 7th evaluation campaign of Natural Language Processing and Speech tools for Italian (EVALITA 2020). (seleziona...). ↗
  8. Paula Fortuna and Sérgio Nunes. 2018. A survey on automatic detection of hate speech in text. ACM Computing Surveys (CSUR), 51(4):1–30. ↗
  9. Sameer Hinduja and Justin W Patchin. 2020. Bullying, cyberbullying, and LGBTQ students. Cyberbullying Research Center. http://hdl.handle. net/20.500, 11990:2073. ↗
  10. Gagandeep Kaur, Abhishek Kaushik, and Shubham Sharma. 2019. Cooking is creating emotion: A study on Hinglish sentiments of YouTube cookery channels using semi-supervised approach. Big Data and Cognitive Computing, 3(3):37. ↗
  11. Subramaniam Kazhuparambil and Abhishek Kaushik. 2020a. Classification of Malayalam-English mix-code comments using current state of art. In 2020 IEEE International Conference for Innovation in Technology (INOCON), pages 1–6. IEEE. find
  12. Subramaniam Kazhuparambil and Abhishek Kaushik. 2020b. Cooking is all about people: Comment classification on cookery channels using BERT and classification models (Malayalam-English mix-code). arXiv preprint arXiv:2007.04249. ↗
  13. Simran Khanuja, Diksha Bansal, Sarvesh Mehtani, Savya Khosla, Atreyee Dey, Balaji Gopalan, Dilip Kumar Margam, Pooja Aggarwal, Rajiv Teja Nagipogu, Shachi Dave, et al. 2021. MuRIL: Multilingual representations for Indian languages. arXiv preprint arXiv:2103.10730. ↗
  14. Ritesh Kumar, Guggilla Bhanodai, Rajendra Pamula, and Maheshwar Reddy Chennuru. 2018. TRAC-1 shared task on aggression identification: IIT (ISM)@ COLING’18. In Proceedings of the first workshop on trolling, aggression and cyberbullying (TRAC-2018), pages 58–65. find
  15. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. ↗
  16. Prasanna Kumar Kumaresan, Rahul Ponnusamy, Ruba Priyadharshini, Paul Buitelaar, and Bharathi Raja Chakravarthi. 2023. Homophobia and transphobia detection for low-resourced languages in social media comments. Natural Language Processing Journal, page 100041. ↗
  17. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  18. Prasanna Kumar Kumaresan, Ruba Priyadharshini, Bharathi Raja Chakravarthi, Paul Buitelaar, Asha Hegde, Hosahalli Lakshmaiah Shashirekha, Saranya Rajiakodi, Miguel Ángel García-Cumbreras, Salud María Jiménez-Zafra, José Antonio García-Díaz, Rafael Valencia-García, Kishore Kumar Ponnusamy, Poorvi Shetty, and Daniel García-Baena. 2024. Overview of Third Shared Task on Homophobia and Transphobia Detection in Social Media Comments. In Proceedings of the Fourth Workshop on Language Technology for Equality, Diversity and Inclusion, Malta. European Chapter of the Association for Computational Linguistics. find
  19. Shrey Satapara, Sandip Modha, Thomas Mandl, Hiren Madhu, and Prasenjit Majumder. 2021. Overview of the HASOC subtrack at FIRE 2021: Conversational hate speech detection in code-mixed language. Working Notes of FIRE, pages 13–17. find
  20. Sonali Rajesh Shah and Abhishek Kaushik. 2019. Sentiment analysis on Indian indigenous languages: a review on multilingual opinion mining. arXiv preprint arXiv:1911.12848. ↗
  21. Thomas Mandl, Sandip Modha, Gautam Kishore Shahi, Hiren Madhu, Shrey Satapara, Prasenjit Majumder, Johannes Schäfer, Tharindu Ranasinghe, Marcos Zampieri, Durgesh Nandini, et al. 2021. Overview of the HASOC subtrack at FIRE 2021: Hate speech and offensive content identification in English and Indo-aryan languages. arXiv preprint arXiv:2112.09301. ↗
  22. Sonali Rajesh Shah, Abhishek Kaushik, Shubham Sharma, and Janice Shah. 2020. Opinion-mining on Marglish and Devanagari comments of YouTube cookery channels using parametric and non-parametric learning models. Big Data and Cognitive Computing, 4(1):3. ↗
  23. Deepalakshmi Manikandan, Malliga Subramanian, and Kogilavani Shanmugavadivel. 2022. A System For Detecting Abusive Contents Against LGBT Community Using Deep Learning Based Transformer Models. In Working Notes of FIRE 2022-Forum for Information Retrieval Evaluation (Hybrid). CEUR. find
  24. Sidney Wong, Matthew Durward, Benjamin Adams, and Jonathan Dunn. 2023. cantnlp@ LT-EDI-2023: Homophobia/Transphobia Detection in Social Media Comments using Spatio-Temporally Retrained Language Models. In Proceedings of the Third Workshop on Language Technology for Equality, Diversity and Inclusion, pages 103–108. ↗
  25. Puneet Mathur, Ramit Sawhney, Meghna Ayyar, and Rajiv Shah. 2018. Did you offend me? classification of offensive tweets in Hinglish language. In Proceedings of the 2nd workshop on abusive language online (ALW2), pages 138–148. find
  26. Chris McCormick. 2016. Word2vec tutorial-the skip-gram model. Apr-2016.[Online]. Available: http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model. ↗
  27. Megan A Moreno, Aubrey D Gower, Heather Brittain, and Tracy Vaillancourt. 2019. Applying natural language processing to evaluate news media coverage of bullying and cyberbullying. Prevention science, 20(8):1274–1283. ↗
  28. Marzieh Mozafari, Reza Farahbakhsh, and Noel Crespi. 2020a. A BERT-based transfer learning approach for hate speech detection in online social media. In Complex Networks and Their Applications VIII: Volume 1 Proceedings of the Eighth International Conference on Complex Networks and Their Applications COM-PLEX NETWORKS 2019 8, pages 928–940. Springer. find
  29. Marzieh Mozafari, Reza Farahbakhsh, and Noël Crespi. 2020b. Hate speech detection and racial bias mitigation in social media based on BERT model. PloS one, 15(8):e0237861. ↗
  30. Sargam Yadav and Abhishek Kaushik. 2022. Contextualized Embeddings from Transformers for Sentiment Analysis on Code-Mixed Hinglish Data: An Expanded Approach with Explainable Artificial Intelligence. In International Conference on Speech and Language Technologies for Low-resource Languages, pages 99–119. Springer. find
  31. Sargam Yadav, Abhishek Kaushik, and Kevin McDaid. 2023a. Hate Speech is not Free Speech: Explainable Machine Learning for Hate Speech Detection in Code-Mixed Languages. In 2023 IEEE International Symposium on Technology and Society (ISTAS), pages 1–8. IEEE. find
  32. Sargam Yadav, Abhishek Kaushik, and Shubham Sharma. 2021. Cooking well, with love, is an art: Transformers on YouTube Hinglish data. In 2021 International Conference on Computational Performance Evaluation (ComPE), pages 836–841. IEEE. find
  33. Sargam Yadav, Abhishek Kaushik, and Surbhi Sharma. 2023b. Comprehensive Analysis of the Artificial Intelligence Approaches for Detecting Misogynistic Mixed-Code Online Content in South Asian Countries: A Review. Cyberfeminism and Gender Violence in Social Media, pages 350–368. ↗
  34. Dean Ninalga. 2023. Cordyceps@ LT-EDI: Patching Language-Specific Homophobia/Transphobia Classifiers with a Multilingual Understanding. arXiv preprint arXiv:2309.13561. ↗
  35. María Antonia Paz, Julio Montero-Díaz, and Alicia Moreno-Delgado. 2020. Hate speech: A systematized review. Sage Open, 10(4):2158244020973022. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yadav-2024-dkit-lt-edi-2024-detecting/