Elamur Research⌕
☰
home / papers / 2026 / SigJBS@LT-EDI 2026: Multimodal Homophobia and Transphobia Meme Classification

SigJBS@LT-EDI 2026: Multimodal Homophobia and Transphobia Meme Classification

Paper · 2026 · Gaurangi Sinha, Rajarajeswari Palacharla and Manoj Balaji Jagadeeshan

Summary

The paper describes a system for the LT-EDI@ACL 2026 shared task on detecting homophobic and transphobic memes in English, Hindi, and Chinese. In these memes, hateful meaning often appears only when the picture and its overlaid caption are read together. Instead of designing a heavy custom architecture, the authors assemble a compact pipeline from established parts. Frozen CLIP ViT-L/14 image embeddings are joined with TF–IDF vectors built from text that EasyOCR extracts using script-specific readers. The combined vector feeds a class-balanced multinomial logistic regression. Two optional experts, a LoRA-tuned Qwen2-VL-2B and a CLIP zero-shot scorer, are added through fixed-weight voting. Under this scheme, the main classifier can be overridden only when both auxiliary experts disagree with it. The submission placed first for Hindi, third for English, and fifth for Chinese. The work shows that pretrained features combined with explicit OCR can deliver competitive, cheap, and reproducible multilingual moderation. It also identifies OCR quality and class imbalance as the main weaknesses.

Key points

  • Data and labels: three classes (Homophobia, Transphobia, Non_Anti_LGBT). Training/test sizes are roughly 560/141 for English, 798/200 for Hindi, and 956/239 for Chinese. The Chinese set is highly imbalanced, with only 55 transphobic training memes.
  • Features and OCR setup: 768-dim normalized CLIP features plus 3,000-dim unigram+bigram TF–IDF. OCR languages are en, hi+en, and ch_sim+en. For mixed-script images, the system keeps the longest OCR output.
  • Ensemble design: voting weights are 4/2/1 for CLIP+TF–IDF, LoRA-Qwen2-VL (r=16, 4-bit, trained via Unsloth), and CLIP zero-shot. The expert with the weakest validation macro-F1 is removed before inference.
  • Official macro-F1 scores: Hindi 0.8081 (rank 1), English 0.6396 (rank 3), and Chinese 0.6492 (rank 5). Chinese accuracy was high (0.8285), but macro-F1 was lower because of class imbalance.
  • English validation results (N=112):
    • CLIP+TF–IDF reached 0.9029 macro-F1, compared with 0.7826 for LoRA-Qwen2-VL and 0.5546 for zero-shot.
    • Image-only (0.9128) slightly outperformed the fused model (0.9011). However, adding TF–IDF increased precision on the harmful classes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Texas A&M UniversityCollege Station · United States26 works · 1,871 citations🎓Indian Institute of Technology KharagpurKharagpur · India24 works · 714 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 657 works

Cite

Cite as Gaurangi Sinha, Rajarajeswari Palacharla and Manoj Balaji Jagadeeshan. 2026. SigJBS@LT-EDI 2026: Multimodal Homophobia and Transphobia Meme Classification. Proceedings of the Sixth Workshop on Language Technology for Equality, Diversity, Inclusion.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.ltedi-1.28

@inproceedings{sinha2026sigjbs,
  title = {{SigJBS@LT-EDI 2026: Multimodal Homophobia and Transphobia Meme Classification}},
  author = {Gaurangi Sinha, Rajarajeswari Palacharla and Manoj Balaji Jagadeeshan},
  booktitle = {Proceedings of the Sixth Workshop on Language Technology for Equality, Diversity, Inclusion},
  year = {2026},
  doi = {10.18653/v1/2026.ltedi-1.28},
  url = {https://doi.org/10.18653/v1/2026.ltedi-1.28}
}

References

13 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bharathi Raja Chakravarthi. 2024. Detection of homophobia and transphobia in YouTube comments. International Journal of Data Science and Analytics, 18(1):49–68. find
  2. Bharathi Raja Chakravarthi, Manoj Balaji Jagadeeshan, Vasanth Palanikumar, and Ruba Priyadharshini. 2023a. Offensive language identification in Dravidian languages using MPNet and CNN. International Journal of Information Management Data Insights, 3(1):100151. find
  3. Bharathi Raja Chakravarthi, Ruba Priyadharshini, Shubanker Banerjee, Manoj Balaji Jagadeeshan, Prasanna Kumar Kumaresan, Rahul Ponnusamy, Sean Benhur, and John Philip McCrae. 2023b. Detecting abusive comments at a fine-grained level in a low-resource language. Natural Language Processing Journal, 3:… find
  4. Daniel Han. 2023. Unsloth: Fast and memory-efficient LLM fine-tuning. https://github.com/ unslothai/unsloth. Accessed: 2026-03-01. ↗
  5. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR).
  6. Jinfa Huang, Jinsheng Pan, Zhongwei Wan, Hanjia Lyu, and Jiebo Luo. 2025. Evolver: Chain-of-evolution prompting to boost large multimodal models for hateful meme detection. In Proceedings of the 31st International Conference on Computational Linguistics, pages 7321–7330, Abu Dhabi, UAE. Association… find
  7. JaidedAI. 2020. EasyOCR: Ready-to-use OCR with 80+ supported languages. https://github.com/ JaidedAI/EasyOCR. Accessed: 2026-03-01. ↗
  8. Douwe Kiela, Hamed Firooz, Aakanksha Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. 2020. The hateful memes challenge: Detecting hate speech in multimodal memes. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 2611–2624.
  9. Jingbiao Mei, Jinghong Chen, Weizhe Lin, Bill Byrne, and Marcus Tomalin. 2024. Improving hateful meme detection through retrieval-guided contrastive learning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5333–5347, Bangkok,… find
  10. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, and 2 others. 20… — PyTorch: An Imperative Style, High-Performance Deep Learning Library
  11. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn:… find
  12. Kishore Kumar Ponnusamy, Bharathi Raja Chakravarthi, Mahesh Susaladi, Junru Ren, Prasanna Kumar Kumaresan, B. Premjith, Durairaj Thenmozhi, Ruba Priyadharshini, and Subalalitha Chinnaudayar Navaneethakrishnan. 2026. Overview of multimodal homophobia and transphobia meme classification. In Proceeding… find
  13. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the 38th Internatio…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sinha-2026-sigjbs-lt-edi-2026-multimodal/