Elamur Research⌕
☰
home / papers / 2021 / Exploring Transformer Based Models to Identify Hate Speech and Offensive Content in English and Indo-Aryan Languages

Exploring Transformer Based Models to Identify Hate Speech and Offensive Content in English and Indo-Aryan Languages

Paper · 2021 · Somnath Banerjee, Maulindu Sarkar, Nancy Agrawal et al.

Summary

The paper describes the "Super Mario" team's systems for the HASOC shared task at FIRE 2021, which targets hateful and offensive posts on Twitter. The tracks covered binary HOF/NOT detection for English, Hindi and Marathi, a four-way English split (HATE, OFFN, PRFN, NONE), and a code-mixed Hindi-English task in which a reply's label can depend on its parent tweets. The authors did not design a new architecture. They compared off-the-shelf Transformer encoders: mBERT, XLMR-base, XLMR-large, Indic-BERT and dehate-BERT. Their pipeline cleaned noisy tweet text, adapted the models to the task data, and fine-tuned them with cross-entropy loss, optionally weighted by class to offset label skew.

XLMR-large was the most reliable model overall, and mBERT won on Hindi. The team ranked 2nd on the code-mixed and Hindi binary tracks. A further finding is that XLM-R scores changed much less across random seeds than mBERT scores. This matters for anyone who reports single-run results on small, low-resource datasets.

Key points

  • Official placements: 2nd on Code-Mixed (Macro F1 0.7107, XLMR-large) and 2nd on Hindi binary (0.7797, mBERT). The abstract lists 4th for English four-class (0.8006) and 12th for English two-class (0.6447). Tables 5–6 attach these two scores to the opposite tracks, so 0.8006 appears as English two-class and 0.6447 as English four-class.
  • Marathi was evaluated only after the competition. XLMR-large reached 0.8756 Macro F1, which equals the score of the official 3rd-place team.
  • Class imbalance was handled with normalized class weights instead of over- or undersampling. For example, English used NOT 1.4318 / HOF 0.7682. The four-class weights ranged from 0.7159 (NONE) to 1.5446 (OFFN).
  • Training setup:
    • adaptation on the target data for 20 epochs
    • Adam optimizer with learning rate 2e-5
    • no early stopping
    • HuggingFace and PyTorch
  • Across seeds, mBERT scores moved by roughly 6–7%, while XLM-R moved by only about 1–2%. Indic-BERT and dehate-BERT, despite being specialized for Indian languages or hate speech, ranked below XLM-R and mBERT.

Interesting ideas

  • I1. Running several random seeds showed which model is stable and which only looks good on a lucky run; this spread was reported as a result in its own right.
  • I2. Weighting the loss by inverse class frequency corrects label skew without duplicating or discarding training examples.
  • I3. In the conversational task, a reply such as "Amine" only reads as hateful once its parent tweet is known, so labels depend on the thread rather than the isolated sentence.
  • I4. Continuing to adapt a pretrained encoder on in-domain text before fine-tuning helped the models fit noisy tweet language.
  • I5. Specialized models (Indic-BERT, dehate-BERT) lost to general multilingual encoders, which shows that a strong general baseline should always be in the comparison.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Train and evaluate gparse over several seeds and publish the spread of LAS. This tells us whether a parser change beats seed noise.
  • I2 (tree): Rare UD relations in treebanks could receive larger loss weights in gparse, so the parser stops collapsing them into frequent labels.
  • I3 (frames): Speech-act and irony detection in prag should consult the preceding discourse context before assigning a label, not just the current sentence.
  • I4 (tree): Before training gparse on treebanks, its encoder could first be adapted on raw text from the target genre (e.g., fables or school texts).
  • I5 (learn): Every specialized component or witness parser should be checked against a plain general baseline, used as a control, before it is trusted.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Indian Institute of Technology KharagpurKharagpur · India10 works · 634 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 3 English · 1,373 works Sentiment & emotion · 608 works Bias, fairness & safety · 209 works

Cite

Cite as Somnath Banerjee, Maulindu Sarkar, Nancy Agrawal, Punyajoy Saha and Mithun Das. 2021. Exploring Transformer Based Models to Identify Hate Speech and Offensive Content in English and Indo-Aryan Languages. arXiv:2111.13974.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2111.13974 · arXiv 2111.13974

@misc{banerjee2021exploring,
  title = {{Exploring Transformer Based Models to Identify Hate Speech and Offensive Content in English and Indo-Aryan Languages}},
  author = {Somnath Banerjee, Maulindu Sarkar, Nancy Agrawal, Punyajoy Saha and Mithun Das},
  year = {2021},
  doi = {10.48550/arxiv.2111.13974},
  eprint = {2111.13974},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2111.13974}
}

References

32 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. B. Mathew, R. Dutt, P. Goyal, A. Mukherjee, Spread of hate speech in online social media, in: Proceedings of WebSci, ACM, 2019. find
  2. C. Newton, The terror queue, 2019. URL: https://www.theverge.com/2019/12/16/21021005/google-youtube-moderators-ptsd-accenture-violent-disturbing-content-interviews-video. ↗
  3. A. Robertson, Facebook says ai has fueled a hate speech crackdown, 2020. URL: https://www.theverge.com/2020/11/19/21575139/facebook-moderation-ai-hate-speech. ↗
  4. B. Perrigo, Facebook’s hate speech algorithms leave out some languages, 2019. URL: https://time.com/5739688/facebook-hate-speech-languages/. ↗
  5. M. Das, P. Saha, R. Dutt, P. Goyal, A. Mukherjee, B. Mathew, You too brutus! trapping hateful users in social media: Challenges, solutions & insights, 2021, pp. 79–89. doi: 10.1145/3465336.3475106 . ↗
  6. M. Das, B. Mathew, P. Saha, P. Goyal, A. Mukherjee, Hate speech in online social media, ACM SIGWEB Newsletter (2020) 1–8. doi: 10.1145/3427478.3427482 . ↗
  7. S. Modha, T. Mandl, G. K. Shahi, H. Madhu, S. Satapara, T. Ranasinghe, M. Zampieri, Overview of the hasoc subtrack at fire 2021: Hate speech and offensive content identification in english and indo-aryan languages and conversational hate speech, in: FIRE 2021: Forum for Information Retrieval Evaluation, Virtual Event, 13th-17th December 2021, ACM, 2021. our card
  8. T. Mandl, S. Modha, G. K. Shahi, H. Madhu, S. Satapara, P. Majumder, J. Schäfer, T. Ranasinghe, M. Zampieri, D. Nandini, A. K. Jaiswal, Overview of the HASOC subtrack at FIRE 2021: Hate Speech and Offensive Content Identification in English and Indo-Aryan Languages, in: Working Notes of FIRE 2021 - Forum for Information Retrieval Evaluation, CEUR, 2021. URL: http://ceur-ws.org/. ↗ our card
  9. S. Satapara, S. Modha, T. Mandl, H. Madhu, P. Majumder, Overview of the HASOC Subtrack at FIRE 2021: Conversational Hate Speech Detection in Code-mixed language , in: Working Notes of FIRE 2021 - Forum for Information Retrieval Evaluation, CEUR, 2021. find
  10. B. Mathew, P. Saha, S. M. Yimam, C. Biemann, P. Goyal, A. Mukherjee, Hatexplain: A benchmark dataset for explainable hate speech detection, 2020. arXiv:2012.10289 . ↗
  11. Y. Chen, Y. Zhou, S. Zhu, H. Xu, Detecting offensive language in social media to protect adolescent online safety, in: 2012 International Conference on Privacy, Security, Risk and Trust and 2012 International Confernece on Social Computing, 2012, pp. 71–80. doi: 10.1109/SocialCom-PASSAT.2012.55 . ↗
  12. P. L. Teh, C.-B. Cheng, W. M. Chee, Identifying and categorising profane words in hate speech, in: Proceedings of the 2nd International Conference on Compute and Data Analysis, ICCDA 2018, Association for Computing Machinery, New York, NY, USA, 2018, p. 65–69. URL: https://doi.org/10.1145/3193077.3193078. doi: 10.1145/3193077.3193078 . ↗
  13. Y. Chen, Y. Zhou, S. Zhu, H. Xu, Detecting offensive language in social media to protect adolescent online safety, in: 2012 International Conference on Privacy, Security, Risk and Trust and 2012 International Confernece on Social Computing, 2012, pp. 71–80. doi: 10.1109/SocialCom-PASSAT.2012.55 . ↗
  14. T. Davidson, D. Warmsley, M. Macy, I. Weber, Automated hate speech detection and the problem of offensive language, in: ICWSM, 2017. our card
  15. G. K. Pitsilis, H. Ramampiaro, H. Langseth, Detecting offensive language in tweets using deep learning, ArXiv abs/1801.04433 (2018). ↗
  16. Y. Goldberg, A primer on neural network models for natural language processing, Journal of Artificial Intelligence Research 57 (2015). doi: 10.1613/jair.4992 . ↗
  17. G. L. D. la Peña Sarracén, R. G. Pons, C. E. Muñiz-Cuza, P. Rosso, Hate speech detection using attention-based lstm, in: EVALITA@CLiC-it, 2018. find
  18. G. I. Sigurbergsson, L. Derczynski, Offensive language and hate speech detection for Danish, in: Proceedings of the 12th Language Resources and Evaluation Conference, European Language Resources Association, Marseille, France, 2020, pp. 3498–3508. URL: https://aclanthology.org/2020.lrec-1.430. ↗ our card
  19. Z. Pitenis, M. Zampieri, T. Ranasinghe, Offensive language identification in greek, in: LREC, 2020. our card
  20. Çagri Çöltekin, A corpus of turkish offensive language on social media, in: LREC, 2020. our card
  21. T. Mandl, S. Modha, P. Majumder, D. Patel, M. Dave, C. Mandalia, A. Patel, Overview of the hasoc track at fire 2019: Hate speech and offensive content identification in indo-european languages, Proceedings of the 11th Forum for Information Retrieval Evaluation (2019). find
  22. S. Mishra, 3idiots at hasoc 2019: Fine-tuning transformer neural networks for hate speech identification in indo-european languages, in: FIRE, 2019. find
  23. S. Renjit, S. M. Idicula, Cusatnlp@hasoc-dravidian-codemix-fire2020:identifying offensive language from manglishtweets, ArXiv abs/2010.08756 (2020). ↗
  24. A. Vaswani, N. M. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, ArXiv abs/1706.03762 (2017). ↗ our card
  25. J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, in: NAACL, 2019. our card
  26. S. Gaikwad, T. Ranasinghe, M. Zampieri, C. M. Homan, Cross-lingual offensive language identification for low resource languages: The case of marathi, in: Proceedings of RANLP, 2021. find
  27. A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, V. Stoyanov, Unsupervised cross-lingual representation learning at scale, 2020. arXiv:1911.02116 . ↗
  28. S. S. Aluru, B. Mathew, P. Saha, A. Mukherjee, A deep dive into multilingual hate speech classification, in: Machine Learning and Knowledge Discovery in Databases. Applied Data Science and Demo Track: European Conference, ECML PKDD 2020, Ghent, Belgium, September 14–18, 2020, Proceedings, Part V, Springer International Publishing, 2021, pp. 423–439. find
  29. S. Mannor, D. Peleg, R. Rubinstein, The cross entropy method for classification, in: Proceedings of the 22nd International Conference on Machine Learning, ICML ’05, Association for Computing Machinery, New York, NY, USA, 2005, p. 561–568. URL: https://doi.org/10.1145/1102351.1102422. doi: 10.1145/1102351.1102422 . ↗
  30. T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, A. M. Rush, Huggingface’s transformers: State-of-the-art natural language processing, 2020. arXiv:1910.03771 . ↗ our card
  31. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, S. Chintala, Pytorch: An imperative style, high-performance deep learning library, 2019. arXiv:1912.01703 . ↗ our card
  32. I. Loshchilov, F. Hutter, Decoupled weight decay regularization, 2019. arXiv:1711.05101 . ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/banerjee-2021-exploring-transformer-based-models-identify/