Elamur Research⌕
☰
home / papers / 2018 / A Dataset of Hindi-English Code-Mixed Social Media Text for Hate Speech Detection

A Dataset of Hindi-English Code-Mixed Social Media Text for Hate Speech Detection

Paper · 2018 · Aditya Bohra, Deepanshu Vijay, Vinay Singh et al.

Summary

The paper tackles hate speech detection in Hindi-English code-mixed tweets, where users switch between the two languages and usually write Hindi in Roman script. The authors say no such resource existed before. They collected about 112,718 tweets using hashtags and keywords linked to politics, protests and riots. After filtering out noisy tweets and those written purely in English or purely in Hindi, 4,575 code-mixed tweets remained. Each tweet received a binary label (Hate Speech or Normal Speech), and each token was tagged with its source language.

As a baseline, the authors trained supervised classifiers on several feature types:

  • character n-grams
  • word n-grams
  • punctuation counts
  • negation counts
  • a small hate-word lexicon

They compared an RBF-kernel SVM with a Random Forest. The best result was 71.7% accuracy, from the SVM with all features combined. The main contribution is the dataset and its annotation scheme, which give a starting point for abusive-content research on informal, mixed-language text from multilingual societies.

Key points

  • Corpus composition: 4,575 code-mixed tweets, with 1,661 labelled hate speech and 2,914 labelled normal. The tweet IDs and annotations are released on GitHub.
  • Annotation scheme: tweets are stored in a tag-based format. Every token carries a language tag: eng, hin, or other (symbols, emoticons, punctuation, named entities, acronyms, URLs).
  • Agreement: two bilingual annotators were calibrated on a shared reference set of 50 tweets (25 hate, 25 normal). Their Cohen's Kappa on the full corpus was 0.982.
  • Features and setup:
    • character and word n-grams with n from 1 to 3
    • punctuation counts
    • negation counts based on Christopher Potts' list
    • a lexicon of 177 Hindi and English hate words
    • chi-square selection reduced the feature vector to 1,200 dimensions
    • Scikit-learn with 10-fold cross-validation
  • Results:
    • SVM: 71.7% with all features; character n-grams alone reached 71.6%.
    • Random Forest: word n-grams were best at 69.9%; all features gave 66.7%.
    • Punctuation, negation and lexicon features alone scored about 63–64%. This roughly equals the share of the normal class (about 64%), so these features add little on their own.

Interesting ideas

  • I1. Use a corpus-level code-mixing filter. The authors dropped tweets written entirely in one language and kept only genuinely mixed ones. The same filter could help build mixed Ukrainian-Russian or Surzhyk datasets, so the benchmark actually tests mixed-language handling rather than monolingual text.
  • I2. Tag language per token during annotation. Adding word-level language labels alongside the task label costs little extra effort. It supports later work such as language identification, per-language error analysis, or features based on switch points.
  • I3. Calibrate annotators with a small balanced set. Giving both annotators the same 50 tweets, half hate and half normal, before labelling the full corpus is a cheap way to align their notion of the classes. It likely helps explain the very high agreement and fits any subjective labelling task.
  • I4. Rely on character n-grams for noisy, transliterated text. Character n-grams were the strongest single feature. They cope with misspellings and non-standard romanized spellings that word-level vocabularies miss, which matters for any informal or Latin-transliterated text.
  • I5. Check each feature group in isolation. Reporting every feature group on its own quickly shows which signals carry real information. Comparing those scores against the majority-class share exposes weak features, such as punctuation counts here, that look acceptable but barely beat always predicting the larger class.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓International Institute of Information Technology, HyderabadHyderabad · India24 works · 1,719 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Social Sciences · 744 works Sociology and Political Science · 221 works

🧭 Topics 4 English · 1,373 works Sentiment & emotion · 608 works Multilingual & cross-lingual NLP · 467 works Bias, fairness & safety · 209 works

Cite

Cite as Aditya Bohra, Deepanshu Vijay, Vinay Singh, Syed Sarfaraz Akhtar and Manish Shrivastava. 2018. A Dataset of Hindi-English Code-Mixed Social Media Text for Hate Speech Detection. Proceedings of the Second Workshop on Computational Modeling of People's Opinions, Personality, and Emotions in Social Media.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-1105

@inproceedings{bohra2018dataset,
  title = {{A Dataset of Hindi-English Code-Mixed Social Media Text for Hate Speech Detection}},
  author = {Aditya Bohra, Deepanshu Vijay, Vinay Singh, Syed Sarfaraz Akhtar and Manish Shrivastava},
  booktitle = {Proceedings of the Second Workshop on Computational Modeling of People's Opinions, Personality, and Emotions in Social Media},
  year = {2018},
  doi = {10.18653/v1/w18-1105},
  url = {https://doi.org/10.18653/v1/w18-1105}
}

References

17 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kalika Bali, Jatin Sharma, Monojit Choudhury, and Yogarshi Vyas. 2014. ” i am borrowing ya mixing?” an analysis of english-hindi code mixing in facebook. In Proceedings of the First Workshop on Computational Approaches to Code Switching, pages 116–126. our card
  2. Huma Lodhi, Craig Saunders, John Shawe-Taylor, Nello Cristianini, and Chris Watkins. 2002. Text classification using string kernels. Journal of Machine Learning Research, 2(Feb):419–444. find
  3. Somnath Banerjee, Kunal Chakma, Sudip Kumar Naskar, Amitava Das, Paolo Rosso, Sivaji Bandyopadhyay, and Monojit Choudhury. 2016. Overview of the mixed script information retrieval (msir) at fire-2016. Organization (ORG), 67:24. find
  4. Shervin Malmasi and Marcos Zampieri. 2017. Detecting hate speech in social media. arXiv preprint arXiv:1712.06427. — [arXiv:1712.06427](https://arxiv.org/abs/1712.06427) find
  5. Utsab Barman, Amitava Das, Joachim Wagner, and Jennifer Foster. 2014. Code mixing: A challenge for language identification in the language of social media. In Proceedings of the first workshop on computational approaches to code switching, pages 13– 23. find
  6. Saif Mohammad. 2012. Portable features for classifying emotional text. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 587–591. Association for Computational Linguistics. find
  7. William B Cavnar and John M Trenkle. 1994. Ngram-based text categorization. Ann arbor mi, 48113(2):161–175. find
  8. Pieter Muysken. 2000. Bilingual speech: A typology of code-mixing, volume 11. Cambridge University Press. find
  9. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. arXiv preprint arXiv:1703.04009. our card
  10. Carol Myers-Scotton. 1993. Dueling languages: Grammatical structure in code-switching. Oxford University Press. find
  11. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in python. Journal of machine learning research, 12(Oct):2825–2830. find
  12. Matthew Purver and Stuart Battersby. 2012. Experimenting with distant supervision for emotion classification. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 482–491. Association for Computational Linguistics. find
  13. Khyathi Chandu Raghavi, Manoj Kumar Chinnakotla, and Manish Shrivastava. 2015. Answer ka type kya he?: Learning to classify questions in code-mixed language. In Proceedings of the 24th International Conference on World Wide Web, pages 853–858. ACM. find
  14. Anna Schmidt and Michael Wiegand. 2017. A survey on hate speech detection using natural language processing. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, pages 1–10. our card
  15. Arnav Sharma, Sakshi Gupta, Raveesh Motlani, Piyush Bansal, Manish Srivastava, Radhika Mamidi, and Dipti M Sharma. 2016. Shallow parsing pipeline for hindi-english code-mixed social media text. arXiv preprint arXiv:1604.03136. — [arXiv:1604.03136](https://arxiv.org/abs/1604.03136) find
  16. Yogarshi Vyas, Spandana Gella, Jatin Sharma, Kalika Bali, and Monojit Choudhury. 2014. Pos tagging of english-hindi code-mixed social media content. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 974–979. our card
  17. William Warner and Julia Hirschberg. 2012. Detecting hate speech on the world wide web. In Proceedings of the Second Workshop on Language in Social Media, pages 19–26. Association for Computational Linguistics. our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bohra-2018-dataset-hindi-english-code-mixed/