Elamur Research⌕
☰
home / papers / 2020 / A Corpus of Turkish Offensive Language on Social Media

A Corpus of Turkish Offensive Language on Social Media

Summary

This paper addresses the lack of any annotated resource for offensive language in Turkish, a language with a very active Twitter user base but almost no attention in this research area. The author built a corpus of 36 232 tweets drawn at random from the Twitter stream over about a year and a half. They are labeled with a scheme modeled on recent annotation efforts for other languages, especially target-oriented schemes like OffensEval. Roughly one tweet in five turned out to contain some form of offense. Offensive tweets were further split into targeted and untargeted, and targeted ones into three subtypes by target.

Strong baseline classifiers reached:

  • solid F1 on the first two decisions (offensive or not, targeted or not),
  • clearly weaker results on assigning the target type.

The resource matters for three reasons:

  • Turkish differs typologically from the mostly Indo-European languages studied so far.
  • Its sociopolitical setting may shape how offense is expressed.
  • It is released openly, so it supports both practical moderation tools and cross-lingual research.

Key points

  • The paper presents itself as the first offensive-language corpus for Turkish: 36 232 randomly sampled tweets, collected April 2018 – September 2019.
  • About 19 % of tweets contain offensive language, subcategorized by whether and whom it targets.
  • Baseline F1 scores are 77.3 % for offensive detection, 77.9 % for targeted vs. untargeted, and 53.0 % for the three-way target classification.
  • The author argues that several properties of Turkish could challenge existing systems:
    • agglutination,
    • flexible word order,
    • frequent argument dropping,
    • a strong polite/informal split.
  • The data is to be released under a permissive license at coltekin.github.io/offensive-turkish.

Interesting ideas

  • I1. Random sampling instead of keyword or hashtag filtering. Pulling tweets randomly from the stream gives a realistic picture of how often offense occurs and avoids topic bias. This is worth copying for any new-language corpus where the true base rate is unknown.
  • I2. Reusing an established label hierarchy. The guidelines are built on a review of existing schemes rather than invented from scratch. This keeps the corpus comparable with resources in other languages and enables cross-lingual transfer or evaluation.
  • I3. Scoring each level of the hierarchy separately. Reporting results per level (offensive, then targeted, then target type) shows exactly where models break down. Here it is the fine-grained target step that lags well behind.
  • I4. Treating sociopolitical context as an annotation factor. The author expects censorship, surveillance and arrests to push users toward indirect, veiled offense. This is relevant for any setting with political polarization, conflict or refugee discourse, where surface-level lexicon cues may miss much of the offensive content.
  • I5. Choosing a language that tests existing tools. Turkish is positioned as a stress test because it is morphologically rich and differs typologically from the languages most systems were built for. The same argument can justify building comparable resources for other under-studied languages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

84 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 79 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{coltekin2020corpus,
  title = {{A Corpus of Turkish Offensive Language on Social Media}},
  author = {Çöltekin, Çağrı},
  booktitle = {Proceedings of the Twelfth Language Resources and Evaluation Conference},
  year = {2020},
  url = {https://aclanthology.org/2020.lrec-1.758/}
}

References

74 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agarwal, S. and Sureka, A. (2015). Using KNN and SVM based one-class classifier for detecting online radicalization on Twitter. In International Conference on Distributed Computing and Internet Technology, pages 431– 442. Springer. find
  2. Agarwal, S. and Sureka, A. (2017). Characterizing linguistic attributes for automatic classification of intent based racist/radicalized posts on Tumblr micro-blogging website. arXiv preprint arXiv:1701.04931. — [arXiv:1701.04931](https://arxiv.org/abs/1701.04931) find
  3. Akgül, M. and Kırlıdoğ, M. (2015). Internet censorship in Turkey. Internet Policy Review, 4(2):1–22. find
  4. Álvarez-Carmona, M. Á., Guzmán-Falcón, E., Montes-y Gómez, M., Escalante, H. J., Villasenor-Pineda, L., Reyes-Meza, V., and Rico-Sulayes, A. (2018). Overview of MEX-A3T at IberEval 2018: Authorship and aggressiveness analysis in Mexican Spanish tweets. In Notebook Papers of 3rd SEPLN Workshop on Eva… find
  5. American Psychological Association. (2004). APA resolution on bullying among children and youth. http://www.apa.org/about/governance/council/policy/bullying.pdf. ↗
  6. Article 19. (2018). Responding to ‘hate speech’: Comparative overview of six EU countries. https://www.article19.org/wp-content/uploads/2018/03/ECA-hate-speech-compilation-report_March-2018.pdf. ↗
  7. Basile, V., Bosco, C., Fersini, E., Nozza, D., Patti, V., Rangel Pardo, F. M., Rosso, P., and Sanguinetti, M. (2019). SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in Twitter. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 54… find
  8. Bulut, E. and Yörük, E. (2017). Digital populism: Trolls and political polarization of Twitter in Turkey. International Journal of Communication, 11:25. find
  9. Burnap, P. and Williams, M. L. (2014). Hate speech, machine classification and statistical modelling of information flows on Twitter: Interpretation and communication for policy decision making. In Proceedings of Internet, Policy & Politics, pages 1–18. find
  10. Chen, Y., Zhou, Y., Zhu, S., and Xu, H. (2012). Detecting offensive language in social media to protect adolescent online safety. In 2012 International Conference on Privacy, Security, Risk and Trust and 2012 International Confernece on Social Computing, pages 71–80. IEEE. find
  11. Çöltekin, Ç., Rama, T., and Blaschke, V. (2018). Tübingen-Oslo team at the VarDial 2018 evaluation campaign: An analysis of n-gram features in language variety identification. In Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects (Var-Dial 2018), pages 55–65. find
  12. Dadvar, M., Trieschnigg, D., Ordelman, R., and de Jong, F. (2013). Improving cyberbullying detection with user context. In European Conference on Information Retrieval, pages 693–696. Springer. find
  13. Dadvar, M., Trieschnigg, D., and de Jong, F. (2014). Experts and machines against bullies: A hybrid approach to detect cyberbullies. In Canadian Conference on Artificial Intelligence, pages 275–281. Springer. find
  14. Davidson, T., Warmsley, D., Macy, M., and Weber, I. (2017). Automated hate speech detection and the problem of offensive language. In Eleventh international AAAI conference on web and social media, pages 512– 515. find our card
  15. Del Vigna, F., Cimino, A., Dell’Orletta, F., Petrocchi, M., and Tesconi, M. (2017). Hate me, hate me not: Hate speech detection on Facebook. In Proceedings of the First Italian Conference on Cybersecurity (ITASEC17). find
  16. Dinakar, K., Jones, B., Havasi, C., Lieberman, H., and Picard, R. (2012). Common sense reasoning for detection, prevention, and mitigation of cyberbullying. ACM Transactions on Interactive Intelligent Systems (TiiS), 2(3):18. find
  17. Djuric, N., Zhou, J., Morris, R., Grbovic, M., Radosavljevic, V., and Bhamidipati, N. (2015). Hate speech detection with comment embeddings. In Proceedings of the 24th international conference on World Wide Web, pages 29–30. ACM. find
  18. ElSherief, M., Kulkarni, V., Nguyen, D., Wang, W. Y., and Belding, E. (2018). Hate lingo: A target-based linguistic analysis of hate speech in social media. In Twelfth International AAAI Conference on Web and Social Media. find
  19. European Commission. (2018). Countering illegal hate speech online. https://ec.europa.eu/commission/presscorner/detail/en/ MEMO_18_262 [accessed: 2019-11-20]. ↗
  20. Fersini, E., Nozza, D., and Rosso, P. (2018a). Overview of the EVALITA 2018 task on automatic misogyny identification (AMI). In EVALITA@ CLiC-it. find
  21. Fersini, E., Rosso, P., and Anzovino, M. (2018b). Overview of the task on automatic misogyny identification at IberEval 2018. In IberEval@ SEPLN, pages 214– 228. find
  22. Fortuna, P. C. T. (2017). Automatic detection of hate speech in text: an overview of the topic and dataset annotation with hierarchical classes. Master’s thesis, University of Porto. find
  23. Gao, L. and Huang, R. (2017). Detecting online hate speech using context aware models. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, pages 260–266, Varna, Bulgaria, September. INCOMA Ltd. find
  24. Gitari, N. D., Zuping, Z., Damien, H., and Long, J. (2015). A lexicon-based approach for hate speech detection. International Journal of Multimedia and Ubiquitous Engineering, 10(4):215–230. find
  25. Gürcanlı, Ö., Nakipoğlu Demiralp, M., and Özyurek, A. (2007). Shared information and argument omission in Turkish. In 31st Annual Boston University Conference on Language Development, pages 267–273. Cascadilla Press. find
  26. Göksel, A. and Kerslake, C. (2005). Turkish: A Comprehensive Grammar. London: Routledge. find
  27. Ibrohim, M. O. and Budi, I. (2018). A dataset and preliminaries study for abusive language detection in Indonesian social media. Procedia Computer Science, 135:222–229. find
  28. Jaki, S. and De Smedt, T. (2018). Right-wing German hate speech on Twitter: Analysis and automatic detection. Manuscript submitted. find
  29. Karkın, N., Yavuz, N., Parlak, İ., and İkiz, Ö. Ö. (2015). Twitter use by politicians during social uprisings: an analysis of Gezi park protests in Turkey. In Proceedings of the 16th Annual International Conference on Digital Government Research, pages 20–28. ACM. find
  30. Kayany, J. M. (1998). Contexts of uninhibited online behavior: Flaming in social newsgroups on usenet. Journal of the American Society for Information Science, 49(12):1135–1141. find
  31. Kinikoglu, B. (2014). Evaluating the regulation of access to online content in Turkey in the context of freedom of speech. J. Int’t Com. L. & Tech., 9:36. find
  32. Kumar, R., Ojha, A. K., Malmasi, S., and Zampieri, M. (2018a). Benchmarking aggression identification in social media. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018), pages 1–11, Santa Fe, New Mexico, USA, August. Association for Computational Linguistics. find
  33. Kumar, R., Reganti, A. N., Bhatia, A., and Maheshwari, T. (2018b). Aggression-annotated corpus of Hindi-English code-mixed data. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May. European Languages Resources Association (E… find our card
  34. Kwok, I. and Wang, Y. (2013). Locate the hate: Detecting tweets against blacks. In Twenty-seventh AAAI conference on artificial intelligence. find
  35. Lea, M., O’Shea, T., Fung, P., and Spears, R., (1992). ‘Flaming’in computer-mediated communication: Observations, explanations, implications., pages 89–112. Harvester Wheatsheaf. find
  36. Leinonen, T., Çöltekin, Ç., and Nerbonne, J. (2015). Using Gabmap. Lingua, 178:71–83. find
  37. Liu, P., Li, W., and Zou, L. (2019). NULI at SemEval-2019 task 6: Transfer learning for offensive language detection using bidirectional transformers. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 87–91, Minneapolis, Minnesota, USA, June. Association for Computation… find
  38. Malmasi, S. and Zampieri, M. (2018). Challenges in discriminating profanity from hate speech. Journal of Experimental & Theoretical Artificial Intelligence, 30(2):187–202. find
  39. Mehl, M. R. and Pennebaker, J. W. (2003). The sounds of social life: A psychometric analysis of students’ daily social environments and natural conversations. Journal of personality and social psychology, 84(4):857. find
  40. Mojica de la Vega, L. G. and Ng, V. (2018). Modeling trolling in social media conversations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May. European Languages Resources Association (ELRA). find
  41. Montani, J. P. and Schüller, P. (2018). TUWienKBS at GermEval 2018: German abusive Tweet detection. In Proceedings of the GermEval 2018 Workshop at KON-VENS 2018, pages 45–50. find
  42. Mubarak, H., Darwish, K., and Magdy, W. (2017). Abusive language detection on Arabic social media. In Proceedings of the First Workshop on Abusive Language Online, pages 52–56, Vancouver, BC, Canada, August. Association for Computational Linguistics. find
  43. Nerbonne, J., Colen, R., Gooskens, C., Kleiweg, P., and Leinonen, T. (2011). Gabmap – a web application for dialectology. Dialectologia, Special Issue II:65–89. find
  44. Nikolov, A. and Radivchev, V. (2019). Nikolov-radivchev at SemEval-2019 task 6: Offensive tweet classification with BERT and ensembles. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 691–695, Minneapolis, Minnesota, USA, June. Association for Computational Linguistic… find
  45. Nitta, T., Masui, F., Ptaszynski, M., Kimura, Y., Rzepka, R., and Araki, K. (2013). Detecting cyberbullying entries on informal school websites based on category relevance maximization. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 579–586, Nagoya,… find
  46. Nobata, C., Tetreault, J., Thomas, A., Mehdad, Y., and Chang, Y. (2016). Abusive language detection in online user content. In Proceedings of the 25th international conference on World Wide Web, pages 145–153. International World Wide Web Conferences Steering Committee. find
  47. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning… find
  48. Rama, T. and Çöltekin, Ç. (2017). Fewer features perform well at native language identification task. In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, pages 255–260, Copenhagen, Denmark. find
  49. Ross, B., Rist, M., Carbonell, G., Cabrera, B., Kurowsky, N., and Wojatzki, M. (2017). Measuring the reliability of hate speech annotations: The case of the European refugee crisis. arXiv preprint arXiv:1701.08118. — [arXiv:1701.08118](https://arxiv.org/abs/1701.08118) find
  50. Ruhi, Ş. and Işık-Güler, H. (2007). Conceptualizing face and relational work in (im)politeness: Revelations from politeness lexemes and idioms in Turkish. Journal of Pragmatics, 39(4):681–711. find
  51. Ruppenhofer, J., Siegel, M., and Wiegand, M. (2018). Guidelines for IGGSA shared task on the identification of offensive language. https://github.com/uds-lsv/GermEval-2018-Data/ blob/master/guidelines-iggsa-shared.pdf [accessed: 2019-11-20]. ↗
  52. Saka, E. (2018). Social media in Turkey as a space for political battles: AKTrolls and other politically motivated trolling. Middle East Critique, 27(2):161–177. find
  53. Sanguinetti, M., Poletto, F., Bosco, C., Patti, V., and Stranisci, M. (2018). An Italian Twitter corpus of hate speech against immigrants. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May. European Languages Resources Asso… find our card
  54. Schmidt, A. and Wiegand, M. (2017). A survey on hate speech detection using natural language processing. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, pages 1–10, Valencia, Spain, April. Association for Computational Linguistics. find our card
  55. Smith, G. A., Baum, C. R., Dowd, M. D., Durbin, D. R., Quinian, K. P., Sege, R. D., Turner, M. S., Weiss, J. C., and Wright, J. L. (2009). Policy statement-role of the pediatrician in youth violence prevention. Pediatrics, 124(1):393–402. find
  56. Spertus, E. (1997). Smokey: Automatic recognition of hostile messages. In Proceedings of the Fourteenth National conference on Aritficial Intelligence and Innovative Applications of Artificial Intelligence (AAi/IAAI ’97), pages 1058–1065. find
  57. Struß, J. M., Siegel, M., Ruppenhofer, J., Wiegand, M., and Klenner, M. (2019). Overview of GermEval task 2, 2019 shared task on the identification of offensive language. In Preliminary proceedings of the 15th Conference on Natural Language Processing (KONVENS 2019), pages 352– 363, Erlangen, German… find
  58. Sunata, U. and Yıldız, E. (2018). Representation of Syrian refugees in the Turkish media. Journal of Applied Journalism & Media Studies, 7(1):129–151. find
  59. Twitter. (2019). Twitter transparency report: removal requests. https://transparency.twitter.com/en/removal-requests.html [accessed: 24 Oct 2019]. ↗
  60. Van Hee, C., Lefever, E., Verhoeven, B., Mennes, J., Desmet, B., De Pauw, G., Daelemans, W., and Hoste, V. (2015). Detection and fine-grained classification of cyberbullying events. In Proceedings of the International Conference Recent Advances in Natural Language Processing, pages 672–680, Hissar,… find
  61. Wang, W., Chen, L., Thirunarayan, K., and Sheth, A. P. (2014). Cursing in English on Twitter. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing, pages 415–425. ACM. find
  62. Warner, W. and Hirschberg, J. (2012). Detecting hate speech on the World Wide Web. In Proceedings of the Second Workshop on Language in Social Media, pages 19–26, Montréal, Canada, June. Association for Computational Linguistics. find
  63. Waseem, Z. and Hovy, D. (2016). Hateful symbols or hateful people? predictive features for hate speech detection on Twitter. In Proceedings of the NAACL Student Research Workshop, pages 88–93, San Diego, California, June. Association for Computational Linguistics. find
  64. Waseem, Z., Davidson, T., Warmsley, D., and Weber, I. (2017). Understanding abuse: A typology of abusive language detection subtasks. In Proceedings of the First Workshop on Abusive Language Online, pages 78–84, Vancouver, BC, Canada, August. Association for Computational Linguistics. find our card
  65. Waseem, Z. (2016). Are you a racist or am I seeing things? annotator influence on hate speech detection on Twitter. In Proceedings of the First Workshop on NLP and Computational Social Science, pages 138–142, Austin, Texas, November. Association for Computational Linguistics. find
  66. Wiedemann, G., Ruppert, E., Jindal, R., and Biemann, C. (2018). Transfer learning from LDA to BiLSTM-CNN for offensive language detection in Twitter. pages 85–94. find
  67. Wiegand, M., Siegel, M., and Ruppenhofer, J. (2018). Overview of the GermEval 2018 shared task on the identification of offensive language. In Proceedings of the GermEval 2018 Workshop at KONVENS 2018, pages 1– 10. find
  68. Wu, N., DeMattos, E., So, K., Chen, P.-z., and Çöltekin, Ç. (2019). Language discrimination and transfer learning for similar languages: Experiments with feature combinations and adaptation. In Proceedings of the Sixth Workshop on NLP for Similar Languages, Varieties and Dialects, pages 54–63, TOBEF… find
  69. Xu, J.-M., Jun, K.-S., Zhu, X., and Bellmore, A. (2012). Learning from bullying traces in social media. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 656–666, Montréal, Canada, June. Associatio… find
  70. Yesil, B. and Sozeri, E. K. (2017). Online surveillance in Turkey: Legislation, technology and citizen involvement. Surveillance & Society, 15(3/4):543–549. find
  71. Zampieri, M., Malmasi, S., Nakov, P., Rosenthal, S., Farra, N., and Kumar, R. (2019a). Predicting the type and target of offensive posts in social media. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find our card
  72. Zampieri, M., Malmasi, S., Nakov, P., Rosenthal, S., Farra, N., and Kumar, R. (2019b). SemEval-2019 task 6: Identifying and categorizing offensive language in social media (OffensEval). In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 75–86, Minneapolis, Minnesota, USA… find our card
  73. Zeyrek, D., (2001). Politeness in Turkish and its linguistic manifestations, pages 43–73. Amsterdam: Benjamins Publ. Co. find
  74. Çöltekin, Ç. and Rama, T. (2018). Tübingen-Oslo at SemEval-2018 task 2: SVMs perform better than RNNs at emoji prediction. In Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 34––38, New Orleans, LA, United States. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/coltekin-2020-corpus-turkish-offensive-language-social/