Elamur Research⌕
☰
home / papers / 2018 / An Italian Twitter Corpus of Hate Speech against Immigrants

An Italian Twitter Corpus of Hate Speech against Immigrants

Summary

Online hate speech (HS) is hard to detect automatically, partly because a yes/no label hides the many factors that make a message more or less harmful. The authors built an Italian Twitter corpus of about 6,000 tweets that targets HS against immigrants and also includes Roma and Muslims as comparison targets. Tweets were collected with neutral rather than abusive keywords, and the keyword list was then refined using words frequent in tweets already labeled as HS. The multi-layer annotation scheme records HS together with aggressiveness, offensiveness, irony and stereotype, plus an experimental intensity layer. Part of the corpus was labeled by expert annotators and part by CrowdFlower contributors, which allowed the two groups to be compared. Agreement was low in both groups but higher among experts. Intensity proved the most contested layer. The resource is meant as a gold standard for an HS monitoring system. Its richer labels also make it possible to study the linguistic patterns behind each category, not only behind HS itself.

Key points

  • Data collection and corpus size:
    • Keyword filtering produced 236,193 tweets.
    • A first batch of 1,827 tweets was annotated.
    • A second keyword-based retrieval used terms such as invasione, basta, fuori and barcon*, and yielded 4,182 more tweets.
    • The final corpus contains 6,009 tweets.
  • Expert annotation: 3,154 tweets (the first batch plus 1,327 new tweets) were labeled by four experts working in pairs. A fifth annotator resolved any tweet where the pair disagreed on at least one category.
  • CrowdFlower annotation:
    • 2,855 tweets were labeled on CrowdFlower.
    • 600 gold tweets served as test questions.
    • Contributor reliability was checked on the HS answers only, with a minimum of 65%.
    • Five contributors produced more than 90% of the judgments.
  • Agreement measures: Cohen's κ was used for the expert pairs and Krippendorff's α for the crowd, where each tweet had at least three judgments. Both were low, and the experts were more consistent. Intensity was the layer where experts disagreed most.
  • Availability: The corpus and guidelines were planned for free non-commercial release by the end of 2018 (GitHub: msang/hate-speech-corpus).

Interesting ideas

  • I1. Neutral seed keywords. Collecting with neutral, topic-related words and then adding words that frequently co-occur with HS in already-labeled data captures hate that contains no slurs. This bootstrapping loop could work for any harmful-content dataset where a profanity lexicon would bias the sample.
  • I2. Separate, independent layers. HS, offensiveness and stereotype are annotated as distinct dimensions, so a stereotype alone does not count as hate. Keeping related but different phenomena apart prevents models from conflating rudeness with hate.
  • I3. Lenient crowd quality control for complex schemes. When a task has many multi-choice questions, exact-match checking discards too many workers. Scoring reliability only on the core label, and removing test questions that workers repeatedly and reasonably contest, keeps the quality filter fair.
  • I4. Experts versus crowd as a diagnostic. Annotating different subsets with experts and crowd workers, with agreement measures suited to each setup, reveals which layers are inherently ambiguous and which suffer from inattentive annotators.
  • I5. Simplify layers that rest on pragmatics. The intensity results suggest that fine scales built on pragmatic cues, such as first-person versus impersonal phrasing, are unreliable. A coarser weak/strong split may be a better starting point for subjective graded labels.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

83 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 79 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{sanguinetti2018italian,
  title = {{An Italian Twitter Corpus of Hate Speech against Immigrants}},
  author = {Sanguinetti, Manuela and Poletto, Fabio and Bosco, Cristina and Patti, Viviana and Stranisci, Marco},
  booktitle = {Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)},
  year = {2018},
  doi = {10.63317/26vnss59hfcq},
  url = {https://doi.org/10.63317/26vnss59hfcq}
}

References

18 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aroyo, L. and Welty, C. (2015). Truth is a lie: Crowd truth and the seven myths of human annotation. Artificial Intelligence Magazine, pages 15–24. find our card
  2. Bosco, C., Patti, V., and Bolioli, A. (2013). Developing corpora for sentiment analysis: The case of irony and SENTI-TUT. IEEE Intelligent Systems, 28(2):55–63. find
  3. Bosco, C., Viviana, P., Bogetti, M., Conoscenti, M., Ruffo, G., Schifanella, R., and Stranisci, M. (2017). Tools and resources for detecting hate and prejudice against immigrants in social media. In Proceedings of First Symposium on Social Interactions in Complex Intelligent Systems (SICIS), AISB Co… find
  4. Brown, R. (2011). Prejudice: Its Social Psychology. Wiley. find
  5. Carletta, J. (1996). Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22(2):249–254, June. find our card
  6. Del Vigna, F., Cimino, A., Dell’Orletta, F., Petrocchi, M., and Tesconi, M. (2017). Hate me, hate me not: Hate speech detection on Facebook. In Proceedings of the First Italian Conference on Cybersecurity (ITASEC17), Venice, Italy, January 17-20, 2017., pages 86–95. find
  7. Erjavec, K. and Kovačič, M. P. (2012). ”You don’t understand, this is a new war!” Analysis of hate speech in news web sites’ comments. Mass Communication and Society, 15(6):899–920. find
  8. Krippendorff, K. (2007). Computing Krippendorff’s alpha reliability. Departmental papers (ASC) 43. find
  9. Kwok, I. and Wang, Y. (2013). Locate the hate: Detecting tweets against blacks. In Proceedings of the Twenty-Seventh AAAI Conference on Artificial Intelligence, pages 1621–1622. find
  10. Musto, C., Semeraro, G., de Gemmis, M., and Lops, P. (2016). Modeling community behavior through semantic analysis of social data: The italian hate map experience. In Proceedings of the 2016 Conference on User Modeling Adaptation and Personalization, UMAP 2016, Halifax, NS, Canada, July 13 - 17, 201… find
  11. Poletto, F., Stranisci, M., Sanguinetti, M., Patti, V., and Bosco, C. (2017). Hate speech annotation: Analysis of an italian Twitter corpus. In Proceedings of the Fourth Italian Conference on Computational Linguistics (CLiC-it 2017), volume 2006 of CEUR Workshop Proceedings, Rome, Italy. CEUR-WS.org… find
  12. Ross, B., Rist, M., Carbonell, G., Cabrera, B., Kurowsky, N., and Wojatzki, M. (2017). Measuring the reliability of hate speech annotations: The case of the European refugee crisis. CoRR, abs/1701.08118. — [arXiv:1701.08118](https://arxiv.org/abs/1701.08118) find
  13. Searle, J. R. (1969). Speech acts: an essay in the philosophy of language, volume 626. Cambridge University Press. find
  14. Silva, L., Mondal, M., Correa, D., Benevenuto, F., and Weber, I. (2016). Analyzing the targets of hate in online social media. In Proceedings of the 10th International Conference on Web and Social Media, ICWSM 2016, pages 687–690. AAAI Press. find
  15. Warner, W. and Hirschberg, J. (2012). Detecting hate speech on the world wide web. In Proceedings of the Second Workshop on Language in Social Media, LSM ’12, pages 19–26, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  16. Waseem, Z. and Hovy, D. (2016). Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In Proceedings of the NAACL Student Research Workshop, pages 88–93, San Diego, California, June. Association for Computational Linguistics. find
  17. Waseem, Z. (2016). Are you a racist or am i seeing things? annotator influence on hate speech detection on Twitter. In Proceedings of the First Workshop on NLP and Computational Social Science, pages 138–142, Austin, Texas, November. Association for Computational Linguistics. find
  18. Weber, A. (2009). Manual on hate speech. Council of Europe. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/sanguinetti-2018-italian-twitter-corpus-hate-speech/