Elamur Research⌕
☰
home / papers / 2018 / Overview of the EVALITA 2018 Hate Speech Detection Task

Overview of the EVALITA 2018 Hate Speech Detection Task

Summary

HaSpeeDe was the first shared evaluation campaign for detecting hate speech (HS) in Italian social media, held at EVALITA 2018. The organizers took two existing Italian corpora, Facebook comments from Pisa and tweets from Turin, and reduced their richer annotation schemes to one common binary label (HS present or absent). From each corpus they released 4,000 messages, split into 3,000 for development and 1,000 for testing. The campaign had three tasks: two in-domain tasks (Facebook→Facebook and Twitter→Twitter) and a cross-platform task, in which systems trained on one network were evaluated on the other. Nine teams took part, and macro F1 was the official metric. Top scores were 0.8288 on HaSpeeDe-FB and 0.7993 on HaSpeeDe-TW. Cross-platform results were clearly lower, around 0.65–0.70. The organizers attribute the drop mainly to different class distributions and target variety in the two corpora. The campaign gave Italian NLP a common benchmark for HS detection and showed that HS classifiers transfer poorly between platforms.

Key points

  • Tasks: HaSpeeDe-FB, HaSpeeDe-TW, and Cross-HaSpeeDe, which has two sub-tasks.
    • Cross-HaSpeeDe FB trains on Facebook and tests on Twitter.
    • Cross-HaSpeeDe TW trains on Twitter and tests on Facebook.
    • Each team could submit up to 2 runs per task.
  • Facebook source corpus (Del Vigna et al., 2017):
    • 17,567 comments were collected from 99 posts on public pages likely to attract hateful replies.
    • Five students annotated them, and 3,685 comments received at least three annotations.
    • The original labels were no/weak/strong hate, plus several hate categories.
  • Twitter source corpus (Sanguinetti et al., 2018):
    • It contains 6,928 tweets collected with neutral keywords about immigrants, Muslims and Roma.
    • Its annotation scheme is rich: HS, intensity, aggressiveness, offensiveness, irony and stereotype.
    • Some of the ironic tweets were also reused in IronITA.
  • Class balance differs across sets:
    • In the Facebook data, HS makes up about 46% of the training set and 68% of the test set.
    • In the Twitter data, HS makes up about 32% of both sets.
  • Results:
    • ItaliaNLP placed first on both in-domain tasks and on Cross-HaSpeeDe TW (0.6985). It combined a polarity/subjectivity lexicon with word embeddings.
    • The best Cross-HaSpeeDe FB score was about 0.654.

Interesting ideas

  • I1. Harmonizing labels across corpora. Two corpora built separately, with different annotation schemes, were mapped onto one shared minimal label. This turns existing resources into a joint benchmark without new annotation. The approach could suit low-resource languages where several small, incompatible datasets already exist.
  • I2. Testing in both cross-domain directions. Training on platform A and testing on B, and also the reverse, exposes asymmetric failures. Here, systems over-predicted HS on Twitter and under-predicted it on Facebook. A single in-domain score would miss this kind of generalization weakness.
  • I3. Reading cross-domain errors through label priors. The organizers explained the cross-domain drop largely by different class proportions and target diversity, rather than by model quality alone. Checking label distributions and target coverage is a quick diagnostic before blaming the classifier in any domain-transfer setup.
  • I4. Keeping a deliberately simple release format. The data shipped as a three-column TSV (ID, text, binary label) with pseudonymized IDs to comply with GDPR. Participation stays easy, and the original rich annotations can be kept for later, finer-grained tasks. The same pattern fits releasing sensitive social-media data responsibly.
  • I5. Combining an affective lexicon with embeddings. The organizers credit the winning system's robustness across settings to pairing a polarity/subjectivity lexicon with embeddings, since embeddings alone did less well. This is a cheap way to add domain-independent signal to classifiers for subjective phenomena.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

83 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 79 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{bosco2018overview,
  title = {{Overview of the EVALITA 2018 Hate Speech Detection Task}},
  author = {Bosco, Cristina and Dell'Orletta, Felice and Poletto, Fabio and Sanguinetti, Manuela and Tesconi, Maurizio},
  booktitle = {Proceedings of the Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2018)},
  year = {2018},
  doi = {10.4000/books.aaccademia.4503},
  url = {https://doi.org/10.4000/books.aaccademia.4503}
}

References

44 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Xiaoyu Bai, Flavio Merenda, Claudia Zaghi, Tommaso Caselli, and Malvina Nissim. 2018. RuG @ EVALITA 2018: Hate Speech Detection In Italian Social Media. In Proceedings of Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. find
  2. Final Workshop (EVALITA 2018). CEUR.org. find
  3. Francesco Barbieri, Valerio Basile, Danilo Croce, Malvina Nissim, Nicole Novielli, and Viviana Patti. 2016. Overview of the Evalita 2016 SENTIment POLarity Classification Task. In Proceedings of the Fifth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. find
  4. Final Workshop (EVALITA 2016). find
  5. Giulio Bianchini, Lorenzo Ferri, and Tommaso Giorni. 2018. Text Analysis for Hate Speech Detection in Italian Messages on Twitter and Facebook. In Proceedings of Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. find
  6. Final Workshop (EVALITA 2018). CEUR.org. find
  7. Cristina Bosco, Patti Viviana, Marcello Bogetti, Michelangelo Conoscenti, Giancarlo Ruffo, Rossano Schifanella, and Marco Stranisci. 2017. Tools and Resources for Detecting Hate and Prejudice Against Immigrants in Social Media. In Proceedings of First Symposium on Social Interactions in Complex Inte… find
  8. Pete Burnap and Matthew L. Williams. 2015. Cyber Hate Speech on Twitter: An Application of Machine Classification and Statistical Modeling for Policy and Decision Making. Policy & Internet, 7(2). find
  9. Miguel Ángel Álvarez Carmona, Estefanı́a Guzmán-Falcón, Manuel Montes-y-Gómez, Hugo Jair Escalante, Luis Villaseñor Pineda, Verónica Reyes-Meza, and Antonio Rico Sulayes. 2018. Overview of MEX-A3T at IberEval 2018: Authorship and Aggressiveness Analysis in Mexican Spanish Tweets. In IberEval@… find
  10. Tommaso Caselli, Nicole Novielli, Viviana Patti, and Paolo Rosso. 2018. find
  11. EVALITA 2018: Overview of the 6th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. In Proceedings of Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. find
  12. Final Workshop (EVALITA 2018). CEUR.org. find
  13. Alessandra Teresa Cignarella, Simona Frenda, Valerio Basile, Cristina Bosco, Viviana Patti, and Paolo Rosso. 2018. Overview of the Evalita 2018 Task on Irony Detection in Italian Tweets (IronITA). In Proceedings of Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian… find our card
  14. Final Workshop (EVALITA 2018). CEUR.org. find
  15. Andrea Cimino, Lorenzo De Mattei, and Felice Dell’Orletta. 2018. Multi-task Learning in Deep Neural Networks at EVALITA 2018. In Proceedings of the 6th evaluation campaign of Natural Language Processing and Speech tools for Italian (EVALITA’18), Turin, Italy. CEUR.org. find
  16. Michele Corazza, Stefano Menini, Pinar Arslan, Rachele Sprugnoli, Elena Cabrio, Sara Tonelli, and Serena Villata. 2018. Comparing Different Supervised Approaches to Hate Speech Detection. In Proceedings of Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. find
  17. Final Workshop (EVALITA 2018). CEUR.org. find
  18. Thomas Davidson, Dana Warmsley, Michael W. Macy, and Ingmar Weber. 2017. Automated Hate Speech Detection and the Problem of Offensive Language. CoRR, abs/1703.04009. find our card
  19. Gretel Liz De la Peña Sarracén, Reynaldo Gil Pons, Carlos Enrique Muñiz Cuza, and Paolo Rosso. 2018. Hate Speech Detection Using Attention-based LSTM. In Proceedings of Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. find
  20. Final Workshop (EVALITA 2018). CEUR.org. find
  21. Fabio Del Vigna, Andrea Cimino, Felice Dell’Orletta, Marinella Petrocchi, and Maurizio Tesconi. 2017. Hate Me, Hate Me Not: Hate Speech Detection on Facebook. In Proceedings of the First Italian Conference on Cybersecurity (ITASEC17). find
  22. Karmen Erjavec and Melita Poler Kovačič. 2012. ”You Don’t Understand, This is a New War!” Analysis of Hate Speech in News Web Sites’ Comments. Mass Communication and Society, 15(6). find
  23. Elisabetta Fersini, Debora Nozza, and Paolo Rosso. 2018a. Overview of the EVALITA 2018 Task on Automatic Misogyny Identification (AMI). In Proceedings of Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. find our card
  24. Final Workshop (EVALITA 2018). CEUR.org. find
  25. Elisabetta Fersini, Paolo Rosso, and Maria Anzovino. 2018b. Overview of the Task on Automatic Misogyny Identification at IberEval 2018. In IberEval@SEPLN. CEUR-WS.org. find
  26. Paula Fortuna, Ilaria Bonavita, and Sérgio Nunes. 2018. Merging datasets for hate speech classification in Italian. In Proceedings of Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. find
  27. Final Workshop (EVALITA 2018). CEUR.org. find
  28. Björn Gambäck and Utpal Kumar Sikdar. 2017. Using Convolutional Neural Networks to Classify Hate-Speech. In Proceedings of the First Workshop on Abusive Language. find our card
  29. Njagi Dennis Gitari, Zhang Zuping, Hanyurwimfura Damien, and Jun Long. 2015. A lexicon-based approach for hate speech detection. International Journal of Multimedia and Ubiquitous Engineering, 10(4). find
  30. Ritesh Kumar, Atul Kr. Ojha, Marcos Zampieri, and Shervin Malmasi, editors. 2018. Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018). Association for Computational Linguistics. find
  31. Irene Kwok and Yuzhou Wang. 2013. Locate the Hate: Detecting Tweets Against Blacks. In Proceedings of the Twenty-Seventh AAAI Conference on Artificial Intelligence. AAAI Press. find
  32. Yashar Mehdad and Joel Tetreault. 2016. Do Characters Abuse More Than Words? In 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue. find
  33. Cataldo Musto, Giovanni Semeraro, Marco de Gemmis, and Pasquale Lops. 2016. Modeling Community Behavior through Semantic Analysis of Social Data: The Italian Hate Map Experience. In Proceedings of the 2016 Conference on User Modeling Adaptation and Personalization, UMAP 2016. find
  34. Serena Pelosi, Alessandro Maisto, Pierluigi Vitale, and Simonetta Vietri. 2017. Mining Offensive Language on Social Media. In Proceedings of the Fourth Italian Conference on Computational Linguistics (CLiC-it 2017). find
  35. Fabio Poletto, Marco Stranisci, Manuela Sanguinetti, Viviana Patti, and Cristina Bosco. 2017. Hate Speech Annotation: Analysis of an Italian Twitter Corpus. In Proceedings of the Fourth Italian Conference on Computational Linguistics (CLiC-it 2017). CEUR. find
  36. Marco Polignano and Pierpaolo Basile. 2018. HanSEL: Italian Hate Speech Detection through Ensemble Learning and Deep Neural Networks. In Proceedings of Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. find
  37. Final Workshop (EVALITA 2018). CEUR.org. find
  38. Manuela Sanguinetti, Fabio Poletto, Cristina Bosco, Viviana Patti, and Marco Stranisci. 2018. An Italian Twitter Corpus of Hate Speech against Immigrants. In Proceedings of the 11th Language Resources and Evaluation Conference 2018. find our card
  39. Valentino Santucci, Stefania Spina, Alfredo Milani, Giulio Biondi, and Gabriele Di Bari. 2018. Detecting Hate Speech for Italian Language in Social Media. In Proceedings of Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. find
  40. Final Workshop (EVALITA 2018). CEUR.org. find
  41. Anna Schmidt and Michael Wiegand. 2017. A Survey on Hate Speech Detection using Natural Language Processing. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media. Association for Computational Linguistics. find our card
  42. Dirk von Grünigen, Ralf Grubenmann, Fernando Benites, Pius Von Däniken, and Mark Cieliebak. 2018. spMMMP at GermEval 2018 Shared Task: Classification of Offensive Content in Tweets using Convolutional Neural Networks and Gated Recurrent Units. In Proceedings of GermEval 2018, 14th Conference on Na… find
  43. Zeerak Waseem, Wendy Hui Kyong Chung, Dirk Hovy, and Joel Tetreault, editors. 2017. Proceedings of the First Workshop on Abusive Language Online. Association for Computational Linguistics. find
  44. Michael Wiegand, Melanie Siegel, and Josef Ruppenhofer. 2018. Overview of the GermEval 2018 Shared Task on the Identification of Offensive Language. In Proceedings of GermEval 2018, 14th Conference on Natural Language Processing (KON-VENS 2018). find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bosco-2018-overview-evalita-2018-hate-speech/