Elamur Research⌕
☰
home / papers / 2019 / A Hierarchically-Labeled Portuguese Hate Speech Dataset

A Hierarchically-Labeled Portuguese Hate Speech Dataset

Paper · 2019 · Paula Fortuna, João Rocha da Silva, Juan Soler-Company et al.

Summary

Most annotated resources for hate speech detection cover English, and most of them reduce the task to a yes/no decision or a few broad classes such as racism or sexism. Fortuna et al. address both gaps with a Portuguese Twitter corpus of 5,668 tweets. They gathered the tweets through Twitter's API in two ways: searching hate-related keywords and hashtags, and crawling profiles found with hate-related queries. Annotation happened in two layers. Non-expert annotators first gave each tweet a binary hate/no-hate label. Expert annotators then applied a hierarchical, multilabel scheme of 81 categories organized around the groups being targeted. Annotator agreement differed considerably across categories, which suggests that some forms of hate are subtler and more perception-dependent than others. As a sanity check, the authors trained a baseline on the binary labels: pre-trained word embeddings, an LSTM, and xgBoost on top. It reached F1 scores the authors consider competitive. The resource is useful because it brings Portuguese into hate speech research and lets researchers analyse subtypes of hate and their overlaps.

Key points

  • The corpus has 5,668 Portuguese tweets and is released on GitHub (paulafortuna/Portuguese-Hate-Speech-Dataset) for research use only.
  • The binary layer uses labels from three non-expert annotators per tweet as ground truth. The expert layer is a hierarchical multilabel taxonomy of 81 hate categories.
  • The scheme adopts the hate speech definition of Fortuna and Nunes (2018) and draws on the 29-category taxonomy of Salminen et al. (2018). Unlike Salminen et al., it covers only targets of hate and leaves out discourse type and abstract entities such as media or religion.
  • In the baseline, embeddings from the last LSTM layer are fed into xgBoost. Micro-averaged F1 is 0.78 in cross-validation (N=5,099) and 0.72 on the test set (N=567).
  • Tweet ids were removed for privacy. The authors acknowledge sampling bias from both the Twitter API and their own keyword and profile choices.

Interesting ideas

  • I1. Two-tier annotation by expertise: Cheap non-expert labels can cover a coarse binary decision, while scarce experts handle the detailed taxonomy. This saves expert effort in any labelling project with a fine-grained schema.
  • I2. Taxonomy built on targets only: The authors keep the group being attacked separate from the style of the message, so that categories do not mix two dimensions. This makes it easier to study where categories intersect, and it applies to any multi-aspect labelling scheme.
  • I3. Complementary collection channels: Keyword search returns many authors but only explicit vocabulary. Profile crawling returns more natural discourse from fewer users. Combining both is a useful pattern for collecting rare phenomena from social media.
  • I4. Per-category agreement as a diagnostic: Low agreement on a category signals subtlety or subjectivity rather than just noise. It can show which guidelines need sharper definitions or which annotators need more training.
  • I5. Neural features plus a boosted classifier: Using LSTM-derived representations as input to gradient-boosted trees gives a strong, simple baseline for validating a new labelled dataset.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Universitat Pompeu FabraBarcelona · Spain36 works · 11,433 citations🎓Universidade do PortoPorto · Portugal11 works · 1,963 citations🔬INESC TECPorto · Portugal8 works · 1,176 citations🤝Institució Catalana de Recerca i Estudis AvançatsBarcelona · Spain5 works · 445 citations🤝Institute for Systems Engineering and ComputersLisbon · Portugal3 works · 361 citations

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 works

🧭 Topics 2Research subjects and languages the work is about.Sentiment & emotion · 608 worksBias, fairness & safety · 209 works

Cite

Cite as Paula Fortuna, João Rocha da Silva, Juan Soler-Company, Leo Wanner and Sérgio Nunes. 2019. A Hierarchically-Labeled Portuguese Hate Speech Dataset. Proceedings of the Third Workshop on Abusive Language Online.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-3510

@inproceedings{fortuna2019hierarchically,
  title = {{A Hierarchically-Labeled Portuguese Hate Speech Dataset}},
  author = {Paula Fortuna, João Rocha {da Silva}, Juan Soler-Company, Leo Wanner and Sérgio Nunes},
  booktitle = {Proceedings of the Third Workshop on Abusive Language Online},
  year = {2019},
  doi = {10.18653/v1/w19-3510},
  url = {https://doi.org/10.18653/v1/w19-3510}
}

References

30 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pinkesh Badjatiya, Shashank Gupta, Manish Gupta, and Vasudeva Varma. 2017. Deep learning for hate speech detection in tweets. In Proceedings of the 26th International Conference on World Wide Web Companion, pages 759–760. International World Wide Web Conferences Steering Committee. find
  2. Björn Gambäck and Utpal Kumar Sikdar. 2017. Using Convolutional Neural Networks to Classify Hate-speech. In Proceedings of the First Workshop on Abusive Language Online, pages 85–90.
  3. Tianqi Chen and Carlos Guestrin. 2016. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, pages 785–794, New York, NY, USA. ACM. find
  4. Matthias Gamer, Jim Lemon, Maintainer Matthias Gamer, A Robinson, and W Kendall’s. 2012. Package ?irr? Various coefficients of interrater reliability and agreement. find
  5. François Chollet et al. 2015. Keras. https://kera s.io, accessed last time in February 2019. find
  6. Barney G Glaser and Anselm L Strauss. 2017. Discovery of grounded theory: Strategies for qualitative research. Routledge. find
  7. Francois Chollet. 2017. Deep learning with python. Manning Publications Co. find
  8. Pei-Yi Hao, Jung-Hsien Chiang, and Yi-Kun Tu. 2007. Hierarchically svm classification based on support vector clustering method and its application to document categorization. Expert Systems with applications, 33(3):627–635. find
  9. Patricia Hill Collins. 2015. Intersectionality’s definitional dilemmas. Annual Review of Sociology, 41:1– 20. find
  10. Kimberle Crenshaw. 2018. Demarginalizing the intersection of race and sex: A black feminist critique of antidiscrimination doctrine, feminist theory, and antiracist politics [1989]. In Feminist legal theory, pages 57–80. Routledge. find
  11. Nathan Hartmann, Erick Fonseca, Christopher Shulby, Marcos Treviso, Jéssica Silva, and Sandra Aluı́sio. 2017. Portuguese word embeddings: Evaluating on word analogies and natural language tasks. In Radim Řehůřek and Petr Sojka. 2010. Software Framework for Topic Modelling with Large Corpora. In… find
  12. Proceedings of the 11th Brazilian Symposium in Information and Human Language Technology, pages 122–131, Uberlândia, Brazil. Sociedade Brasileira de Computação. find
  13. Hatebase. 2019. Hatebase. Available in https://www.hatebase.org/, accessed last time in February 2019. ↗
  14. Björn Ross, Michael Rist, Guillermo Carbonell, Ben Cabrera, Nils Kurowsky, and Michael Wojatzki. 2016. Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis. In Proceedings of NLP4CMC III: 3rd Workshop on Natural Language Processing for Computer-Mediated Comm… find
  15. Jigsaw. 2018. Toxic comment classification challenge identify and classify toxic online comments. Available in https://www.kaggle.com/c/jigsaw-toxic -comment-classification-challenge, accessed last time in 23 May 2018. ↗
  16. Ritesh Kumar, Atul Kr. Ojha, Shervin Malmasi, and Marcos Zampieri. 2018. Benchmarking Aggression Identification in Social Media. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbulling (TRAC), Santa Fe, USA.
  17. Joni Salminen, Hind Almerekhi, Milica Milenković, Soon-gyo Jung, Jisun An, Haewoon Kwak, and Bernard J Jansen. 2018. Anatomy of online hate: developing a taxonomy and machine learning models for identifying and classifying hate in online news media. In Twelfth International AAAI Conference on Web a… find
  18. Yashar Mehdad and Joel Tetreault. 2016. Do characters abuse more than words? In Proceedings of the SIGdial 2016 Conference: The 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 299–303. find
  19. Manuela Sanguinetti, Fabio Poletto, Cristina Bosco, Viviana Patti, and Marco Stranisci. 2018. An italian Twitter corpus of hate speech against immigrants. In Proceedings of LREC.
  20. William A Schabas. 2000. Hate speech in rwanda: The road to genocide. McGill Law Journal, 46:141. find
  21. Chikashi Nobata, Joel Tetreault, Achint Thomas, Yashar Mehdad, and Yi Chang. 2016. Abusive language detection in online user content. In Proceedings of the 25th International Conference on World Wide Web, pages 145–153. International World Wide Web Conferences Steering Committee. find
  22. Anna Schmidt and Michael Wiegand. 2017. A survey on hate speech detection using natural language processing.
  23. William Warner and Julia Hirschberg. 2012. Detecting hate speech on the world wide web. In Proceedings of the Second Workshop on Language in Social Media, pages 19–26. Association for Computational Linguistics.
  24. Ji Ho Park and Pascale Fung. 2017. One-step and Two-step Classification for Abusive Language Detection on Twitter. In Proceedings of the First Workshop on Abusive Language Online.
  25. Zeerak Waseem. 2016. Are you a racist or am i seeing things? annotator influence on hate speech detection on Twitter. In Proceedings of the 1st Workshop on Natural Language Processing and Computational Social Science, pages 138–142.
  26. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:282… find
  27. Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In Proceedings of NAACL-HLT, pages 88–93.
  28. Rogers Prates de Pelle and Viviane P Moreira. 2017. Offensive comments in the brazilian web: a dataset and baseline results. In 6 o Brazilian Workshop on Social Network Analysis and Mining (BraSNAM 2017), volume 6. SBC. . find
  29. Georgios K Pitsilis, Heri Ramampiaro, and Helge Langseth. 2018. Detecting offensive language in tweets using deep learning. arXiv preprint arXiv:1801.04433. find
  30. Fabio Poletto, Marco Stranisci, Manuela Sanguinetti, Viviana Patti, and Cristina Bosco. 2017. Hate speech annotation: Analysis of an italian Twitter corpus. In Ceur Workshop Proceedings, volume 2006, pages 1–6. CEUR-WS. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/fortuna-2019-hierarchically-labeled-portuguese-hate-speech/