Elamur Research⌕
☰
home / papers / 2025 / A Disaggregated Dataset on English Offensiveness Containing Spans

A Disaggregated Dataset on English Offensiveness Containing Spans

Paper · 2025 · Pia Pachinger, Janis Goldzycher, Anna M. Planitzer et al.

Summary

Moderation systems usually learn from one majority-vote label per document. That hides how differently people judge harm, and it does not show which words caused a judgement. The authors address both problems by re-annotating 1,983 English Wikipedia comments from the Jigsaw Toxic Comment Classification Challenge. They keep every annotator's individual labels instead of collapsing them. Each post gets a binary offensive/not-offensive label. Annotators also mark spans for vulgar wording in any post and, in offensive posts, spans for the target, typed as individual, group or other. The schema follows the German AustroTox dataset, so the two resources can be compared across languages.

A manual study shows that five judgements per item are enough to tell genuine disagreement apart from sloppy annotation. Disagreement concentrates on:

  • borderline cases,
  • attacks on non-human targets,
  • words whose acceptability depends on context.

Fine-tuned encoders learn the post-level and vulgarity tasks well, but target spans remain hard. The release adds a perspectivist, explainable benchmark under CC BY 4.0.

Key points

  • 1,983 Jigsaw comments were re-annotated, and 1,561 of them have labels from four or five annotators. The annotations are on Hugging Face (pia-p/disaggregated_offensiveness) under CC BY 4.0; the comment text remains under Wikipedia's CC-SA-3.0.

  • The schema is adapted from AustroTox. Because few posts were labelled incite to hate or violence, the authors merged insult and incitement into a single offensive class.

  • If a target is mentioned only through a pronoun, annotators mark that pronoun as the target span.

  • Ten-fold cross-validation on the aggregated labels gives these results for RoBERTa-large:

    TaskScore
    Post-level offensivenessbinary F1 ≈ .90
    Vulgarity tokensbinary F1 ≈ .77
    Target tokensmicro F1 ≈ .27

    Electra-large scores slightly lower. Mistral, Llama 3, GPT-3.5 and GPT-4 were run zero- and five-shot without fine-tuning and trail the fine-tuned encoders, especially on vulgar-token detection.

  • The aggregation scheme leans toward minority "toxic" votes, and models trained on it produce fewer false negatives.

Interesting ideas

  • I1. Store every annotator's label and use high disagreement as evidence that an item is subjective, rather than discarding it as noise; five votes were enough to separate this from plain mistakes.
  • I2. Vulgar wording is annotated separately from whether the post as a whole is offensive, so crude words in harmless posts are captured, and the same word can be acceptable in one context and not another.
  • I3. When a target appears only as a pronoun, the pronoun itself is the target span; repeated and pronominal mentions were exactly where humans and models struggled most.
  • I4. An aggregation rule that leans toward minority positive votes lowers false negatives, which is a deliberate trade-off for safety-critical labels.
  • I5. Micro-F1 for sparse span tasks is computed only over the positive classes, so the huge "no target" class cannot inflate the score; untuned LLM numbers are reported explicitly as a lower bound.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when several witness parsers disagree on silver data, keep their individual votes and flag high-disagreement sentences as genuinely ambiguous instead of forcing one consensus tree.
  • I2 (frames): keep the lexical register of a word apart from the speech act it performs, so a coarse word does not automatically make an utterance hostile or ironic.
  • I3 (frames): target and patient roles should be resolved through coreference chains, and evaluation should accept any mention in the chain rather than one fixed span.
  • I4 (learn): for rare but costly categories, such as negation or implicature flags, a minority-sensitive merge of teacher votes could cut missed cases.
  • I5 (learn): score sparse role or entity spans without the dominant null label, and label untuned baselines honestly as lower bounds.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of ZurichZurich · Switzerland80 works · 12,260 citations🎓University of ViennaVienna · Austria18 works · 38,729 citations🎓TU WienVienna · Austria8 works · 786 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,379 works

Cite

Cite as Pia Pachinger, Janis Goldzycher, Anna M. Planitzer, Julia Neidhardt and Allan Hanbury. 2025. A Disaggregated Dataset on English Offensiveness Containing Spans. Proceedings of the The 4th Workshop on Perspectivist Approaches to NLP.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.nlperspectives-1.1

@inproceedings{pachinger2025disaggregated,
  title = {{A Disaggregated Dataset on English Offensiveness Containing Spans}},
  author = {Pia Pachinger, Janis Goldzycher, Anna M. Planitzer, Julia Neidhardt and Allan Hanbury},
  booktitle = {Proceedings of the The 4th Workshop on Perspectivist Approaches to NLP},
  year = {2025},
  doi = {10.18653/v1/2025.nlperspectives-1.1},
  url = {https://doi.org/10.18653/v1/2025.nlperspectives-1.1}
}

References

43 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aida Mostafazadeh Davani, M. C. D’iaz, Dylan K. Baker, and Vinodkumar Prabhakaran. 2023. Disentangling perceptions of offensiveness: Cultural and moral correlates. Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency. find
  2. Gavin Abercrombie, Dirk Hovy, and Vinodkumar Prabhakaran. 2023. Temporal and second language influence on intra-annotator agreement and stability in hate speech labelling. In Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII), pages 96–103, Toronto, Canada. Association for Computational Linguistics. find
  3. OpenAI et al. 2024. Gpt-4 technical report.
  4. Simona Frenda, Gavin Abercrombie, Valerio Basile, Alessandro Pedrani, Raffaella Panizzon, Alessandra Teresa Cignarella, Cristina Marco, and Davide Bernardi. 2024. Perspectivist approaches to natural language processing: a survey. Language Resources and Evaluation, pages 1–28. find
  5. AI@Meta. 2024. Llama 3 model card. find
  6. Lora Aroyo, Alex Taylor, Mark Diaz, Christopher Homan, Alicia Parrish, Gregory Serapio-García, Vinodkumar Prabhakaran, and Ding Wang. 2023. Dices dataset: Diversity in conversational ai evaluation for safety. Advances in Neural Information Processing Systems, 36:53330–53342. find
  7. Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1161–1166. find
  8. Enrico Bunde. 2021. Ai-assisted and explainable hate speech detection for social media moderators–a design science approach. Proceedings of the 54th Hawaii International Conference on System Sciences. find
  9. Federico Cabitza, Andrea Campagner, and Valerio Basile. 2023. Toward a perspectivist turn in ground truthing for predictive computing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 6860–6868.
  10. Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. 2022. Challenges and strategies in cross-cultural NLP. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6997–7013, Dublin, Ireland. Association for Computational Linguistics.
  11. Agostina Calabrese, Leonardo Neves, Neil Shah, Maarten Bos, Björn Ross, Mirella Lapata, and Francesco Barbieri. 2024. Explainability and hate speech: Structured explanations make social media moderators faster. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 398–408. find
  12. Agostina Calabrese, Björn Ross, and Mirella Lapata. 2022a. Explainable abuse detection as intent classification and slot filling. Transactions of the Association for Computational Linguistics, 10:1440–1454. find
  13. Shagun Jhaver, Alice Qian Zhang, Quan Ze Chen, Nikhila Natarajan, Ruotong Wang, and Amy X Zhang. 2023. Personalizing content moderation on social media: User perspectives on moderation choices, interface design, and labor. Proceedings of the ACM on Human-Computer Interaction, 7(CSCW2):1–33. find
  14. Agostina Calabrese, Björn Ross, and Mirella Lapata. 2022b. Explainable abuse detection as intent classification and slot filling. Transactions of the Association for Computational Linguistics, 10:1440–1454. cjadams, Jeffrey Sorensen, Julia Elliott, Lucas Albert Q Jiang, Alexandre Sablayrolles, Arthur Men-Dixon, Mark McDonald, nithum, and Will sch, Chris Bamford, Devendra Singh Chaplot, Diego Cukierski. 2017. Toxic comment classification de las Casas, Florian Bressand, Gianna Lengyel, Guil-challenge. https://kaggle.com/competitions/ laume Lample, Lucile Saulnier, et al. 2023. Mistral jigsaw-toxic-comment-classification-challenge. 7b. arXiv preprint arXiv:2310.06825. Kaggle. ↗
  15. Chris J Kennedy, Geoff Bacon, Alexander Sahn, and Kevin Clark, Minh-Thang Luong, Quoc V Le, and Claudia von Vacano. 2020. Constructing interval Christopher D Manning. 2020. Electra: Pre-training variables via faceted rasch measurement and multi-text encoders as discriminators rather than generators. task deep learning: a hate speech application. arXiv arXiv preprint arXiv:2003.10555. preprint arXiv:2009.10277. ↗
  16. Line H Clemmensen and Rune D Kjærsgaard. 2022. find
  17. Jiyun Kim, Byounghan Lee, and Kyung-Ah Sohn. 2022. Data representativity for machine learning and ai Why is it hate speech? masked rationale prediction systems. arXiv preprint arXiv:2203.04706. for explainable hate speech detection. In Proceed-Stefano Cresci, Amaury Trujillo, and Tiziano Fagni. ings of the 29th International Conference on Com-2022. Personalized interventions for online modera-putational Linguistics, pages 6644–6655, Gyeongju, tion. In Proceedings of the 33rd ACM Conference on Republic of Korea. International Committee on Com-Hypertext and Social Media, pages 248–251. putational Linguistics. ↗
  18. Hannah Kirk, Wenjie Yin, Bertie Vidgen, and Paul Röttger. 2023. SemEval-2023 task 10: Explainable detection of online sexism. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2193–2210, Toronto, Canada. Association for Computational Linguistics.
  19. Pia Pachinger, Janis Goldzycher, Anna Planitzer, Wojciech Kusa, Allan Hanbury, and Julia Neidhardt. 2024. Austrotox: A dataset for target-based austrian german offensive language detection. In Findings of the Association for Computational Linguistics ACL 2024, pages 11990–12001. find
  20. Jan Kocoń, Alicja Figas, Marcin Gruza, Daria Puchalska, Tomasz Kajdanowicz, and Przemysław Kazienko. 2021. Offensive, aggressive, and hate speech analysis: From data-centric to human-centered approach. Information Processing & Management, 58(5):102643. find
  21. Pia Pachinger, Allan Hanbury, Julia Neidhardt, and Anna Planitzer. 2023. Toward disambiguating the definitions of abusive, offensive, toxic, and uncivil comments. In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pages 107–113, Dubrovnik, Croatia. Association for Computational Linguistics. find
  22. Klaus Krippendorff. 2004. Reliability in content analysis: Some common misconceptions and recommendations. Human communication research, 30(3):411– 433. ↗
  23. John Pavlopoulos, Jeffrey Sorensen, Léo Laugier, and Ion Androutsopoulos. 2021. SemEval-2021 task 5: Toxic spans detection. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 59–69, Online. Association for Computational Linguistics.
  24. Deepak Kumar, Patrick Gage Kelley, Sunny Consolvo, Joshua Mason, Elie Bursztein, Zakir Durumeric, Kurt Thomas, and Michael Bailey. 2021. Designing toxic content classification for a diversity of perspectives. In Seventeenth Symposium on Usable Privacy and Security (SOUPS 2021), pages 299–318. find
  25. Barbara Plank. 2022. The “problem” of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671–10682.
  26. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  27. Julian Risch, Robin Ruff, and Ralf Krestel. 2020. Offensive language detection explained. In Proceedings of the second workshop on trolling, aggression and cyberbullying, pages 137–143. find
  28. Qing Lyu, Marianna Apidianaki, and Chris Callison-Burch. 2024. Towards faithful model explanation in nlp: A survey. Computational Linguistics, pages 1–70.
  29. Julian Risch, Anke Stoll, Lena Wilms, and Michael Wiegand. 2021. Overview of the germeval 2021 shared task on the identification of toxic, engaging, and fact-claiming comments. In Proceedings of the GermEval 2021 Shared Task on the Identification of Toxic, Engaging, and Fact-Claiming Comments, pages 1–12. find
  30. Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. 2021. Hatexplain: A benchmark dataset for explainable hate speech detection. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 14867–14875. find
  31. Paul Röttger, Bertie Vidgen, Dirk Hovy, and Janet Pierrehumbert. 2022. Two contrasting data annotation paradigms for subjective NLP tasks. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 175–190, Seattle, United States. Association for Computational Linguistics. ↗
  32. Maria D Molina and S Shyam Sundar. 2022. When ai moderates online content: effects of human collaboration and interactive transparency on user trust. Journal of Computer-Mediated Communication, 27(4):zmac010. find
  33. Hiroki Nakayama. 2018. seqeval: A python framework for sequence labeling evaluation. Software available from https://github.com/chakki-works/seqeval. ↗
  34. Marta Sandri, Elisa Leonardelli, Sara Tonelli, and Elisabetta Jezek. 2023. Why don‘t you do it right? analysing annotators’ disagreement in subjective tasks. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2428–2441, Dubrovnik, Croatia. Association for Computational Linguistics. find
  35. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. ↗
  36. Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power implications of language. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5477–5490, Online. Association for Computational Linguistics.
  37. Pia Pachinger. 2024. Austrotox: A dataset for target-based austrian german offensive language detection. Comited to ACL. find
  38. Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5884–5906, Seattle, United States. Association for Computational Linguistics.
  39. Maximilian Wich, Hala Al Kuwatly, and Georg Groh. 2020. Investigating annotator bias with a graph-based approach. In Proceedings of the Fourth Workshop on Online Abuse and Harms, pages 191–199, Online. Association for Computational Linguistics. find
  40. Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex machina: Personal attacks seen at scale. In Proceedings of the 26th international conference on world wide web, pages 1391–1399. find
  41. Marcos Zampieri, Skye Morgan, Kai North, Tharindu Ranasinghe, Austin Simmmons, Paridhi Khandelwal, Sara Rosenthal, and Preslav Nakov. 2023. Target-based offensive language identification. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 762–770, Toronto, Canada. Association for Computational Linguistics. find
  42. Qiang Zhang, Jason Naradowsky, and Yusuke Miyao. 2022. Rethinking offensive text detection as a multi-hop reasoning problem. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3888–3905, Dublin, Ireland. Association for Computational Linguistics. find
  43. Xuhui Zhou, Hao Zhu, Akhila Yerukola, Thomas Davidson, Jena D Hwang, Swabha Swayamdipta, and Maarten Sap. 2023. Cobra frames: Contextual reasoning about effects and harms of offensive statements. In Findings of the Association for Computational Linguistics: ACL 2023, pages 6294–6315. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pachinger-2025-disaggregated-dataset-english-offensiveness-containing/