Elamur Research⌕
☰
home / papers / 2026 / Can LLMs Evaluate What They Cannot Annotate? Revisiting LLM Reliability in Hate Speech Detection

Can LLMs Evaluate What They Cannot Annotate? Revisiting LLM Reliability in Hate Speech Detection

Summary

Hate speech labelling is subjective, and standard agreement measures such as Cohen's κ treat every disagreement as a mistake. Prior studies relying on these metrics concluded that LLMs cannot stand in for human annotators. The authors ask whether a subjectivity-aware measure, cross-Replication Reliability (xRR), changes that conclusion. They also ask whether LLMs could serve a different purpose: ranking hate speech classifiers.

They used English and Spanish posts with per-annotator labels, drawn from the MetaHate and MetaHateES meta-collections. Three LLMs (Nemo, Llama 3.1, DeepSeek) acted as a three-member "annotator group" and were compared with three human raters.

Under xRR, LLM labels resemble human tendencies somewhat more than κ suggests, but they still fall short of human reliability. However, when LLM labels replace human labels as the test reference, the resulting ordering of classifiers mostly matches the human-based ordering.

This matters for low-resource settings. Carefully chosen LLMs could support cheap comparative audits of models when human labels are scarce. The ranking agreement must first be checked on the target dataset.

Key points

  • Candidate datasets: HateXplain and Measuring Hate Speech (MHS), taken from MetaHate (36 English sources), and DETESTS and EXIST, taken from MetaHateES (10 Spanish sources). The task is binary: hate vs. non-hate.
  • Agreement with humans: Under κ, the LLMs reach only slight-to-fair agreement. Nemo comes closest to fair agreement, whereas the agreement of Llama 3.1 and DeepSeek does not rise above the slight level.
  • Majority voting hurts: A majority vote across the three LLMs is dominated by the two weaker models, which cancels out Nemo's advantage.
  • Ranking experiment: Classifier rankings based on LLM labels correlate strongly with human-based rankings (Kendall's τ roughly 0.84–0.96).
    • Nemo reaches τ ≥ 0.95, with a mean F1 gap as small as 0.041 on DETESTS.
    • DeepSeek produces a reversed ranking on MHS (negative τ).
  • Absolute scores and limits: Even with good rankings, absolute F1 estimates stay biased, especially for Llama 3.1 and DeepSeek. The study also has several limitations:
    • only binary labels and only English and Spanish;
    • simple prompts;
    • only items with exactly three human annotators.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/22n5hekovvrz

@inproceedings{piot2026can,
  title = {{Can LLMs Evaluate What They Cannot Annotate? Revisiting LLM Reliability in Hate Speech Detection}},
  author = {Piot, Paloma and Otero, David and Martin-Rodilla, Patricia and Parapar, Javier},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/22n5hekovvrz},
  url = {https://doi.org/10.63317/22n5hekovvrz}
}

References

61 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anti-Defamation League. 2024. Online hate and harassment: The american experience 2024. Accessed: 03/01/2024. find
  2. Lora Aroyo, Lucas Dixon, Nithum Thain, Olivia Redfield, and Rachel Rosen. 2019. Crowdsourcing subjective tasks: The case study of understanding toxicity in online discussions. In Companion Proceedings of The 2019 World Wide Web Conference, WWW ’19, page 1100–1105, New York, NY, USA. Association for… find
  3. Dennis Assenmacher, Paloma Piot, Katarina Laken, David Jurgens, and Claudia Wagner. 2025. Beyond the explicit: A bilingual dataset for dehumanization detection in social media. find
  4. Baran Barbarestani, Isa Maks, and Piek T.J.M. Vossen. 2024. Content moderation in online platforms: A study of annotation methods for inappropriate language. In Proceedings of the Fourth Workshop on Threat, Aggression & Cyberbullying @ LREC-COLING-2024, pages 96–104, Torino, Italia. ELRA and ICCL. find
  5. Valerio Basile. 2020. It’s the end of the gold standard as we know it: Leveraging non-aggregated data for better evaluation and explanation of subjective tasks. In AIxIA 2020 – Advances in Artificial Intelligence: XIXth International Conference of the Italian Association for Artificial Intelligence,… find
  6. Davide Bassi, Dimitar Iliyanov Dimitrov, Bernardo D’Auria, Firoj Alam, Maram Hasanain, Christian Moro, Luisa Orrù, Gian Piero Turchi, Preslav Nakov, and Giovanni Da San Martino. 2025. Annotating the annotators: Analysis, insights and modelling from an annotation campaign on persuasion techniques det… find
  7. Joachim Baumann, Paul Röttger, Aleksandra Urman, Albert Wendsjö, Flor Miriam Plaza del Arco, Johannes B. Gruber, and Dirk Hovy. 2025. Large language model hacking: Quantifying the hidden risks of using llms for text annotation. find
  8. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37–46. find
  9. Mithun Das, Rohit Raj, Punyajoy Saha, Binny Mathew, Manish Gupta, and Animesh Mukherjee. 2023. Hatemm: A multi-modal dataset for hate video classification. Proceedings of the ICWSM 2023, 17:1014–1023. find
  10. Aida Mostafazadeh Davani, Mohammad Atari, Brendan Kennedy, and Morteza Dehghani. 2023. Hate speech classifiers learn normative social stereotypes. Transactions of the Association for Computational Linguistics, 11:300–319. find
  11. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. Proceedings of the ICWSM 2017, 11(1):512–515. find our card
  12. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. find our card
  13. Somaiyeh Dehghan, Mehmet Umut Sen, and Berrin Yanikoglu. 2025. Dealing with annotator disagreement in hate speech classification. find
  14. Senjuti Dutta, Sid Mittal, Sherol Chen, Deepak Ramachandran, Ravi Rajakumar, Ian Kivlichan, Sunny Mak, Alena Butryna, and Praveen Paritosh. 2023. Modeling subjectivity (by mimicking annotator annotation) in toxic comment identification across diverse communities. find
  15. Mai ElSherief, Vivek Kulkarni, Dana Nguyen, William Yang Wang, and Elizabeth Belding. 2018a. Hate lingo: A target-based linguistic analysis of hate speech in social media. Proceedings of the ICWSM 2018, 12(1). find
  16. Mai ElSherief, Shirin Nilizadeh, Dana Nguyen, Giovanni Vigna, and Elizabeth Belding. 2018b. Peer to peer hate: Hate speech instigators and their targets. Proceedings of the ICWSM 2018, 12(1). find
  17. Paula Fortuna and Sérgio Nunes. 2018. A survey on automatic detection of hate speech in text. ACM Comput. Surv., 51(4). find
  18. Antigoni Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, ..., and Nicolas Kourtellis. 2018. Large scale crowdsourcing and characterization of twitter abusive behavior. Proceedings of the ICWSM 2018, 12(1). find
  19. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30):e2305016120. find our card
  20. Tommaso Giorgi, Lorenzo Cima, Tiziano Fagni, Marco Avvenuti, and Stefano Cresci. 2025. Human and llm biases in hate speech annotations: A socio-demographic analysis of annotators and targets. Proceedings of the International AAAI Conference on Web and Social Media, 19(1):653–670. find
  21. Goran Glavaš, Vanja Mladen Karan, and Ivan Vulić. 2020. XHate-999: Analyzing and detecting abusive language across domains and languages. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6350–6365, Barcelona, Spain (Online). International Committee on Computati… find
  22. Sandra González-Bailón and Yphtach Lelkes. 2022. Do social media undermine social cohesion? a critical review. Social Issues and Policy Review, 17(1):155–180. find
  23. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, ..., and Zhiyu Ma. 2024. The llama 3 herd of models. find
  24. Imane Guellil, Yousra Houichi, Sara Chennoufi, Mohamed Boubred, Anfal Yousra Boucetta, and Faical Azouaou. 2024. The annotators agree to not agree on the fine-grained annotation of hate-speech against women in Algerian dialect comments. In Proceedings of the Fifth Workshop on Resources for African I… find
  25. Andrew F. Hayes and Klaus Krippendorff. 2007. Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1):77–89. find
  26. Danula Hettiachchi, Indigo Holcombe-James, Stephanie Livingstone, Anjalee de Silva, Matthew Lease, Flora D. Salim, and Mark Sanderson. 2023. How crowd worker factors influence subjective annotations: A study of tagging misogynistic hate speech in tweets. Proceedings of the AAAI Conference on Human C… find
  27. Kamil Kanclerz, Marcin Gruza, Konrad Karanowski, Julita Bielaniewicz, Piotr Milkowski, Jan Kocon, and Przemyslaw Kazienko. 2022. What if ground truth is subjective? personalized deep neural hate speech detection. In Proceedings of the 1st Workshop on Perspectivist Approaches to NLP @LREC2022, pages… find
  28. Julia Kansok-Dusche, Cindy Ballaschk, Norman Krause, Anke Zeißig, Lisanne Seemann-Herz, Sebastian Wachs, and Ludwig Bilz. 2022. A systematic review on hate speech among children and adolescents: Definitions, prevalence, and overlap with related phenomena. Trauma, Violence, & Abuse, 24(4):2598–2615. find
  29. Maurice G. Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1/2):81–93. find
  30. Chris J. Kennedy, Geoff Bacon, Alexander Sahn, and Claudia von Vacano. 2020. Constructing interval variables via faceted rasch measurement and multitask deep learning: a hate speech application. find
  31. Petra Kralj Novak, Teresa Scantamburlo, Andraž Pelicon, Matteo Cinelli, Igor Mozetič, and Fabiana Zollo. 2022. Handling disagreement in hate speech modelling. In Information Processing and Management of Uncertainty in Knowledge-Based Systems, pages 681–695, Cham. Springer International Publishing. find
  32. Elisa Leonardelli, Stefano Menini, Alessio Palmero Aprosio, Marco Guerini, and Sara Tonelli. 2021. Agreeing to disagree: Annotating offensive language datasets with annotators’ disagreement. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10528–10539,… find
  33. Nikola Ljubešić, Igor Mozetič, and Petra Kralj Novak. 2022. Quantifying the impact of context on the quality of manual hate speech annotation. Natural Language Engineering, 29(6):1481–1494. find
  34. Marian Marchal, Merel Scholman, Frances Yung, and Vera Demberg. 2022. Establishing annotation quality in multi-label annotations. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3659–3668, Gyeongju, Republic of Korea. International Committee on Computational L… find
  35. Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. 2020. Hatexplain: A benchmark dataset for explainable hate speech detection. In Proceedings of the AAAI 2020. find
  36. Daniel Matter, Miriam Schirmer, Nir Grinberg, and Jürgen Pfeffer. 2024. Investigating the increase of violent speech in incel communities with human-guided gpt-4 prompt iteration. Frontiers in Social Psychology, 2. find
  37. Mistral AI team. 2025. Mistral NeMo \| Mistral AI. https://mistral.ai/news/mistral-nemo. Accessed: 13/08/2025. ↗
  38. Sandip Modha, Thomas Mandl, Prasenjit Majumder, and Daksh Patel. 2019. Overview of the hasoc track at fire 2019: Hate speech and offensive content identification in indo-european languages. Working Notes of FIRE 2019, 2517. find
  39. Jay Mohta, Kenan Ak, Yan Xu, and Mingwei Shen. 2023. Are large language models good annotators? In Proceedings on "I Can’t Believe It’s Not Better: Failure Modes in the Age of Foundation Models" at NeurIPS 2023 Workshops, volume 239 of Proceedings of Machine Learning Research, pages 38–48. PMLR. find
  40. Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. 2022. Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Transactions of the Association for Computational Linguistics, 10:92–110. find our card
  41. Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677–694. find our card
  42. Paloma Piot, José Ramom Pichel Campos, and Javier Parapar. 2025. Bridging gaps in hate speech detection: Meta-collections and benchmarks for low-resource iberian languages. find
  43. Paloma Piot, Patricia Martín-Rodilla, and Javier Parapar. 2024. Metahate: A dataset for unifying efforts on hate speech detection. Proceedings of the International AAAI Conference on Web and Social Media, 18(1):2025–2039. find
  44. Paloma Piot and Javier Parapar. 2025. Towards Efficient and Explainable Hate Speech Detection via Model Distillation, page 376–392. Springer Nature Switzerland. find
  45. Barbara Plank. 2022. The “problem” of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671–10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find our card
  46. Flor Miriam Plaza-del arco, Debora Nozza, and Dirk Hovy. 2023. Respectful or toxic? using zero-shot learning with language models to detect hate speech. In The 7th Workshop on Online Abuse and Harms (WOAH), pages 60–68, Toronto, Canada. Association for Computational Linguistics. find
  47. Fabio Poletto, Valerio Basile, Manuela Sanguinetti, Cristina Bosco, and Viviana Patti. 2020. Resources and benchmark corpora for hate speech detection: a systematic review. Language Resources and Evaluation, 55(2):477–523. find
  48. Vinodkumar Prabhakaran, Christopher Homan, Lora Aroyo, Aida Mostafazadeh Davani, Alicia Parrish, Alex Taylor, Mark Diaz, Ding Wang, and Gregory Serapio-García. 2024. GRASP: A disagreement analysis framework to assess group associations in perspectives. In Proceedings of the 2024 Conference of the No… find
  49. Francisco Rodríguez-Sánchez, Jorge Carrillo-de Albornoz, Laura Plaza, Adrián Mendieta-Aragón, Guillermo Marco-Remón, Maryna Makeienko, María Plaza, Julio Gonzalo, Damiano Spina, and Paolo Rosso. 2022. Overview of exist 2022: sexism identification in social networks. Procesamiento del Lenguaje Natura… find
  50. Paul Röttger, Bertie Vidgen, Dirk Hovy, and Janet Pierrehumbert. 2022. Two contrasting data annotation paradigms for subjective NLP tasks. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 175–190,… find our card
  51. Sarthak Roy, Ashish Harshvardhan, Animesh Mukherjee, and Punyajoy Saha. 2023. Probing LLMs for hate speech detection: strengths and vulnerabilities. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6116–6128, Singapore. Association for Computational Linguistics. find
  52. Pratik Sachdeva, Renata Barreto, Geoff Bacon, Alexander Sahn, Claudia von Vacano, and Chris Kennedy. 2022. The measuring hate speech corpus: Leveraging rasch measurement theory for data perspectivism. In Proceedings of the 1st Workshop on Perspectivist Approaches to NLP @LREC2022, pages 83–94, Marse… find
  53. Wolfgang S. Schmeisser-Nieto, Pol Pasterlls, Simona Frenda, Alejandro Ariza-Casabona, Mireia Farrús, Paolo Rosso, and Mariona Tulé. 2024. Overview of detests-dis at iberlef 2024: Detection and classification of racial stereotypes in spanish - learning with disagreement. Procesamiento del Lenguaje Na… find
  54. Patrick E. Shrout and Joseph L. Fleiss. 1979. Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2):420–428. find
  55. Cagri Toraman, Furkan Şahinuç, and Eyup Yilmaz. 2022. Large-scale hate speech detection with cross-domain transfer. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2215–2225, Marseille, France. European Language Resources Association. find
  56. Yu-Min Tseng, Wei-Lin Chen, Chung-Chi Chen, and Hsin-Hsi Chen. 2025. Evaluating large language models as expert annotators. In Second Conference on Language Modeling. find
  57. Petter Törnberg. 2024. Large language models outperform expert coders and supervised classifiers at annotating political social media messages. Social Science Computer Review, 0(0):08944393241286471. find
  58. Alexandra N. Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2022. Learning from disagreement: A survey. J. Artif. Int. Res., 72:1385–1470. find
  59. Bertie Vidgen and Leon Derczynski. 2020. Directions in abusive language training data, a systematic review: Garbage in, garbage out. PLOS ONE, 15(12):e0243300. find
  60. Yau-Shian Wang and Yingshan Chang. 2022. Toxicity detection with generative prompt-based inference. find
  61. Ka Wong, Praveen Paritosh, and Lora Aroyo. 2021. Cross-replication reliability - an empirical approach to interpreting inter-rater reliability. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language P… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/piot-2026-can-llms-evaluate-what-they/