Elamur Research⌕
☰
home / papers / 2026 / Nuanced Toxicity Detection in Spanish: A New Corpus and Benchmark Study

Nuanced Toxicity Detection in Spanish: A New Corpus and Benchmark Study

Paper · 2026 · Alba María Mármol-Romero, Robiert Sepúlveda-Torres, Estela Saquete et al.

Summary

Spanish has few annotated resources for toxic language detection, especially for subtle forms such as irony, sarcasm, mockery or delegitimization. Existing Spanish datasets are either tied to one topic (DETOXIS, immigration) or to one source (NECOS-TOX, El Mundo). The authors extend NECOS-TOX into SocialTox, a corpus of 4,011 reader comments from 16 Spanish news outlets, labelled on a three-level scale: Non-Toxic, Slightly Toxic and Toxic. To keep annotation affordable, they used human-in-the-loop active learning: a model proposed labels and annotators corrected them. Guidelines were revised over several rounds until agreement reached κ = 0.74.

The corpus is benchmarked with classical ML, Spanish transformer encoders and instruction-tuned LLMs. The compact RoBERTa-base-bne encoder performs on par with far larger generative models such as LLaMA-3.1-8B, which suggests that in-domain adaptation matters more than model size. Error analysis shows that borderline categories, sarcasm and implicit insults remain hard. Models also use named entities as shortcuts for predicting toxicity, and anonymization reduces this. The openly released data and models give Spanish moderation research a nuanced, multi-source benchmark.

Key points

  • SocialTox combines the 1,419 original NECOS-TOX comments with 2,592 new ones, for 4,011 comments in total. It is released on Hugging Face as gplsi/SocialTOX.
  • Pilot annotation with four classes gave low agreement (κ < 0.30), mostly because annotators confused Toxic with Very Toxic. Very Toxic comments were also rare, since moderators remove them. Merging the scheme into three levels raised Fleiss' κ to 0.74.
  • The data was labelled by three annotators. Two were trained in linguistics, and the third came from sociology.
  • RoBERTa-base-bne (125M parameters) rivals LLaMA-3.1-8B. Performance drops in the multi-class setting, mainly around the Slightly Toxic class.
  • Calibration is weakest for the Toxic class, where the model is underconfident (ECE = 0.218, MCE = 0.461). Non-Toxic is the best calibrated class (ECE = 0.091). Anonymizing named entities reduces entity bias at a small cost in accuracy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AlicanteAlicante · Spain21 works · 1,098 citations🎓Universidad de JaénJaén · Spain8 works · 1,671 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksBias, fairness & safety · 517 works

Cite

Cite as Alba María Mármol-Romero, Robiert Sepúlveda-Torres, Estela Saquete, María-Teresa Martín-Valdivia and L. Alfonso Ureña. 2026. Nuanced Toxicity Detection in Spanish: A New Corpus and Benchmark Study. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.100

@inproceedings{marmolromero2026nuanced,
  title = {{Nuanced Toxicity Detection in Spanish: A New Corpus and Benchmark Study}},
  author = {Alba María Mármol-Romero, Robiert Sepúlveda-Torres, Estela Saquete, María-Teresa Martín-Valdivia and L. Alfonso Ureña},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.100},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.100}
}

References

28 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A next-generation hyperparameter optimization framework. KDD ’19, page 2623–2631, New York, NY, USA. Association for Computing Machinery.
  2. Moez Ali. 2020. PyCaret: An open source, low-code machine learning library in Python. PyCaret version 1.0.0. find
  3. Beatriz Botella-Gil, Robiert Sepúlveda-Torres, Alba Bonet-Jover, Patricio Martínez-Barco, and Estela Saquete. 2024. Semi-automatic dataset annotation applied to automatic violent message detection. IEEE Access. find
  4. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. CoRR, abs/1911.02116. find
  5. Marta R Costa-jussà, Mariano Coria Meglioli, Pierre Andrews, David Dale, Prangthip Hansanti, Elahe Kalbassi, Alex Mourachko, Christophe Ropers, and Carleigh Wood. 2024. Mutox: Universal multilingual audio-based toxicity dataset and zero-shot detector. arXiv preprint arXiv:2401.05060. find
  6. Asier Gutiérrez Fandiño, Jordi Armengol Estapé, Marc Pàmies, Joan Llop Palao, Joaquin Silveira Ocampo, Casimiro Pio Carrino, Carme Armentano Oller, Carlos Rodriguez Penagos, Aitor Gonzalez Agirre, and Marta Villegas. 2022. Maria: Spanish language models. Procesamiento del Lenguaje Natural, 68. find
  7. Sreyan Ghosh, Samden Lepcha, Sahni Sakshi, Rajiv Ratn Shah, and Srinivasan Umesh. 2021. Detoxy: A large-scale multimodal dataset for toxicity classification in spoken utterances. arXiv preprint arXiv:2110.07592. — [arXiv:2110.07592](https://arxiv.org/abs/2110.07592) find
  8. Ian Kivlichan, Jeffrey Sorensen, Julia Elliott, Lucy Vasserman, Martin Görner, and Phil Culliton. 2020. Jigsaw multilingual toxic comment classification. Kaggle. find
  9. Varada Kolhatkar, Hanhan Wu, Luca Cavasso, Emilie Francis, Kavan Shukla, and Maite Taboada. 2020. The sfu opinion and comments corpus: A corpus for the analysis of online news comments. Corpus pragmatics, 4:155–190. find
  10. J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159–174. find
  11. Pilar López-Úbeda, Flor Miriam Plaza-del Arco, Manuel-Carlos Díaz-Galiano, and M Teresa Martín-Valdivia. 2024. Toxicity in spanish news comments and its relationship with constructiveness. Procesamiento del lenguaje natural, 73:43–53. find
  12. R Monarch. 2021. Munro. active learning and annotation for human-centered ai. find
  13. Preslav Nakov, Vibha Nayak, Kyle Dent, Ameya Bhatawdekar, Sheikh Muhammad Sarwar, Momchil Hardalov, Yoan Dinkov, Dimitrina Zlatkova, Guillaume Bouchard, and Isabelle Augenstein. 2021. Detecting abusive language on online platforms: A critical analysis. arXiv preprint arXiv:2103.00153. — [arXiv:2103.00153](https://arxiv.org/abs/2103.00153) find
  14. John Pavlopoulos, Léo Laugier, Alexandros Xenos, Jeffrey Sorensen, and Ion Androutsopoulos. 2022. From the detection of toxic spans in online discussions to the analysis of toxic-to-civil transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022).… find
  15. John Pavlopoulos, Jeffrey Sorensen, Léo Laugier, and Ion Androutsopoulos. 2021. Semeval-2021 task 5: Toxic spans detection. In Proceedings of the 15th international workshop on semantic evaluation (SemEval-2021), pages 59–69.
  16. Juan Manuel Pérez, Damián Ariel Furman, Laura Alonso Alemany, and Franco M. Luque. 2022. RoBERTuito: a pre-trained language model for social media text in Spanish. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 7235–7243, Marseille, France. European Language Res… find
  17. Flor Miriam Plaza-del Arco, M Dolores Molina-González, L Alfonso Urena-López, and M Teresa Martín-Valdivia. 2021. Comparing pre-trained language models for spanish hate speech detection. Expert Systems with Applications, 166:114120. find
  18. María Estrella Vallecillo Rodríguez, Maria Victoria Cantero Romero, Isabel Cabrera De Castro, Arturo Montejo Ráez, and María Teresa Martín Valdivia. 2024. Conan-mt-sp: A spanish corpus for counternarrative using gpt models. In Proceedings of the 2024 Joint International Conference on Computational L… find
  19. Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Marcos Zampieri, and Preslav Nakov. 2020. Solid: A large-scale semi-supervised dataset for offensive language identification. arXiv preprint arXiv:2004.14454. find
  20. Amit Sheth, Valerie L Shalin, and Ugur Kursuncu. 2022. Defining and detecting toxicity on social media: context and knowledge are key. Neurocomputing, 490:312–318. find
  21. Mariona Taulé, Alejandro Ariza, Montserrat Nofre, Enrique Amigó, and Paolo Rosso. 2021. Overview of detoxis at iberlef 2021: detection of toxicity in comments in spanish. Procesamiento del lenguaje natural, 67:209–221. find
  22. Serra Sinem Tekiroglu, Helena Bonaldi, Margherita Fanton, and Marco Guerini. 2022. Using pre-trained language models for producing counter narratives against hate speech: a comparative study. arXiv preprint arXiv:2204.01440. find
  23. Guillermo Villate-Castillo, Javier Del Ser, and Borja Sanz Urquijo. 2024. A systematic review of toxicity in large language models: Definitions, datasets, detectors, detoxification methods and challenges. find
  24. Sara Zaheri, Jeff Leath, and David Stroud. 2020. Toxic comment classification. SMU Data Science Review, 3(1):13. find
  25. Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Çağrı Çöltekin. 2020. Semeval-2020 task 12: Multilingual offensive language identification in social media (offenseval 2020). arXiv preprint arXiv:2006.07235. find
  26. Zhixue Zhao, Ziqi Zhang, and Frank Hopfgartner. 2021. A comparative study of using pre-trained language models for toxic comment classification. In Companion Proceedings of the Web Conference 2021, pages 500–507. find
  27. A Newspaper Okdiario Marca El Mundo El País NTV España The Objective Alerta Digital El Debate Vozpópuli El Español ABC El Confidencial El Diario Moncloa Periodista Digital 20 Minutos • Normalization: Min-Max scaling applied to feature values. • Cross-validation: Stratified k-fold splits to preserve… find
  28. Evaluation metric: Macro-F1, chosen for its sensitivity to imbalanced and multi-class scenarios. • Reproducibility: Random seed fixed to 42. B.3 Optimal Model and Hyperparameters Across all tested algorithms, logistic regression consistently yielded the highest performance. The final configuration w… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/romero-2026-nuanced-toxicity-detection-spanish-new/