Elamur Research⌕
☰
home / papers / 2026 / Nürnberg NLP @ GermEval Shared Task 2026: Harmful Content Detection in German Social Media through Error-Independent LLM Voters

Nürnberg NLP @ GermEval Shared Task 2026: Harmful Content Detection in German Social Media through Error-Independent LLM Voters

Paper · 2026 · Philipp Steigerwald, Eric Rudolph and Jens Albrecht

Summary

The paper describes the winning entry to the GermEval 2026 shared task, which targets harmful content in German tweets that were collected from a right-wing extremist network and stripped of identifying information. There are four subtasks: calls to action (C2A), attacks on the democratic basic order (DBO), violence-related content (VIO) and criminal defamation (DEF). Each subtask is dominated by a harmless class, and macro-F1 gives the rare harmful classes equal weight, even though they look superficially like the majority.

The authors argue that diverse voters whose mistakes do not coincide matter more than one stronger model. For each subtask they build nine voters by plain majority vote. The voters vary along three axes:

  • the LLM (five candidates),
  • the training method (generative SFT or a classification head with focal loss),
  • the class scope (generalist, specialist, or a specialist that ignores the majority class).

Training data is enlarged with GPT-generated minority-class tweets. Ensemble compositions are chosen mostly by five-fold cross-validation. The system ranked first on all four subtasks. This suggests that structured diversity is an effective response to severe class imbalance.

Key points

  • The hidden-test macro-F1 scores are 89.56 (C2A), 71.63 (DBO), 54.84 (VIO) and 83.02 (DEF), first place in every subtask.

  • The training data has 23,829 tweets, with majority classes at 87–97%. Some DBO and VIO classes fall below 1%; for example, VIO glorification has 27 examples.

  • The authors produced 2,634 synthetic minority tweets with GPT-5.4. The per-class budget was min(1000 − n_c, 3·n_c), and C2A received none. These tweets appear only in training folds and are released under CC BY-NC 4.0.

  • The model pool has five LLMs, all 4-bit NF4 QLoRA:

    • Ministral-8B-Instruct
    • Phi-4-14B
    • LLäMmlein-7B, in base and chat variants
    • Mistral-Medium-3.5 (128B)

    The 128B model also served as the OPRO prompt optimiser. Fine-tuned voters outperformed every optimised prompt.

  • Agreement among voters, measured as Krippendorff's α over the nine votes, runs from .825 on C2A down to .142 on DBO. The third branch acts on 3.5–7.1% of tweets. On DBO it reverses 71% of the contested decisions, mostly moving tweets out of "nothing". Removing it costs 2.3–8.3 cross-validation F1 points.

Interesting ideas

  • I1. A "positive-class specialist" is trained only on the non-majority labels, so it learns the fine distinctions among rare classes and works as the ensemble's recall lever on undecided items.
  • I2. The models from each cross-validation fold serve two purposes: they provide the performance estimate and they become the ensemble's voters. Diversity and evaluation therefore come from a single training budget.
  • I3. The authors check whether diversity is real by measuring inter-voter agreement (Krippendorff's α) and by locating the small "contested zone" where a branch changes the outcome. They do not assume diversity simply because the models differ.
  • I4. Synthetic augmentation is capped at three times a class's real count and is kept out of validation and test, so real data always outweighs generated data and evaluation stays clean.
  • I5. The authors note that removing a branch also shrinks the vote, so the measured loss is only an upper bound on that branch's distinct contribution.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): For irony or indirect speech-act labelling, where "literal" dominates, a specialist trained only on non-literal cases could reconsider borderline clauses that the main classifier leaves at the default.
  • I2 (learn): The fold-trained copies of the perceptron parser or gparse could act as extra witness parsers in the silver-data vote at no added training cost.
  • I3 (learn): Computing α between witness parsers per dependency relation would show where their errors are actually independent. Arcs in the contested zone could then be sent to the LLM teacher tier.
  • I4 (tree): The same cap could limit synthetic sentences written to boost rare UD relations in treebank training, with scoring done only on gold treebank data.
  • I5 (learn): A proper negative control would swap a witness parser for an extra copy of an existing one, which separates the effect of a distinct view from the effect of simply having more votes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Georg Simon Ohm University of Applied Sciences NurembergNuremberg · Germany3 works · 2 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Bias, fairness & safety · 517 works

Cite

Cite as Philipp Steigerwald, Eric Rudolph and Jens Albrecht. 2026. Nürnberg NLP @ GermEval Shared Task 2026: Harmful Content Detection in German Social Media through Error-Independent LLM Voters. arXiv:2608.22246.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2608.22246 · arXiv 2608.22246

@misc{steigerwald2026nurnberg,
  title = {{Nürnberg NLP @ GermEval Shared Task 2026: Harmful Content Detection in German Social Media through Error-Independent LLM Voters}},
  author = {Philipp Steigerwald, Eric Rudolph and Jens Albrecht},
  year = {2026},
  doi = {10.48550/arxiv.2608.22246},
  eprint = {2608.22246},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2608.22246}
}

References

29 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, Harkirat Behl, and 1 others. 2024. Phi-4 technical report. Preprint, arXiv:2412.08905. ↗
  2. Samuel Bullard. 2025. abullardur@GermEval shared task 2025: Fine-tuning ModernGBERT on highly imbalanced German social media for harmful content. In Proceedings of the 21st Conference on Natural Language Processing (KONVENS), Volume 2: Workshops, pages 320–326. find
  3. Nitesh V. Chawla, Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. 2002. SMOTE: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16:321–357. ↗
  4. Christoph Demus, Jonas Pitz, Mina Schütz, Nadine Probol, Melanie Siegel, and Dirk Labudde. 2022. DeTox: A comprehensive dataset for german offensive language and conversation analysis. In Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH), pages 143–153. Association for Computational Linguistics. find
  5. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient finetuning of quantized LLMs. In Advances in Neural Information Processing Systems (NeurIPS).
  6. Thomas G. Dietterich. 2000. Ensemble methods in machine learning. In Multiple Classifier Systems (MCS 2000), volume 1857 of Lecture Notes in Computer Science, pages 1–15. Springer. find
  7. Jenny Felser, Thomas Holzhausen, Tony Katzwinkel, Frederike Zufall, Florian Ludwig, Martin Hofer, Melanie Siegel, and Michael Spranger. 2026. Overview of the GermEval 2026 shared task on harmful content detection. In Proceedings of the 22nd Conference on Natural Language Processing (KON-VENS 2026): Workshops, Hamburg, Germany. In press. find
  8. Jenny Felser, Michael Spranger, and Melanie Siegel. 2025. Overview of the GermEval 2025 shared task on harmful content detection. In Proceedings of the 21st Conference on Natural Language Processing (KONVENS), Volume 2: Workshops, pages 306–319. find
  9. Paula Fortuna and Sérgio Nunes. 2018. A survey on automatic detection of hate speech in text. ACM Computing Surveys, 51(4):1–30. ↗
  10. Sophie Henning, William Beluch, Alexander Fraser, and Annemarie Friedrich. 2023. A survey of methods for addressing class imbalance in deep-learning based natural language processing. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 523–540. Association for Computational Linguistics. find
  11. Hanna Köpcke. 2025. nymera@GermEval shared task 2025: One ensemble, many harms: A unified transformer approach to harmful content detection in German social media. In Proceedings of the 21st Conference on Natural Language Processing (KONVENS), Volume 2: Workshops, pages 357–365. find
  12. Anders Krogh and Jesper Vedelsby. 1995. Neural network ensembles, cross validation, and active learning. In Advances in Neural Information Processing Systems 7, pages 231–238. MIT Press. find
  13. Vincent Kums, Florian Meyer, Luisa Pivit, Uliana Vedenina, Jonas Wortmann, Melanie Siegel, and Dirk Labudde. 2025. A novel dataset for classifying german hate speech comments with criminal relevance. In Proceedings of the 9th Workshop on Online Abuse and Harms (WOAH), pages 41–52. Association for Computational Linguistics. find
  14. Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2980– 2988. ↗
  15. Florian Ludwig and Stefan Altmann. 2025. Munich_Z@GermEval shared task 2025: When prompting is not enough: The limits of large language models in GermEval’s 2025 harmful content detection task. In Proceedings of the 21st Conference on Natural Language Processing (KONVENS), Volume 2: Workshops, pages 337–349. find
  16. Thomas Mandl, Sandip Modha, Prasenjit Majumder, Daksh Patel, Mohana Dave, Chintak Mandlia, and Aditya Patel. 2019. Overview of the HASOC track at FIRE 2019: Hate speech and offensive content identification in Indo-European languages. In Proceedings of the 11th Forum for Information Retrieval Evaluation (FIRE ’19), pages 14–17. ACM. find
  17. Mistral AI. 2024. Un ministral, des ministraux. https://mistral.ai/news/ministraux. Ministral-8B-Instruct-2410. ↗
  18. Mistral AI. 2025. Mistral medium 3. https://mistral.ai/news/mistral-medium-3. Mistral-Medium-3.5, used 4-bit as Mis128B. ↗
  19. Christoph Papadatos, Claudia Grote, and Florian Ludwig. 2025. TheDBOs@GermEval shared task 2025: Supporting BERT model training with LLM-based synthetic tweets for harmful content detection. In Proceedings of the 21st Conference on Natural Language Processing (KONVENS), Volume 2: Workshops, pages 366–374. find
  20. Jan Pfister, Julia Wunderle, and Andreas Hotho. 2025. LLäMmlein: Transparent, compact and competitive german-only language models from scratch. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics.
  21. Julian Risch and Ralf Krestel. 2020. Bagging BERT models for robust aggression identification. In Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying (TRAC 2020), pages 55–61. European Language Resources Association (ELRA). find
  22. Anastasia Schwichtenberg. 2025. tweetbusters@GermEval shared task 2025: Comprehensive feature engineering for call to action detection. In Proceedings of the 21st Conference on Natural Language Processing (KONVENS), Volume 2: Workshops, pages 375–384. find
  23. Philipp Steigerwald, Jennifer Burghardt, Eric Rudolph, and Jens Albrecht. 2026a. AI systems in text-based online counselling: Ethical considerations across three implementation approaches. FAIEMA 2025; Springer chapter, in production. ↗
  24. Philipp Steigerwald, Eric Rudolph, and Jens Albrecht. 2026b. Nürnberg NLP at PsyDefDetect: Multi-axis voter ensembles for psychological defence mechanism classification. In Proceedings of the BioNLP 2026 (Shared Tasks), pages 59–65, San Diego, California, USA. Association for Computational Linguistics. ↗
  25. Michael Wiegand, Melanie Siegel, and Josef Ruppenhofer. 2018. Overview of the GermEval 2018 shared task on the identification of offensive language. In Proceedings of the GermEval 2018 Workshop, 14th Conference on Natural Language Processing (KON-VENS), Vienna, Austria. find
  26. Julia Wunderle, Jan Pfister, and Andreas Hotho. 2025. Die SuperGLEBer at GermEval 2025 shared tasks: Growing pains - when more isn’t always better. In Proceedings of the 21st Conference on Natural Language Processing (KONVENS), Volume 2: Workshops, pages 479–493. find
  27. Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. 2024. Large language models as optimizers. In The Twelfth International Conference on Learning Representations (ICLR). ArXiv:2309.03409. ↗
  28. Steven Zimmerman, Udo Kruschwitz, and Chris Fox. 2018. Improving hate speech detection with deep learning ensembles. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Association (ELRA). find
  29. Frederike Zufall, Tobias Horsmann, and Torsten Zesch. 2019. From legal to technical concept: Towards an automated classification of german political Twitter postings as criminal offenses. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 1337–1347. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/steigerwald-2026-n-urnberg-nlp-germeval-shared/