Elamur Research⌕
☰
home / papers / 2026 / RBG-AI at FadeIT: Prompted LLMs with Label Abstraction for Logical Fallacy Detection

RBG-AI at FadeIT: Prompted LLMs with Label Abstraction for Logical Fallacy Detection

Paper · 2026 · Meenakshi, Jairam R, Reshma U et al.

Summary

The paper presents the RBG-AI team's system for the FadeIT shared task at EVALITA 2026. The task asks systems to identify logical fallacies in Italian social media posts whose topics include public health, climate change and migration, with the FAINA dataset as the data source. This is hard for three reasons: the taxonomy is fine-grained, the label distribution is long-tailed, and each post keeps the annotations of two separate annotators, who often disagree.

The authors use no fine-tuning. Instead, they prompt instruction-tuned LLMs with a few examples, chosen to cover rare fallacies, multi-label cases and annotator variation. One shared prompt structure serves both subtasks: multi-label classification of sentences (Subtask A) and BIO span tagging of tokens (Subtask B).

Their main idea is label abstraction. Semantically related fallacies are grouped into subsets. The model is queried once per subset, and the predictions from all queries are merged.

Meta-Llama-3.1-8B-Instruct was the strongest model tested. Grouping labels clearly beat a flat label set on both subtasks. This suggests that careful prompt design and a structured label space can be a practical alternative to supervised fine-tuning when data is scarce and labels are ambiguous.

Key points

  • Data: FAINA covers 20 fallacy categories, allows overlapping spans and preserves the annotations of both annotators. The official test set is 20% of FAINA.

  • Prompting setup: The authors tried 3-, 5- and 10-shot prompting. For Subtask B, the LLM outputs are converted to BIO tags in CoNLL format using span alignment and fuzzy matching.

  • Official results (all runs: Meta-Llama-3.1-8B-Instruct, 3-shot):

    RunLabel setupSubtask A (avg. micro F1)Subtask B (soft micro F1)
    IFlat labels38.5716.17
    IIThree-stage grouping42.0724.31
    IIIFour-stage grouping40.0027.71

  • Granularity trade-off: Moderate grouping worked best for sentence-level classification, while slightly finer grouping helped span detection. Groups that were too fine confused the model, and groups that were too coarse increased false positives.

  • Code: Prompts and code are released at github.com/rbg-research/FadeIT-EVALITA-2026.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

Cite

Cite as Meenakshi, Jairam R, Reshma U, Barathi Ganesh HB and Michal Ptaszynski. 2026. RBG-AI at FadeIT: Prompted LLMs with Label Abstraction for Logical Fallacy Detection. Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{meenakshi2026rbg,
  title = {{RBG-AI at FadeIT: Prompted LLMs with Label Abstraction for Logical Fallacy Detection}},
  author = {Meenakshi, Jairam R, Reshma U, Barathi Ganesh HB and Michal Ptaszynski},
  booktitle = {Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026)},
  year = {2026},
  url = {https://aclanthology.org/2026.evalita-1.66/}
}

References

28 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. C. L. Hamblin, Fallacies, Advanced Reasoning Forum, 2022. find
  2. C. W. Tindale, Fallacies and argument appraisal, Cambridge University Press, 2007. find
  3. O. Balalau, R. Horincar, From the stage to the audience: Propaganda on reddit, in: EACL 2021-16th Conference of the European Chapter of the Association for Computational Linguistics, 2021. find
  4. V. Danciu, et al., Manipulative marketing: persuasion and manipulation of the consumer through advertising, Theoretical and Applied Economics 21 (2014) 19–34. find
  5. F. Macagno, Argumentation profiles and the manipulation of common ground. the arguments of populist leaders on twitter, Journal of Pragmatics 191 (2022) 67–82. find
  6. F. Zappettini, The brexit referendum: How trade and immigration in the discourses of the official campaigns have legitimised a toxic (inter) national logic, in: " Brexit" as a Social and Political Crisis, Routledge, 2021, pp. 23–39. find
  7. S. A. Elsayed, O. Abu-Hammad, A. B. Alolayan, Y. S. Eldeen, N. Dar-Odeh, Fallacies and facts around covid-19: the multifaceted infection, Journal of Craniofacial Surgery 31 (2020) e643–e644. find
  8. G. Da San Martino, S. Yu, A. Barrón-Cedeno, R. Petrov, P. Nakov, Fine-grained analysis of propaganda in news articles, in: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP),…
  9. U. Ecker, J. Roozenbeek, S. Van Der Linden, L. Q. Tay, J. Cook, N. Oreskes, S. Lewandowsky, Misinformation poses a bigger threat to democracy than you might think, Nature 630 (2024) 29–32. find
  10. H. Hansen, Fallacies. en zalta, 2020. find
  11. Z. Jin, A. Lalwani, T. Vaidhya, X. Shen, Y. Ding, Z. Lyu, M. Sachan, R. Mihalcea, B. Schoelkopf, Logical fallacy detection, arXiv preprint arXiv:2202.13758 (2022). — [arXiv:2202.13758](https://arxiv.org/abs/2202.13758) find
  12. B. Plank, The’problem’of human label variation: On ground truth in data, modeling and evaluation, arXiv preprint arXiv:2211.02570 (2022). — [arXiv:2211.02570](https://arxiv.org/abs/2211.02570)
  13. A. Ramponi, S. Tonelli, FadeIT at EVALITA 2026: Overview of the fallacy detection in italian social media texts task, in: Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026), CEUR.org, Bari, Italy, 2026.
  14. A. Ramponi, A. Daffara, S. Tonelli, Fine-grained fallacy detection with human label variation, arXiv preprint arXiv:2502.13853 (2025). find
  15. J. Chen, Z. Liu, X. Huang, C. Wu, Q. Liu, G. Jiang, Y. Pu, Y. Lei, X. Chen, X. Wang, et al., When large language models meet personalization: Perspectives of challenges and opportunities, World Wide Web 27 (2024) 42. find
  16. M. U. Hadi, R. Qureshi, A. Shah, M. Irfan, A. Zafar, M. B. Shaikh, N. Akhtar, J. Wu, S. Mirjalili, et al., Large language models: a comprehensive survey of its applications, challenges, limitations, and future prospects, Authorea preprints 1 (2023) 1–26. find
  17. M. Zhang, J. Li, A commentary of gpt-3 in mit technology review 2021, Fundamental Research 1 (2021) 831–833. find
  18. S. Minaee, T. Mikolov, N. Nikzad, M. Chenaghlu, R. Socher, X. Amatriain, J. Gao, Large language models: A survey, arXiv preprint arXiv:2402.06196 (2024).
  19. J. Yang, H. Jin, R. Tang, X. Han, Q. Feng, H. Jiang, S. Zhong, B. Yin, X. Hu, Harnessing the power of llms in practice: A survey on chatgpt and beyond, ACM Transactions on Knowledge Discovery from Data 18 (2024) 1–32.
  20. T. Gou, B. Zhang, Z. Sun, J. Wang, F. Liu, Y. Wang, J. Wang, Rationality of thought improves reasoning in large language models, in: International Conference on Knowledge Science, Engineering and Management, Springer, 2024, pp. 343–358. find
  21. Y. Tong, D. Li, S. Wang, Y. Wang, F. Teng, J. Shang, Can llms learn from previous mistakes? investigating llms’ errors to boost for reasoning, arXiv preprint arXiv:2403.20046 (2024).
  22. K. He, R. Mao, Q. Lin, Y. Ruan, X. Lan, M. Feng, E. Cambria, A survey of large language models for healthcare: from data, technology, and applications to accountability and ethics, Information Fusion 118 (2025) 102963. find
  23. B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu, et al., Qwen2. 5-coder technical report, arXiv preprint arXiv:2409.12186 (2024).
  24. A. Yang, B. Zhang, B. Hui, B. Gao, B. Yu, C. Li, D. Liu, J. Tu, J. Zhou, J. Lin, et al., Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement, arXiv preprint arXiv:2409.12122 (2024).
  25. G. I. Winata, A. Madotto, Z. Lin, R. Liu, J. Yosinski, P. Fung, Language models are few-shot multilingual learners, arXiv preprint arXiv:2109.07684 (2021).
  26. D. Huang, X. Fu, X. Yin, H. Pen, Z. Wang, Automating maritime risk data collection and identification leveraging large language models, in: 2024 IEEE International Conference on Data Mining Workshops (ICDMW), IEEE, 2024, pp. 433–439. find
  27. A. Ramponi, A. Daffara, S. Tonelli, Fine-grained fallacy detection with human label variation, in: L. Chiruzzo, A. Ritter, L. Wang (Eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1… find
  28. H. Al-Omari, M. Abdullah, O. AlTiti, S. Shaikh, Justdeep at nlp4if 2019 task 1: Propaganda detection using ensemble deep learning models, in: Proceedings of the Second Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda, 2019, pp. 113–118. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/meenakshi-2026-rbg-ai-fadeit-prompted-llms/