Elamur Research⌕
☰
home / papers / 2026 / SemEval-2026 Task 6: CLARITY – Unmasking Political Question Evasions

SemEval-2026 Task 6: CLARITY – Unmasking Political Question Evasions

Paper · 2026 · Konstantinos Thomas, Giorgos Filandrianos, Maria Lymperaiou et al.

Summary

CLARITY, SemEval-2026 Task 6, addresses a gap in NLP: detecting when politicians avoid answering a question while still appearing responsive. Earlier work on fallacies does not cover this, because fallacy detection needs an explicit argument to evaluate. The organizers built a benchmark from U.S. presidential interviews on the official White House website, covering Bush, Obama, Trump and Biden from 2006 to 2023. ChatGPT (gpt3.5-turbo) split multi-part questions into sub-question/sub-answer pairs. Trained annotators then checked these pairs and labelled them under an expert-supervised, two-level taxonomy taken from Thomas et al. (2024).

The task has two subtasks, both scored with Macro-F1:

  • Clarity-level: three coarse classes.
  • Evasion-level: nine fine-grained evasion strategies.

The fine-grained subtask proved much harder. The winning clarity system reached 0.89, against 0.68 for the best evasion system. The strongest systems prompted LLMs and used the hierarchy between the two label levels, rather than treating the subtasks separately. The benchmark makes strategic ambiguity a measurable target for computational discourse analysis.

Key points

  • Data: 287 interviews yielding 3,756 annotated QA pairs (3,448 train, 308 validation), plus 237 test pairs.
  • Annotation: three annotators and a political-science expert labelled the data. Agreement was measured with Fleiss κ on a jointly annotated subset. Inserted counterfactual QAs confirmed that annotators consulted the full interview text.
  • Labels: the clarity level uses Clear Reply, Ambivalent and Clear Non-Reply. The nine evasion-level categories, listed alphabetically, are Claims Ignorance, Clarification, Declining, Deflection, Dodging, Explicit, General, Implicit and Partial.
  • Participation: 124 teams registered and 30 wrote system papers. Teams submitted 946 valid clarity runs and 539 evasion runs.
  • Results: the clarity winner improved on the best baseline by 7 points (0.82 → 0.89). Mid-sized open-weight models were competitive with proprietary ones, and prompt design mattered more than model size.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National Technical University of AthensAthens · Greece8 works · 461 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 works

Cite

Cite as Konstantinos Thomas, Giorgos Filandrianos, Maria Lymperaiou, Chrysoula Zerva and Giorgos Stamou. 2026. SemEval-2026 Task 6: CLARITY – Unmasking Political Question Evasions. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.449

@inproceedings{thomas2026semeval,
  title = {{SemEval-2026 Task 6: CLARITY – Unmasking Political Question Evasions}},
  author = {Konstantinos Thomas, Giorgos Filandrianos, Maria Lymperaiou, Chrysoula Zerva and Giorgos Stamou},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.449},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.449}
}

References

28 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. Michael Alvarez and Jacob Morrier. 2026. Measuring the quality of answers in political qas with large language models. Political Analysis, 34(1):78–95. find
  2. Lisa P. Argyle, Ethan C. Busby, Joshua R. Gubler, Alex Lyman, Justin Olcott, Jackson Pond, and David Wingate. 2025. Testing theories of political persuasion using ai. Proceedings of the National Academy of Sciences, 122(18):e2412815122. find
  3. Joris Baan, Wilker Aziz, Barbara Plank, and Raquel Fernández. 2022. Stop measuring calibration when humans disagree. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1892–1915. find
  4. Eric Baumer, Elisha Elovic, Ying Qin, Francesca Polletta, and Geri Gay. 2015. Testing and comparing computational approaches for identifying the language of framing in political news. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics… find
  5. Janet Beavin Bavelas, Alex Black, Lisa Bryson, and Jennifer Mullett. 1988. Political equivocation: A situational explanation. Journal of Language and Social Psychology, 7(2):137–145. find
  6. Peter Bull. 1998. Equivocation theory and news interviews. Journal of Language and Social Psychology, 17(1):36–51. find
  7. Peter Bull. 2003. The Microanalysis of Political Communication: Claptrap and Ambiguity. Routledge. find
  8. Peter Bull and Kate Mayer. 1993. How not to answer questions in political interviews. Political Psychology, 14(4):651–666. find
  9. Peter Bull and Will Strawson. 2019. Can’t answer? won’t answer? an analysis of equivocal responses by theresa may in prime minister’s questions. Parliamentary Affairs, 73(2):429–449. find
  10. Michael Burnham. 2025. Stance detection: a practical guide to classifying political beliefs in text. Political Science Research and Methods, 13(3):611–628. find
  11. Eva Cantín Larumbe and Adriana Chust Vendrell. 2025. Argumentative fallacy detection in political debates. In Proceedings of the 12th Argument mining Workshop, pages 369–373, Vienna, Austria. Association for Computational Linguistics. find
  12. Steven E. Clayman. 2001. Answers and evasions. Language in Society, 30(3):403–442. find
  13. Hung Du, Srikanth Thudumu, Sankhya Singh, Scott Barnett, Irini Logothetis, Rajesh Vasa, and Kon Mouzakis. 2022. A framework for evaluating mrc approaches with unanswerable questions. In 2022 IEEE 18th International Conference on e-Science (e-Science), pages 435–436. find
  14. Elisa Ferracane, Greg Durrett, Junyi Jessy Li, and Katrin Erk. 2021. Did they answer? subjective acts and intents in conversational discourse. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1626… find
  15. Anjalie Field, Doron Kliger, Shuly Wintner, Jennifer Pan, Dan Jurafsky, and Yulia Tsvetkov. 2018. Framing and agenda-setting in Russian news: a computational analysis of intricate political strategies. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3…
  16. J.L. Fleiss and 1 others. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378–382. find
  17. Arnav Hiray, Yunsong Liu, Mingxiao Song, Agam Shah, and Sudheer Chava. 2024. CoCoHD: Congress committee hearing dataset. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 15529–15542, Miami, Florida, USA. Association for Computational Linguistics. find
  18. Mohit Iyyer, Peter Enns, Jordan Boyd-Graber, and Philip Resnik. 2014. Political ideology detection using recursive neural networks. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1113–1122, Baltimore, Maryland. Association fo…
  19. Zhijing Jin, Abhinav Lalwani, Tejas Vaidhya, Xiaoyu Shen, Yiwen Ding, Zhiheng Lyu, Mrinmaya Sachan, Rada Mihalcea, and Bernhard Schölkopf. 2022. Logical fallacy detection. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 7180–7198, Abu Dhabi, United Arab Emirates. Asso…
  20. Mirko Lai, Alessandra Cignarella, Delia Hernandez Farias, Cristina Bosco, Viviana Patti, and Paolo Rosso. 2020. Multilingual stance detection in social media political debates. Computer Speech & Language, 63:101075. find
  21. Eleonora Mancini, Federico Ruggeri, and Paolo Torroni. 2024. Multimodal fallacy classification in political debates. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pages 170–178, St. Julian’s, Malta. Associatio… find
  22. Barbara Plank. 2022. The “problem” of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671–10682.
  23. Lavanya Prahallad, Sai Utkarsh Choudarypally, Pragna Prahallad, and Pranathi Prahallad. 2026. Prompt-based clarity evaluation and topic detection in political question answering. Preprint, arXiv:2601.08176. — [arXiv:2601.08176](https://arxiv.org/abs/2601.08176) find
  24. Parameswary Rasiah. 2010. A framework for the systematic analysis of evasion in parliamentary discourse. Journal of Pragmatics, 42:664–680. find
  25. Julio Reyes-Montesinos, Álvaro Rodrigo, and Anselmo Peñas. 2025. None of the above: comparing scenarios for answerability detection in question answering systems. Applied Intelligence, 55(12):887. find
  26. Manjari Rudra, Daniel Magleby, and Sujoy Sikdar. 2025. C-queri: Congressional questions, exchanges, and responses in institutions dataset. Preprint, arXiv:2509.21548. — [arXiv:2509.21548](https://arxiv.org/abs/2509.21548) find
  27. Elior Sulem, Jamaal Hay, and Dan Roth. 2021. Do we know what we don’t know? studying unanswerable questions beyond SQuAD 2.0. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4543–4548, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  28. The expected agreement by chance is: P ¯ e = κ = Human annotation find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/thomas-2026-semeval-2026-task-6-clarity/