Elamur Research⌕
☰
home / papers / 2026 / Sentiment Syndicate at SemEval-2026 Task 6: Reframing Political Question–Answer Interactions via Natural Language Inference for Clarity Level Classification

Sentiment Syndicate at SemEval-2026 Task 6: Reframing Political Question–Answer Interactions via Natural Language Inference for Clarity Level Classification

Paper · 2026 · Rafi Rafsan

Summary

This SemEval-2026 Task 6 (CLARITY, Subtask 1) system paper addresses three-way classification of political interview answers as Clear Reply, Ambivalent, or Clear Non-Reply. The data come from the QEvasion dataset, which contains question–answer pairs from televised interviews with US presidents (2006–2023). The authors pooled the public splits and re-divided them for training and validation. They then compared three approaches:

  • a RoBERTa-base classifier over the concatenated question, answer and sub-question;
  • a "hypothesis-augmented" NLI-style setup, in which the answer acts as premise and a templated claim that it clearly addresses the sub-question acts as hypothesis, trained supervised on Longformer-base with 1024-token inputs;
  • few-shot prompting of gpt-oss:120b with JSON output and justifications.

The supervised models used class-weighted cross-entropy to counter strong label imbalance. The NLI-style Longformer system performed best on the hidden test set (macro F1 0.67). It was much better at detecting non-replies and less sensitive to long answers. The authors conclude that framing the task as premise–hypothesis alignment, together with a longer context window, helps capture evasion.

Key points

  • Data: the 3,448 train and 308 test public instances were merged into 3,756 and split 80/20 (3,004 train, 752 dev). The official hidden test set on CodaBench has 237 items. Ambivalent makes up about 60% of the data and Clear Non-Reply about 10%.
  • Best system: allenai/longformer-base-4096 in the NLI-style setup, with max length capped at 1024 tokens due to T4 GPU memory, learning rate 2e-5 and 3 epochs. It reached macro F1 0.67 and accuracy 0.68.
  • Per-class F1:
    • Clear Non-Reply: 0.71 for NLI vs. 0.53 for the RoBERTa baseline and the LLM.
    • Clear Reply: the LLM was best at 0.78.
    • Ambivalent: the RoBERTa baseline was best at 0.73.
  • Length analysis: the test set was split at 500 tokens (213 short vs. 24 long items). RoBERTa fell from 0.62 to 0.52 macro F1 on long items, while Longformer fell from 0.71 to 0.59.
  • Limitations noted by the authors: a single hand-written hypothesis template, a small dataset, and limited hyperparameter tuning. Code is available at github.com/bit-wander/system-syndicate.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksSentiment & emotion · 972 works

Cite

Cite as Rafi Rafsan. 2026. Sentiment Syndicate at SemEval-2026 Task 6: Reframing Political Question–Answer Interactions via Natural Language Inference for Clarity Level Classification. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.350

@inproceedings{rafsan2026sentiment,
  title = {{Sentiment Syndicate at SemEval-2026 Task 6: Reframing Political Question–Answer Interactions via Natural Language Inference for Clarity Level Classification}},
  author = {Rafi Rafsan},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.350},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.350}
}

References

20 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150.
  2. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642.
  3. Peter Bull and Kate Mayer. 1993. How politicians evade questions. Political Psychology, 14(4):651–669. find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages 4171–4186.
  5. Justin M Johnson and Taghi M Khoshgoftaar. 2019. Survey on deep learning with class imbalance. Journal of Big Data, 6(1):1–54. find
  6. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems, 35:22199– 22213.
  7. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  8. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. International Conference on Learning Representations. find
  9. OpenAI. 2025. gpt-oss-120b & gpt-oss-20b model card. Preprint, arXiv:2508.10925.
  10. Alexandr Pak and et al. 2024. Word embeddings: A comprehensive survey. Computación y Sistemas, 28(4):2005–2029. Epub March 25, 2025. find
  11. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, and 1 others. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, pages 8024…
  12. Daniel Preotiuc-Pietro, Ye Liu, Daniel Hopkins, and Lyle Ungar. 2017. Beyond binary labels: Political ideology prediction from text. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 729–740. find
  13. Parameswari Rasiah. 2010. A framework for the analysis of evasion in parliamentary discourse. Journal of Pragmatics, 42(3):664–680. find
  14. Timo Schick and Hinrich Schütze. 2021. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255–269.
  15. Konstantinos Thomas, Giorgos Filandrianos, Maria Lymperaiou, Chrysoula Zerva, and Giorgos Stamou. 2024. “I never said that”: A dataset, taxonomy and baselines on response clarity classification. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 5204–5233, Miami, Florida… find
  16. Konstantinos Thomas, Giorgos Filandrianos, Maria Lymperaiou, Chrysoula Zerva, and Giorgos Stamou. 2026. Semeval-2026 task 6: Clarity – unmasking political question evasions. Preprint, arXiv:2603.14027.
  17. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei-Fei Li, Denny Zhou, Ed Chi, and Quoc V Le. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824– 24837.
  18. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 38 others. 2020. Trans…
  19. Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019. Benchmarking zero-shot text classification: Datasets, evaluation and entailment. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJC… find
  20. Caleb Ziems, William Held, Omar Asiedu, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2024. Can large language models transform computational social science? Nature Computational Science, pages 1–13.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rafsan-2026-sentiment-syndicate-semeval-2026-task/