ThinkVision at SemEval-2026 Task 6: A Transformer-Based Ensemble System for Clarity Detection
Paper · 2026 · Gourav Ghanshyam Purohit, Praveen Swami, Shriyans S Sahoo et al.
Summary
The paper addresses Subtask 1 of SemEval-2026 Task 6 (CLARITY). Given a political interview question and its response, a system must label the response as Clear Reply, Clear Non-Reply, or Ambivalent. The authors used only the organizers' data. Its training split comes from QEvasion, and the labels are heavily skewed toward Ambivalent.
The system fine-tunes DeBERTa-v3-base on a templated question-plus-answer string. It replaces the CLS vector with mask-weighted mean pooling, adds a small feedforward head, and weights the loss by inverse class frequency. Five copies trained with different seeds form an ensemble. At inference, each model contributes one deterministic pass and one Monte Carlo Dropout pass, and the system averages all logits.
The submission reached Macro-F1 0.76 and placed 17th. In the ablation, seed ensembling gave the largest gain. Error analysis shows that partially evasive answers are often mistaken for direct ones. This suggests that topical overlap is a poor signal for whether a question was actually answered, which matters for evasion detection and discourse-aware classification.
Key points
- Data: CLARITY Subtask 1 has 3,450 training, 308 test and 237 evaluation pairs. Ambivalent makes up about 59% of the training split, and Clear Non-Reply only about 10%. No external resources were used.
- Setup: DeBERTa-v3-base with max length 320 was trained for 4 epochs with AdamW (lr 4e-5, batch 20). The ensemble uses seeds {42, 123, 456, 789, 2024}. Training ran on Kaggle T4 GPUs.
- Ablation (Macro-F1):
- Single model: 0.55
- With class-weighted loss: 0.61
- 5-model ensemble: 0.739
- Ensemble with MC Dropout: 0.76
- Caveat: the intermediate configurations were scored on the test split, while the final system was scored on the official evaluation set.
- Per-class F1 (final system): Clear Non-Reply 0.82, Ambivalent 0.73, Clear Reply 0.72. Most errors were Ambivalent responses predicted as Clear Reply, especially answers to multi-part questions and answers using procedural or reframing language.
- Code: github.com/gourav18dart/CLARITY-SEMEVAL2026-System. The authors note two limitations: ensembling raises compute cost, and the system does not explicitly model discourse structure.
Institutions
IndiaFields, topics and concepts
📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksGeneral Social Sciences · 227 works
Cite
Cite as Gourav Ghanshyam Purohit, Praveen Swami, Shriyans S Sahoo, Jenish Bhati, Supriya Nadiger and Sunil Saumya. 2026. ThinkVision at SemEval-2026 Task 6: A Transformer-Based Ensemble System for Clarity Detection. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).
@inproceedings{purohit2026thinkvision,
title = {{ThinkVision at SemEval-2026 Task 6: A Transformer-Based Ensemble System for Clarity Detection}},
author = {Gourav Ghanshyam Purohit, Praveen Swami, Shriyans S Sahoo, Jenish Bhati, Supriya Nadiger and Sunil Saumya},
booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
year = {2026},
doi = {10.18653/v1/2026.semeval-1.248},
url = {https://doi.org/10.18653/v1/2026.semeval-1.248}
}References
- Peter Bull. 1994. On identifying questions, replies, and non-replies in political interviews. Journal of Language and Social Psychology, 13(2):115–131. find
- Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia. Association for Computational Ling…
- Konstantinos Thomas, Giorgos Filandrianos, Maria Lymperaiou, Chrysoula Zerva, and Giorgos Stamou. 2024. I never said that: A dataset, taxonomy and baselines on response clarity classification. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 5204–5233, Miami, Florida,… find
- Konstantinos Thomas, Giorgos Filandrianos, Maria Lymperaiou, Chrysoula Zerva, and Giorgos Stamou. 2026. SemEval-2026 Task 6: CLARITY – Unmasking Political Question Evasions. Preprint, arXiv:2603.14027.
- Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clément Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Transf…
- Elisa Ferracane, Greg Durrett, Junyi Jessy Li, and Katrin Erk. 2021. Did they answer? subjective acts and intents in conversational discourse. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1626… find
- Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning (ICML). find
- Pengcheng He, Jianfeng Gao, Anoop Kumar, Mingjian Huang, Xiaodong He, Mingyang Zhou, Lei Ji, Yu Cao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. Preprint, arXiv:2111.09543.
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/purohit-2026-thinkvision-semeval-2026-task-6/