Elamur Research⌕
☰
home / papers / 2026 / SyntaxMind at SemEval-2026 Task 6: Exploring Transformers and LLMs for Unmasking Political Question Evasions

SyntaxMind at SemEval-2026 Task 6: Exploring Transformers and LLMs for Unmasking Political Question Evasions

Paper · 2026 · Md. Shihab Uddin Riad

Summary

This system paper addresses Subtask 1 (Clarity-level Classification) of SemEval-2026 Task 6 (CLARITY). The task is to label a politician's response to an interview question by how clearly it answers. There are three classes: Clear Reply, Ambivalent and Clear Non-Reply.

The pipeline has three parts:

  • Auxiliary input: small language models (under 2B parameters) write a short "direct answer" from the interview context, which is fed to the classifier next to the original question–answer pair.
  • Domain adaptation: encoder-only models (BERT, RoBERTa, DeBERTaV3, ModernBERT, ALBERT) receive Task-Adaptive Pre-Training (TAPT) via masked language modeling on unlabeled in-domain text, including a released presidential news-conference corpus.
  • Fine-tuning: the encoders are trained with either cross-entropy or focal loss. Small instruction-tuned generative models are also trained with QLoRA.

DeBERTa-V3-base with TAPT and focal loss performed best. The final system scored 0.72 macro-F1 on the official evaluation set, placing 24th of 40 teams. The generative models of about 1B parameters or fewer lagged far behind. The work matters because it shows that compact, domain-adapted encoders are a cheap and competitive way to detect evasive political answers.

Key points

  • Official result: 0.72 macro-F1, rank 24 of 40. The top team, TeleAI, reached 0.89.
  • Test-set comparison of TAPT encoders: DeBERTa-V3-base with focal loss was best (0.61, versus 0.53 with CE). Focal loss also helped ModernBERT-base and ALBERT-V2-base, but slightly hurt BERT-base and RoBERTa-base.
  • QLoRA-tuned small generative models: SmolLM2-360M-Instruct 0.33, LFM2.5-1.2B-Instruct 0.27 and Qwen2.5-0.5B-Instruct 0.26 macro-F1. All were well below the encoders.
  • Error analysis (DeBERTa-V3-base): the Ambivalent class is easiest (170 of 206 correct). Clear Replies are often labeled Ambivalent (51 cases), and Clear Non-Replies are sometimes confused with Ambivalent (7 cases).
  • Resources and setup: the TAPT corpus is released on Hugging Face as shihab-0x0/presidents-news-conferences. Training used AdamW, learning rate 2e-5, batch size 16, 5 epochs, max length 512, and focal loss with γ=2.0 and α=0.25, all run on Kaggle.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Md. Shihab Uddin Riad. 2026. SyntaxMind at SemEval-2026 Task 6: Exploring Transformers and LLMs for Unmasking Political Question Evasions. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.419

@inproceedings{riad2026syntaxmind,
  title = {{SyntaxMind at SemEval-2026 Task 6: Exploring Transformers and LLMs for Unmasking Political Question Evasions}},
  author = {Md. Shihab Uddin Riad},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.419},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.419}
}

References

18 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Liquid AI. 2025. Lfm2 technical report. arXiv preprint arXiv:2511.23404. — [arXiv:2511.23404](https://arxiv.org/abs/2511.23404) find
  2. Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, C… — SmolLM2: When Smol Goes Big — Data-Centric Training of a Small Language Model
  3. Peter Bull and Kate Mayer. 1993. How not to answer questions in political interviews. Political Psychology, 14(4):651. find
  4. I. Íñigo-Mora and K. Deligiorgi. 2007. Evasion in political interviews: An analysis of televised interviews with Tony Blair. Political Linguistics, 23(3):78–90. find
  5. Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2018. Focal loss for dense object detection. Preprint, arXiv:1708.02002.
  6. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
  7. Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, Benjamin Bossan, and Marian Tietz. 2022. PEFT: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft. ↗
  8. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, and 2 others. 2019.… — PyTorch: An Imperative Style, High-Performance Deep Learning Library
  9. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for squad. Preprint, arXiv:1806.03822.
  10. Qwen Team. 2024. Qwen2.5: A party of foundation models. find
  11. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314.
  12. Konstantinos Thomas, Giorgos Filandrianos, Maria Lymperaiou, Chrysoula Zerva, and Giorgos Stamou. 2024. "i never said that": A dataset, taxonomy and baselines on response clarity classification. Preprint, arXiv:2409.13879. find
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
  14. Konstantinos Thomas, Giorgos Filandrianos, Maria Lymperaiou, Chrysoula Zerva, and Giorgos Stamou. 2026. Semeval-2026 task 6: Clarity – unmasking political question evasions. Preprint, arXiv:2603.14027.
  15. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. Preprint, arXiv:2004.10964. find
  16. Ellen M. Voorhees and Dawn M. Tice. 2000. The TREC-8 question answering track. In Proceedings of the Second International Conference on Language Resources and Evaluation (LREC’00), Athens, Greece. European Language Resources Association (ELRA).
  17. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. 2024. Smarter, better, faster, longer: A modern bidirectional encoder for fast,…
  18. A.2 Prompt for LoRA fine-tuning System Prompt You are a classification assistant. Given an interview question and its answer, classify the answer into exactly one of the following categories: - Clear Reply - Clear Non-Reply Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue,… — Transformers: State-of-the-Art Natural Language Processing

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/riad-2026-syntaxmind-semeval-2026-task-6/