📄 ttda704 at SemEval-2026 Task 6: Structured Chain-of-Thought Prompting for Political Evasion Detection

Summary

Politicians often sidestep interview questions, and detecting this automatically is difficult because evasion depends on pragmatic inference rather than obvious wording. This system paper for SemEval-2026 Task 6 (CLARITY) addresses two nested subtasks on English question–answer pairs from U.S. presidential interviews. Subtask 1 is a three-way clarity classification, and Subtask 2 is a nine-way evasion-strategy classification. Both are scored with Macro F1.

The authors compare two paradigms:

• Fine-tuning: Qwen3 models (4B–32B) adapted with QLoRA, using upsampling of rare classes and weighted cross-entropy to counter label imbalance.
• Prompting: reasoning-capable APIs (DeepSeek-V3.2, Grok-4-Fast) given structured CoT templates. These templates ask the model to assess directness and topic fidelity before choosing a label. Variants optionally arrange labels as a hierarchical taxonomy and add few-shot exemplars.

Prompting clearly outperforms the fine-tuning baseline, and enabling extended reasoning modes brings the biggest improvement. However, the top prompt variants are not statistically separable. Error analysis shows persistent confusion among Dodging, Deflection, General and Implicit. The study suggests that explicit reasoning scaffolds help with pragmatically subtle classification.

Key points

• Data: 3,448 training and 308 test samples, heavily skewed. Explicit makes up 30.5% and Dodging 20.5%, while Partial/half-answer (2.3%) and Clarification (2.7%) are rare.
• Best system: Grok-4-Fast with reasoning and few-shot hierarchical CoT scored Macro F1 0.5147 on Subtask 2 and 0.7979 on Subtask 1 on the 308-sample test split.
  • On the official leaderboard it ranked 8/33 on Subtask 2 (0.56) and 13/41 on Subtask 1 (0.79).
  • The revised organizer baseline is 0.57 and 0.82.
• Fine-tuning results: the best Qwen3 PEFT model (Qwen3-32B, few-shot CoT) reached only 0.3630 on Subtask 2. In a 100-sample diagnostic, turning on Qwen3 thinking mode raised evasion F1 from 0.0734 to 0.2362.
• Significance testing: the top three systems have overlapping 95% CIs, and approximate randomization finds no significant difference (p > 0.19). McNemar's test (p = 0.015) suggests Grok and DeepSeek make complementary errors. The best open-weight option is DeepSeek-V3.2 at 0.5115.
• Error patterns: Dodging recall is only 0.1852, 42% of Implicit answers are labelled Explicit, and Partial/half-answer F1 is 0.125. Code and prompts are released on GitHub (taitran501/SemEval-2026-Task6).