📄 Team Evaluators at SemEval-2026 Task 6: Instruction-Tuned LLMs for Clarity and Evasion Classification in Political Interviews

Summary

This paper presents a system for the SemEval-2026 CLARITY shared task (Task 6), which asks whether politicians' answers in interviews and debates actually address the question. Subtask 1 assigns one of three clarity categories: Clear Reply, Ambivalent/Ambiguous, or Clear Non-Reply. Subtask 2 assigns one of nine evasion techniques, such as Dodging, Deflection or Claims ignorance.

The authors use the QEvasion dataset and turn each question–answer pair into an instruction-style prompt that includes label definitions. They then train open-source LLMs (Phi-4-14B-Instruct, Qwen3-14B, Ministral-3-7B) to generate the label, using LoRA adapters with supervised fine-tuning and a separate model per subtask. They also test a hybrid pipeline in which a second LLM reviews the SFT predictions.

Plain LoRA SFT with Phi-4-14B-Instruct worked best, reaching Macro F1 of 0.83 on Subtask 1 (5th place) and 0.54 on Subtask 2 (9th place). The verification stage made results less stable and lowered scores. The study suggests that lightweight adaptation of mid-sized open models is a practical way to detect strategic ambiguity in political speech.

Key points

• QEvasion provides 3,448 training and 308 test instances. The training clarity labels are skewed: 2,040 Ambivalent, 1,052 Clear Reply and 356 Clear Non-Reply. Rare evasion classes include Partial/half-answer (79) and Clarification (92).
• The LoRA configuration uses rank 8, α 32 and dropout 0.1, applied to the attention and feed-forward layers. Training uses AdamW with a learning rate of 2e-4 for 3 epochs, gradient accumulation of 8, and a class-weighted loss.
• On Subtask 1, Phi-4-14B-Instruct scored 0.72 on dev and 0.83 on test, and Qwen3-14B reached 0.80 on test. The hybrid pipeline (Qwen3 + LLaMA) scored only 0.43 on dev.
• On Subtask 2, Phi-4-14B-Instruct scored 0.56 on dev and 0.54 on test, while the hybrid pipeline fell to 0.31 on test. The authors attribute this drop to overcorrection.
• The main sources of error were the boundary between Ambivalent and Clear Reply, and overlapping evasion classes such as Deflection vs. Dodging and Implicit vs. General. The paper is inconsistent about model naming, mentioning both Qwen3-7B and Qwen3-14B.