📄 Duluth at SemEval-2026 Task 6: DeBERTa with LLM-Augmented Data for Unmasking Political Question Evasions

Summary

The Duluth team built a system for SemEval-2026 Task 6 (CLARITY), which asks whether answers in U.S. presidential interviews actually respond to the question. Task 1 sorts question–answer pairs into three clarity labels: Clear Reply, Ambivalent and Clear Non-Reply. Task 2 uses nine finer evasion techniques. The team fine-tuned DeBERTa-V3-base with:

• focal loss,
• layer-wise learning rate decay,
• two binary discourse features from the QEvasion data.

The QEvasion training data is heavily skewed toward Ambivalent, so the authors added synthetic minority-class examples in two ways. Gemini 3 produced frame-based context-aware generations that fully balance the classes. Claude Sonnet 4.5 produced EDA-style lexical paraphrases that balance them only partly.

With Gemini augmentation, the model reached 0.76 Macro F1 when evaluated on the Task 1 evaluation set and placed 8th among 40 teams. Most remaining errors come from mixing up Ambivalent and Clear Reply answers, which is also where human annotators tend to disagree. The study suggests that LLM-generated data can raise minority-class performance on subtle political discourse. Hedged or partial answers remain the main open problem.

Key points

• Training data: QEvasion has 3,448 training pairs (59.2% Ambivalent, 30.5% Clear Reply, 10.3% Clear Non-Reply) and a 308-sample test set. The final evaluation set has 237 samples.
• Augmentation:
  • Gemini 3 (CASA-style) added 2,672 examples, giving 2,040 per class.
  • Claude Sonnet 4.5 (EDA-style) added 1,086 examples.
  • Synthetic examples were down-weighted in the loss: 0.7 for Gemini data, 0.5 for Claude data.
• Task 1 evaluation F1: 0.69 with original data only, 0.74 with Claude augmentation and 0.76 with Gemini augmentation. The top system (TeleAI) reached 0.89, and the participant mean was 0.70.
• Task 2: DeBERTa-V3-large with focal loss reached 0.45 in the development phase but fell to 0.28 in the evaluation phase (30th of 33). The authors attribute the drop to distribution shift.
• Error analysis and other models:
  • Ambivalent–Clear Reply confusion accounts for 68% of test errors and 65% of evaluation errors.
  • Political DEBATE reached 0.57 test F1, and DeBERTa-v3-large-nli reached 0.66.
  • The majority-class baseline scored 0.27.