📄 JIA at SemEval-2026 Task 10: A Dual-Track System with BERT-based Encoders and LLMs for Conspiracy Analysis

Summary

This paper presents a system for SemEval-2026 Task 10, which covers two subtasks on Reddit comments: extracting psycholinguistic conspiracy markers (Subtask 1) and binary conspiracy detection (Subtask 2). The author uses the PsyCoMark dataset, which draws on more than 190 subreddits. After removing duplicates and conflicting annotations, 3,237 training texts remained.

The system runs two tracks in parallel:

• Track A: BERT-family encoders trained with multi-task learning. Text representations are combined with learned embeddings of metadata (subreddit, source ID, annotator) for classification, and BIO tagging handles the markers.
• Track B: Qwen2.5-Instruct models (7B and 14B) prompted to return JSON. They were tested in zero-shot, five-shot and LoRA fine-tuned settings, and the generated marker spans were mapped back to positions in the original text.

For detection, the fine-tuned Qwen2.5-14B scored best (Weighted F1 0.80), ahead of the strongest encoder, DeBERTa-V3-Large (0.76). Marker extraction proved much harder: all scores were low, and they fell sharply from the development set to the test set. The work shows that instruction tuning helps LLMs on narrative-level classification, while span-level extraction of conspiracy markers remains largely unsolved.

Key points

• Data: PsyCoMark contains 4,316 labeled training instances, plus 100 dev and 337 test instances without labels. Cleaning removed 645 duplicates and 434 conflicting samples. The remaining texts were labeled No (1,541), Yes (1,108) or Can't tell (588), and were annotated with five marker types: Actor, Action, Effect, Evidence and Victim.
• Models compared: DistilBERT, BERT-Base, RoBERTa-Large and DeBERTa-V3-Large on the encoder side, and Qwen2.5-7B/14B-Instruct with PEFT/LoRA adapters in the attention projection layers on the generative side.
• Detection results:
  • Qwen2.5-14B fine-tuned on the full set reached Acc/W-F1 of 0.80/0.80.
  • The zero-shot Qwen2.5-7B reached only 0.59 W-F1.
  • DistilBERT dropped from 0.80 on dev to 0.72 on test, which points to weak generalization.
• Marker extraction results: Spans were scored with a token-level IoU threshold of 0.5. The fine-tuned Qwen2.5-7B led with an aggregate F1 of 0.094 and an Actor F1 of 0.144. RoBERTa-Large was best on Victim (0.085). Most zero-shot and few-shot LLM scores stayed below 0.041.
• Data analysis and future work: About 92% of conspiracy texts contain Actor and Action markers. The author proposes adding external knowledge bases and synthetic data augmentation to address the remaining gaps.