Elamur Research⌕
☰
home / papers / 2026 / NJUSTKMG at SemEval 2026 Task 10 PsyCoMark–-Subtask 2:Conspiracy Detection

NJUSTKMG at SemEval 2026 Task 10 PsyCoMark–-Subtask 2:Conspiracy Detection

Paper · 2026 · Yuhan Zheng and Yang Yang

Summary

The paper presents the winning system for SemEval-2026 Task 10 (PsyCoMark), Subtask 2. The task is to decide whether a Reddit submission statement contains a conspiracy-related claim. The authors build a cascade.

  1. An LLM first screens every test item and labels obviously non-conspiratorial texts as No.
  2. The remaining, more ambiguous items go to a second stage. Evidence passages are retrieved from the official training set and reranked. They are then inserted into a custom prompt for an LLM.
  3. The LLM's probability is combined by weighted fusion with the output of a fine-tuned RoBERTa-Large classifier trained with class weights.
  4. Decision thresholds and fusion weights are tuned on the development set.

An ablation shows each component (data processing, class weighting, prompted LLMs, ensembling) adding to the score. The full system reaches 0.8874 and takes first place, about 0.11 ahead of the runner-up.

The work shows that cheap LLM triage combined with retrieval-grounded prompting and a discriminative encoder suits ambiguous, evidence-dependent classification. However, the paper's description is uneven: it alternates between a two-stage and a four-stage framing, and it calls the metric both weighted and macro F1.

Key points

  • Ablation (Macro F1):
    • RoBERTa baseline: 0.7451
    • with data processing: 0.7812
    • with class weights: 0.8191
    • with prompted LLMs: 0.8442
    • with ensemble learning: 0.8874
  • Leaderboard: NJUST_KMG ranked 1st with 0.89. Teams mdok-style, dangphuduy and VARH-AI followed at 0.78, and UMUTeam at 0.77.
  • Stage 1: GPT-5.2 and Qwen2.5-7B-Instruct, with temperature and top-p both 1.0. Items with p_yes below the filter threshold τ_filter = 0.3 are labeled No. Items whose output cannot be parsed are passed on to Stage 2.
  • Retrieval:
    • BM25 plus dense retrieval with bge-large-en-v1.5, recalling k=20 candidates
    • reranking with bge-reranker-large, keeping m=3 evidence passages
  • RoBERTa-Large and fusion:
    • training: learning rate 2×10⁻⁵, 10 epochs, weight decay 0.01, batch size 10
    • fusion weights: w_RoB = 0.35, w_LLM = 0.65
    • final threshold: τ_final = 0.73

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Nanjing University of Science and TechnologyNanjing · China19 works · 1,590 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSociology and Political Science · 626 works

Cite

Cite as Yuhan Zheng and Yang Yang. 2026. NJUSTKMG at SemEval 2026 Task 10 PsyCoMark–-Subtask 2:Conspiracy Detection. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.128

@inproceedings{zheng2026njustkmg,
  title = {{NJUSTKMG at SemEval 2026 Task 10 PsyCoMark–-Subtask 2:Conspiracy Detection}},
  author = {Yuhan Zheng and Yang Yang},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.128},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.128}
}

References

10 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yang Yang, Yuxuan Zhang, Xin Song, and Yi Xu. 2023. Not all out-of-distribution data are harmful to open-set active learning. Advances in Neural Information Processing Systems, 36:13802–13818. find
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  3. Yuxuan Zhanga, Longfei Huanga, and Yang Yanga. Multimodal semantic decoupled prompt for zero-shot referring expression comprehension. find
  4. Lingjiao Chen, Matei Zaharia, and James Zou. 2023. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176.
  5. Debanjan Ghosh, Kai North, Ekaterina Kochmar, Mamoru Komachi, and Marcos Zampieri, editors. 2026. Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026). Association for Computational Linguistics, San Diego, United States. find
  6. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International conference on machine learning, pages 3929–3938. PMLR.
  7. QingYuan Jiang, Longfei Huang, and Yang Yang. 2025. Rethinking multimodal learning from the perspective of mitigating classification ability disproportion. arXiv preprint arXiv:2502.20120. — [arXiv:2502.20120](https://arxiv.org/abs/2502.20120) find
  8. Ruoxuan Li, Xiangyu Wu, and Yang Yang. 2025. Noise self-correction via relation propagation for robust cross-modal retrieval. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 4748–4757. find
  9. Zhiwei Liu, Boyang Liu, Paul Thompson, Kailai Yang, and Sophia Ananiadou. 2024. Conspemollm: Conspiracy theory detection using an emotion-based large language model. arXiv preprint arXiv:2403.06765. — [arXiv:2403.06765](https://arxiv.org/abs/2403.06765) find
  10. Milena Pustet, Elisabeth Steffen, and Helena Mihaljević. 2024. Detection of conspiracy theories beyond keyword bias in german-language telegram using large find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zheng-2026-njustkmg-semeval-2026-task-10/