Elamur Research⌕
☰
home / papers / 2026 / SemEval-2026 Task 10: PsyCoMark – Psycholinguistic Conspiracy Marker Extraction and Detection

SemEval-2026 Task 10: PsyCoMark – Psycholinguistic Conspiracy Marker Extraction and Detection

Summary

PsyCoMark is a SemEval-2026 shared task on recognizing conspiracy-theory language in ordinary online conversation. Most prior benchmarks target a few fixed claims or topics, such as COVID-19 on Twitter. To counter this, the organizers built a Reddit corpus of submission statements, the short comments in which posters explain why shared media fits a subreddit. These come from 226 communities, ranging from openly conspiratorial forums to general news, TV and politics, so models cannot rely on topic cues. Crowdworkers on Prolific labeled each text as conspiracy, not conspiracy or can't tell. They also marked spans for five markers derived from evolutionary psychology: Actor, Action, Effect, Victim and Evidence.

The task has two parts: span extraction of markers (Subtask 1) and document-level detection (Subtask 2). Scoring combined standard metrics with a Bayesian hierarchical analysis of subreddit difficulty. Extraction proved much harder than detection. The best systems paired fine-tuned transformers with LLM components, data augmentation and marker-aware modeling. Few teams exploited the strong link between markers and labels, which leaves room for joint approaches.

Key points

  • Dataset size: 5,354 annotated documents (4,720 unique) from 226 subreddits, labeled by 138 native English speakers. They contain 27,891 marker spans. Documents split into 1,925 "conspiracy", 2,457 "not conspiracy" and 972 "can't tell".
  • Annotator agreement: Krippendorff's α was 0.58 for binary labels, but unitizing α was only 0.15 for marker spans.
  • Markers and labels: Texts containing all five markers had the highest conspiracy rate. However, the particular combination of markers mattered more than their raw count. Actor and Action were the most frequent markers and co-occurred most often.
  • Topic is a weak signal: Conspiracy labels were mixed even within conspiracy-focused subreddits. Polarized, viewpoint-diverse communities such as TheMotte and Israel_Palestine were the hardest to predict.
  • Release and top system: The data are on Zenodo under CC-BY, without raw text; a rehydration script is provided, and 20% was held out as a private test set. The winning system (Zheng and Yang, 2026) combined retrieval-augmented prompts, keyword-based LLM filtering and weighted classification.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Psycholinguistics & language acquisition

Institutions

🎓Sapienza University of RomeRome · Italy29 works · 6 807 citations🎓Karlsruhe Institute of TechnologyKarlsruhe · Germany4 works · 425 citations

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.444

@inproceedings{samory2026semeval,
  title = {{SemEval-2026 Task 10: PsyCoMark – Psycholinguistic Conspiracy Marker Extraction and Detection}},
  author = {Samory, Mattia and Soldner, Felix and Batzdorfer, Veronika},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.444},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.444}
}

References

48 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mian M. Husnain Akram and Mehwish Fatima. 2026. NUST PsyAI at SemEval-2026 task 10: A comparative evaluation of psycholinguistic features, machine learning, and LLMs for conspiracy marker extraction and detection. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026… find
  2. Veronika Batzdorfer, Holger Steinmetz, Marco Biella, and Meysam Alizadeh. 2022. Conspiracy theories on twitter: emerging motifs and temporal dynamics during the covid-19 pandemic. International journal of data science and analytics, 13(4):315–333. find
  3. Roxana Carabas, Anamaria Persida Nacu, Isac Lucian-Constantin, and Daniela Gîfu. 2026. psy detectives at SemEval-2026 task 10: PsyCoMark – psycholinguistic conspiracy marker extraction and detection. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego,… find
  4. Debashish Chakraborty, Sumaiya Tabassum, Sabrina Ib-nath, and Abu Nowshed Chy. 2026. CSECU-DSG at SemEval-2026 task 10: Fine-tuning DeBERTa transformer model for conspiracy detection. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, California, Unit… find
  5. Junpei Chen, You Zhang, Jin Wang, Dan Xu, and Xuejie Zhang. 2026. YNU-HPCC at SemEval-2026 task 10: Pretrained distilbert models for conspiracy marker extraction and detection. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, California, United Stat… find
  6. Duy Dang Phu. 2026. dangphuduy at SemEval-2026 task 10: Span-based conspiracy marker extraction and emotion-aware detection via gated fusion. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, California, United States. Association for Computational L… find
  7. Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker’s guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383–1392, Melbourne, A… find
  8. Marc Faddoul, Guillaume Chaslot, and Hany Farid. 2020. A longitudinal analysis of youtube’s promotion of conspiracy videos. Preprint, arXiv:2003.03318. — [arXiv:2003.03318](https://arxiv.org/abs/2003.03318) find
  9. Faozia Fariha, Lamia Tasnim Khan, Madiha Ahmed Chowdhury, Kawsar Ahmed, and Mohammed Moshiul Hoque. 2026. CUET_320 at SemEval-2026 task 10: Few-shot large language models for psycholinguistic marker extraction and conspiracy detection. In Proceedings of the 20th International Workshop on Semantic Ev… find
  10. Margherita Gambini, Serena Tardelli, and Maurizio Tesconi. 2024. The anatomy of conspiracy theorists: Unveiling traits using a comprehensive twitter dataset. Computer Communications, 217:25–40. find
  11. Jorge Gomez Navalon. 2026. UMUTeam at SemEval-2026 task 10: Transformer ensembles for conspiratorial span extraction and detection. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, California, United States. Association for Computational Linguistics… find
  12. Ekansh Goyal. 2026. TruthGradient at SemEval-2026 task 10: Dual-pooling DeBERTa for conspiracy theory detection. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, California, United States. Association for Computational Linguistics. find
  13. Ishaan Karan. 2026. CredenceAI at SemEval-2026 task 10: A span-consistency network with cross-marker attention for conspiracy marker extraction. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, California, United States. Association for Computationa… find
  14. Muhammad Quddussi Kashaf, Marium Zeeshan, and Shahmir Mustafa Chaudhry. 2026. HU at SemEval-2026 task 10: Leveraging GLiNER with descriptive prompts for conspiracy element extraction. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, California, Unit… find
  15. Damir Korenčić, Berta Chulvi, Xavier Bonet Casals, Alejandro Toselli, Mariona Taulé, and Paolo Rosso. 2024. What distinguishes conspiracy from critical narratives? a computational analysis of oppositional discourse. Expert Systems, 41(11):e13671. find
  16. Johannes Langguth, Daniel Thilo Schroeder, Petra Filkuková, Stefan Brenner, Jesper Phillips, and Konstantin Pogorelov. 2023. COCO: an annotated Twitter dataset of COVID-19 conspiracy theories. Journal of Computational Social Science, pages 1–42. find
  17. Yuanyuan Lei and Ruihong Huang. 2023. Identifying Conspiracy Theories News based on Event Relation Graph. arXiv. find
  18. Dominik Macko. 2026. mdok-style at SemEval-2026 task 10: Finetuning LLMs for conspiracy detection. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, California, United States. Association for Computational Linguistics. find
  19. Teodor-George Marchitan. 2026. Unibuc-NLP at SemEval-2026 task 10: Unmasking conspiracies with pre-trained language models. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, California, United States. Association for Computational Linguistics. find
  20. Ranxiandong. 2026. AGAI at SemEval-2026 task 10: Enhancing conspiracy detection via instruction-tuned LLMs. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, California, United States. Association for Computational Linguistics. find
  21. Dom Marhoefer, Glenn Grant-Richards, Aidan Pinero, Milos Suvakovic, and Ryan King. 2026. UCSC NLP at SemEval-2026 task 10: Boundary-aware span extraction and stratified classification for conspiracy detection. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), S… find
  22. Samuel Richter and Mounika Marreddy. 2026. TT-Lab at SemEval-2026 task 10: Transformer-based approaches for psycholinguistic conspiracy detection in social media discourse. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, California, United States.… find
  23. Alessandro Miani, Thomas Hills, and Adrian Bangerter. 2021. Loco: The 88-million-word language of conspiracy corpus. Behavior research methods, pages 1–24. find
  24. Tanushree Mitra, Scott Counts, and James Pennebaker. 2016. Understanding anti-vaccination attitudes in social media. In Proceedings of the International AAAI Conference on web and Social Media, volume 10, pages 269–278. find
  25. J. D. Moffitt, Catherine King, and Kathleen M. Carley. 2021. Hunting conspiracy theories during the covid-19 pandemic. Social Media + Society, 7(3):20563051211043212. find
  26. Xintong Pan. 2026. Team a at SemEval-2026 task 10: Conspiracy extraction and classification. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, California, United States. Association for Computational Linguistics. find
  27. Samantha C. Phillips, Lynnette Hui Xian Ng, and Kathleen M. Carley. 2022. Hoaxes and hidden agendas: A twitter conspiracy theory dataset: Data paper. In Companion Proceedings of the Web Conference 2022, WWW ’22, page 876–880, New York, NY, USA. Association for Computing Machinery. find
  28. Jan Willem van Prooijen and Mark van Vugt. 2018. Conspiracy Theories: Evolved Functions and Psychological Mechanisms. Perspectives on Psychological Science, 13:770–788. find
  29. Marius Hans Raab, Stefan Andreas Ortlieb, Nikolas Auer, Klara Guthmann, and Claus-Christian Carbon. 2013. Thirty shades of truth: Conspiracy theories as stories of individuation, not of pathological delusion. Frontiers in Psychology, 4:1–9. find
  30. Rofaïda Rabehi, Nicolai Plenk, and Sung-Jin Miriam Han. 2026. Team Macaroni at SemEval-2026 task 10: PsyCoMark – psycholinguistic conspiracy marker extraction and detection. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, California, United States.… find
  31. David Rodriguez Guierrez. 2026. davidinfotec at SemEval-2026 task 10: From lexical baselines to resource-efficient contextual transformers for conspiracy detection. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, California, United States. Associat… find
  32. Mattia Samory and Tanushree Mitra. 2018. ’the government spies using our webcams’ the language of conspiracy theories in online discussions. Proceedings of the ACM on Human-Computer Interaction, 2(CSCW):1–24. find
  33. Shadi Shahsavari, Pavan Holur, Tianyi Wang, Timothy R. Tangherlini, and Vwani Roychowdhury. 2020. Conspiracy in the time of corona: automatic detection of emerging COVID-19 conspiracy theories in social media and the news, volume 3. find
  34. Hritav Singh Solanki, Shubham Sharma, and Manish Prasad. 2026. VARH-AI at SemEval-2026 task 10: Exploiting architectural diversity with transformer-ssm ensembles and confidence-based iterative refinement for conspiracy detection. In Proceedings of the 20th International Workshop on Semantic Evaluati… find
  35. Panagiotis Spanakis, Maria Lymperaiou, Giorgos Filandrianos, Athanasios Voulodimos, and Giorgos Stamou. 2026. AILS-NTUA at SemEval-2026 task 10: Agentic LLMs for psycholinguistic marker extraction and conspiracy endorsement detection. In Proceedings of the 20th International Workshop on Semantic Eva… find
  36. Hidetsune Takahashi. 2026. Hidetsune at SemEval-2026 task 10: A systematic exploration of training and inference strategies for detecting conspiracy beliefs. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, California, United States. Association for… find
  37. Timothy R. Tangherlini, Shadi Shahsavari, Behnam Shahbazi, Ehsan Ebrahimzadeh, and Vwani Roychowdhury. 2020. An automated pipeline for the discovery of conspiracy and conspiracy theory narrative frameworks: Bridgegate, Pizzagate and storytelling on the web. PLoS ONE, 15(6):e0233879. find
  38. Raul Vazquez, Timothee Mickus, Elaine Zosa, Teemu Vahtola, Jörg Tiedemann, Aman Sinha, Vincent Segonne, Fernando Sanchez Vega, Alessandro Raganato, Jindřich Libovický, Jussi Karlgren, Shaoxiong Ji, Jindřich Helcl, Liane Guillou, Ona De Gibert, Jaione Bengoetxea, Joseph Attieh, and Marianna Apidian… find
  39. José-Jorge Vázquez-Cerrillo, Helena Gómez-Adorno, and Gemma Bel Enguix. 2026. LATE-iimas at SemEval-2026 task 10: Conspiracy detection with ensemble models and loss optimization. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, California, United St… find
  40. Kongqiang Wang and Qingli Tan. 2026. wangkongqiang at SemEval-2026 task 10: Psy-CoMark – psycholinguistic conspiracy marker extraction and detection. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, California, United States. Association for Computa… find
  41. Yafei Zhang, Lin Wang, Jonathan J. H. Zhu, and Xiaofan Wang. 2021. Conspiracy vs science: A large-scale analysis of online discussion cascades. In World Wide Web, volume 24 of World Wide Web, pages 585– 606. find
  42. Yuhan Zheng and Yang Yang. 2026. NJUST_KMG at SemEval-2026 task 10: PsyCoMark – subtask 2: Conspiracy detection. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, California, United States. Association for Computational Linguistics. find
  43. Jiayue Zhu. 2026. Jia at SemEval-2026 task 10: A dual-track system with BERT-based encoders and LLMs for conspiracy analysis. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, California, United States. Association for Computational Linguistics. find
  44. Rojin Ziaei, Mahsa Khoshnoodi, and Nazli Goharian. 2026. GUNLP at SemEval-2026 task 10: Improving conspiracy detection via emotion-aware augmentation. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, California, United States. Association for Comput… find
  45. Zijun Wang and Guanyi Chen. 2026. CCNU at SemEval-2026 task 10: Conspiracy marker extraction and detection via multi-task learning and LLM-based data augmentation. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, California, United States. Associati… find
  46. Daichi Yamaga, Fumika Beppu, Yuki Shibata, Aiswariya Manoj Kumar, and Takayuki Hori. 2026. CuriosAI at SemEval-2026 task 10: Hybrid approaches to conspiracy span extraction and binary detection. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, Calif… find
  47. Shao Yi Liaw, Fan Huang, Fabricio Benevenuto, Haewoon Kwak, and Jisun An. 2023. Younicon: Youtube’s community of conspiracy videos. Proceedings of the International AAAI Conference on Web and Social Media, 17(1):1102–1111. find
  48. Peng Zhang and Gehao Lu. 2026. zhangpeng at SemEval-2026 task 10: PsyCoMark – psycholinguistic conspiracy marker extraction and detection. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, California, United States. Association for Computational Ling… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/samory-2026-semeval-2026-task-10-psycomark/