Elamur Research⌕
☰
home / papers / 2026 / AAA at SemEval-2026 Task 10: Psycholinguistic Conspiracy Marker Extraction and Detection

AAA at SemEval-2026 Task 10: Psycholinguistic Conspiracy Marker Extraction and Detection

Paper · 2026 · Xintong Pan

Summary

This system paper addresses SemEval-2026 Task 10, which has two parts. The first is token-level tagging of psycholinguistic conspiracy markers: Actor, Action, Effect, Evidence and Victim. The second is binary detection of whether a social media text conveys a conspiracy narrative. Conspiratorial content often relies on insinuation and implied causal links rather than plainly false claims, so the author argues that surface semantics alone may miss it.

For detection, the system pairs a DeBERTa-v3 encoder with ConspEmoLLM-v2, an emotion-aware LLM. The LLM writes an explanatory rationale for each input. Vectors from the original text and from that rationale are then fused and passed to a light supervised classifier. For extraction, standard pretrained encoders were simply compared as sequence labelers, with no architectural changes.

Using the Reddit-based PsyCo-Mark data, the hybrid detector reached an F1 of 0.849, compared with about 0.75 for encoder-only baselines. This suggests that LLM-generated reasoning can help classifiers pick up implicit conspiratorial framing. Extraction results differed little across encoders and serve mainly as a reproducible baseline.

Key points

  • Dataset: PsyCo-Mark contains more than 4,100 Reddit comments with roughly 4,800 annotations, drawn from over 190 subreddits. Samples labeled "Can't tell" were removed before training.
  • Encoder comparison for detection: DistilBERT, BERT-base, RoBERTa-base and DeBERTa-v3-base were trained for 15 epochs (batch size 16, learning rate 2×10⁻⁴, Adam). Their F1 scores fell between 0.745 and 0.755, with DeBERTa highest.
  • Hybrid model results: combining DeBERTa-v3 with ConspEmoLLM-v2 rationales gave F1 0.849, accuracy 0.845, precision 0.853 and recall 0.845.
  • Data augmentation: EDA-style synonym replacement using WordNet added about 2,000 samples but did not improve test performance.
  • Extraction findings: token accuracy was high for all models, but F1 was much lower. The author attributes this gap to class imbalance, since the many unlabeled tokens inflate accuracy. The paper lists two limitations: possible bias in the LLM-generated rationales, and no multilingual testing yet.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TübingenTübingen · Germany75 works · 41,081 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Psycholinguistics & language acquisition · 102 works

Cite

Cite as Xintong Pan. 2026. AAA at SemEval-2026 Task 10: Psycholinguistic Conspiracy Marker Extraction and Detection. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.250

@inproceedings{pan2026aaa,
  title = {{AAA at SemEval-2026 Task 10: Psycholinguistic Conspiracy Marker Extraction and Detection}},
  author = {Xintong Pan},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.250},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.250}
}

References

13 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  2. Category Model Accuracy F1 Precision Recall Action RoBERTa DeBERTa BERT DistilBERT 0.936 0.938 0.934 0.931 0.368 0.372 0.383 0.346 0.480 0.491 0.480 0.443 0.298 0.299 0.319 0.287 Actor RoBERTa DeBERTa BERT DistilBERT 0.923 0.927 0.915 0.921 0.642 0.666 0.629 0.649 0.711 0.740 0.658 0.691 0.586 0.606… find
  3. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Wei Chen. 2021. Deberta: Decoding-enhanced bert with disentangled attention. In 2021 International Conference on Learning Representations.
  4. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In ICML 2022 Workshop on Knowledge Retrieval and Language Models.
  5. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692.
  6. Zhiwei Liu, Boyang Liu, Paul Thompson, Kailai Yang, and Sophia Ananiadou. 2024. ConspEmoLLM: Conspiracy Theory Detection Using an Emotion-Based Large Language Model. IOS Press. find
  7. Zhiwei Liu, Paul Thompson, Jiaqi Rong, and Sophia Ananiadou. 2025. ConspEmoLLM-v2: A Robust and Stable Model to Detect Sentiment-Transformed Conspiracy Theories. IOS Press. find
  8. Milena Pustet, Elisabeth Steffen, and Helena Mihaljevic. 2024. Detection of conspiracy theories beyond keyword bias in german-language telegram using large language models. In Proceedings of the 8th Workshop on Online Abuse and Harms (WOAH 2024), page 13–27. Association for Computational Linguistics… find
  9. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. In NeurIPS 2019 Workshop on Energy Efficient Machine Learning and Cognitive Computing. find
  10. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, R…
  11. Jason Wei and Kai Zou. 2019. EDA: Easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP…
  12. Orion Weller, Kathryn Ricci, Marc Marone, Antoine Chaffin, Dawn Lawrie, and Benjamin Van Durme. 2025. Seq vs seq: An open suite of paired encoders and decoders. arXiv:2507.11412. — [arXiv:2507.11412](https://arxiv.org/abs/2507.11412) find
  13. Zecharias Zelalem and Peter Guest. 2021. Why facebook keeps failing in ethiopia. Accessed: 2026-02-27. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pan-2026-aaa-semeval-2026-task-10/