Elamur Research⌕
☰
home / papers / 2026 / Towards Consistent Detection of Cognitive Distortions: LLM-Based Annotation and Dataset-Agnostic Evaluation

Towards Consistent Detection of Cognitive Distortions: LLM-Based Annotation and Dataset-Agnostic Evaluation

Summary

Detecting cognitive distortions (CDs) in text is difficult because the task is subjective. Even expert annotators agree poorly, so existing training labels are noisy, and reported classifier scores remain low. The authors re-annotate the public Therapist Q&A dataset with GPT models, running several independent annotation passes per text. They keep only labels that recur across nearly all runs and treat this stability as evidence of reliability. They also propose a dataset-agnostic evaluation that treats Cohen's kappa as an effect size, scoring models relative to a chance baseline instead of relying on raw F1. This allows fair comparison across datasets and studies with different label distributions. GPT-4 produced highly consistent labels across runs. Classifiers trained on the LLM labels beat those trained on the original human labels under the new metric. A check by domain experts, however, showed no clear preference between the two label sources. The work offers a scalable route to internally consistent training data for subjective NLP tasks. It also shows that texts describing events rather than thoughts may be inherently unsuitable for CD detection.

Key points

  • The data is the Therapist Q&A dataset (Kaggle): 2530 user input–response pairs labeled by Shreevastava and Foltz (2021). It uses 10 CD categories plus "No Distortion", and the original inter-annotator agreement was only 33.7% (joint probability).
  • Annotation used GPT-4 and GPT-4o via Microsoft Azure at temperatures 0.5 and 0.7. Two prompt types were tested: a Multi-Label Prompt (MLP) and a Ranked-Label Prompt (RLP) that selects one dominant CD plus an optional secondary one.
  • GPT-4 reached an inter-run agreement of Fleiss' kappa = 0.78. Earlier CD classifiers typically scored a weighted F1 of only about 0.2–0.4.
  • The proposed kappa-based effect size (κF1) is presented as transferable to any field with unstable benchmarks.
  • Limitations: only GPT-family models were tested, the experts disagreed among themselves, and the provenance of the dataset is unverifiable. The code and the LLM-generated labels are released on GitHub.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4joqxtgdgxq2

@inproceedings{sharma2026consistent,
  title = {{Towards Consistent Detection of Cognitive Distortions: LLM-Based Annotation and Dataset-Agnostic Evaluation}},
  author = {Sharma, Neha and Agarwal, Navneet and Sirts, Kairit},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4joqxtgdgxq2},
  url = {https://doi.org/10.63317/4joqxtgdgxq2}
}

References

29 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohammed Aldeen, Joshua Luo, Ashley Lian, Venus Zheng, Allen Hong, Preethika Yetukuri, and Long Cheng. 2023. Chatgpt vs. human annotators: A comprehensive analysis of chatgpt for text annotation. In 2023 International Conference on Machine Learning and Applications (ICMLA), pages 602–609. find
  2. Jelly P. Aureus, Ma. Regina Justina E. Estuar, Dorothy C. Mapua, Roland P. Abao, and Anna Angeline M. Cataluña. 2021. Determining linguistic markers in cognitive distortions from covid-19 pandemic-related reddit texts. In 2021 1st International Conference in Information and Computing Research (iCORE… find
  3. Aaron T Beck. 1963. Thinking and depression: I. idiosyncratic content and cognitive distortions. Archives of general psychiatry, 9(4):324–333. find
  4. Aaron T Beck and Brad A Alford. 2009. Depression: Causes and treatment. University of Pennsylvania Press. find
  5. David D Burns. 1989. The feeling good handbook: Using the new mood therapy in everyday life. William Morrow & Co. find
  6. David D Burns and MD Feeling Good. 1980. The new mood therapy. 6 https://github.com/nehasharma666/llm-cognitive-distortion-detection ↗
  7. Zhiyu Chen, Yujie Lu, and William Wang. 2023. Empowering psychotherapy with large language models: Cognitive distortion detection through diagnosis of thought prompting. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4295–4304, Singapore. Association for Computationa… find
  8. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. find
  9. Xiruo Ding, Kevin Lybarger, Justin Tauscher, and Trevor Cohen. 2022. Improving classification of infrequent cognitive distortions: Domain-specific model vs. data augmentation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human… find
  10. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. find
  11. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30):e2305016120. find our card
  12. Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2024. Annollm: Making large language models to be better crowdsourced annotators. find our card
  13. Peter Henderson and Emma Brunskill. 2018. Distilling information from a flood: A possibility for the use of meta-analysis and systematic review in machine learning research. CoRR, abs/1812.01074. — [arXiv:1812.01074](https://arxiv.org/abs/1812.01074) find
  14. Michael Heseltine and Bernhard Clemm von Hohenberg. 2024. Large language models as a substitute for human experts in annotating political text. Research & Politics, 11(1):20531680241236239. find
  15. Shaoxiong Ji, Tianlin Zhang, Luna Ansari, Jie Fu, Prayag Tiwari, and Erik Cambria. 2022. Mental-BERT: Publicly Available Pretrained Language Models for Mental Healthcare. find
  16. Jutta Joormann and Colin H Stanton. 2016. Examining emotion regulation in depression: A review and future directions. Behaviour research and therapy, 86:35–49. find
  17. Ken Kelley and Kristopher J Preacher. 2012. On effect size. Psychological methods, 17(2):137. find
  18. Jiyi Li. 2024. A comparative study on annotation quality of crowdsourcing and llm via label aggregation. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6525–6529. find
  19. Sehee Lim, Yejin Kim, Chi-Hyun Choi, Jy-yong Sohn, and Byung-Hoon Kim. 2024. Erd: a framework for improving llm reasoning for cognitive distortion classification. find
  20. Kevin Lybarger, Justin Tauscher, Xiruo Ding, Dror Ben-zeev, and Trevor Cohen. 2022. Identifying distorted thinking in patient-therapist text message exchanges by leveraging dynamic multi-turn context. In Proceedings of the Eighth Workshop on Computational Linguistics and Clinical Psychology, pages 1… find
  21. Nawal Ouhmad, Romain Deperrois, Wissam El Hage, and Nicolas Combalbert. 2024. Cognitive distortions, anxiety, and depression in individuals suffering from ptsd. International Journal of Mental Health, 53(4):336–352. find
  22. Lina Rojas-Barahona, Bo-Hsiang Tseng, Yinpei Dai, Clare Mansfield, Osman Ramadan, Stefan Ultes, Michael Crawford, and Milica Gasic. 2018. Deep learning for language understanding of mental health concepts derived from cognitive behavioural therapy. find
  23. Benjamin Shickel, Scott Siegel, Martin Heesacker, Sherry Benton, and Parisa Rashidi. 2020. Automatic detection and classification of cognitive distortions in mental health text. In 2020 IEEE 20th International Conference on Bioinformatics and Bioengineering (BIBE), pages 275–280. find
  24. Sagarika Shreevastava and Peter Foltz. 2021. Detecting cognitive distortions from patient-therapist interactions. In Proceedings of the Seventh Workshop on Computational Linguistics and Clinical Psychology: Improving Access, pages 151–158, Online. Association for Computational Linguistics. find
  25. T. Simms, C. Ramstedt, M. Rich, M. Richards, T. Martinez, and C. Giraud-Carrier. 2017. Detecting cognitive distortions through machine learning text analytics. In 2017 IEEE International Conference on Healthcare Informatics (ICHI), pages 508–512. find
  26. Justin S Tauscher, Kevin Lybarger, Xiruo Ding, Ayesha Chander, William J Hudenko, Trevor Cohen, and Dror Ben-Zeev. 2023. Automated detection of cognitive distortions in text exchanges between clinicians and people with serious mental illness. Psychiatric services, 74(4):407–410. find
  27. Bichen Wang, Pengfei Deng, Yanyan Zhao, and Bing Qin. 2023. C2D2 dataset: A resource for the cognitive distortion analysis and its impact on mental health. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10149–10160, Singapore. Association for Computational Linguistic… find
  28. Gülin Yazici-Çelebi and Feridun Kaya. 2022. Interpersonal cognitive distortions and anxiety: The mediating role of emotional intelligence. International Journal of Psychology and Educational Studies, 9(3):741–753. find
  29. Ruoyu Zhang, Yanzeng Li, Yongliang Ma, Ming Zhou, and Lei Zou. 2023. Llmaaa: Making large language models as active annotators. find

↑ back to top

This page in Markdown →