Elamur Research⌕
☰
home / papers / 2026 / Automatic Annotation of Mental Health Recovery Narratives: A Benchmark Study

Automatic Annotation of Mental Health Recovery Narratives: A Benchmark Study

Paper · 2026 · Shrankhla Pandey, Graham Murray, Ben Laws et al.

Summary

Characterising mental health recovery narratives with the INCRESE framework makes them searchable and safer for readers. Doing this by hand is costly, slow and emotionally taxing for coders, and no automated alternative existed. The authors built the first benchmark for automatic INCRESE annotation, using 355 text narratives from the NEON Collection. They treated each of 67 characteristics (INCRESE sections 3–7) as a separate supervised classification problem. Support vector classifiers were trained on three families of text representations: bag-of-words, GloVe static embeddings and BERT-family contextual embeddings. Class weighting addressed heavy label imbalance, and cross-validation was grouped by narrative source.

Diagnosis and turning-point characteristics were predicted reasonably well. Content warnings reached decent balanced accuracy but missed many positive cases. This is risky because readers could unexpectedly encounter distressing material. Transformer embeddings did not clearly beat simpler representations. A lived-experience advisory group supported the goals and raised prioritisation issues that the metrics alone did not reveal. The study sets baselines and shows that metric choice should reflect how costly each type of error is.

Key points

  • Data and labels:
    • The data are 355 NEON Collection narratives (357 before removing two length outliers), with a mean length of about 1,202 words and a range of 86–3,387.
    • INCRESE defines 77 characteristics, of which 67 were modelled.
    • The data were split 267/88 for train and test.
  • Representations: CountVectorizer, TfIdfVectorizer, GloVe50d, GloVe300d, BERT-base, ModernBERT and SentenceBERT.
  • Classifiers and validation:
    • Logistic regression and random forest performed similarly to SVC, so only SVC results are reported.
    • Cross-validation was five-fold, stratified and group-aware by source.
  • Results:
    • Diagnosis and turning-point characteristics exceeded 0.67 balanced accuracy.
    • Content warnings reached 0.72 balanced accuracy but had low recall.
    • Six characteristics scored above 0.8: four diagnosis, one content-warning and one turning-point characteristic.
  • Context and limitations:
    • Manual annotation for NEON cost an estimated £21,840.
    • The dataset cannot be released publicly.
    • Models used text only, whereas human coders also saw images.
    • Pooling BERT outputs into fixed-length vectors may have weakened the signal.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom243 works · 121,396 citations🎓University of NottinghamNottingham · United Kingdom3 works · 17 citations

Also on the byline: Nord University.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksSocial Psychology · 306 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Shrankhla Pandey, Graham Murray, Ben Laws, Stefan Rennick-Egglestone, Mike Slade and Sarah Morgan. 2026. Automatic Annotation of Mental Health Recovery Narratives: A Benchmark Study. Proceedings of the 11th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.clpsych-1.8

@inproceedings{pandey2026automatic,
  title = {{Automatic Annotation of Mental Health Recovery Narratives: A Benchmark Study}},
  author = {Shrankhla Pandey, Graham Murray, Ben Laws, Stefan Rennick-Egglestone, Mike Slade and Sarah Morgan},
  booktitle = {Proceedings of the 11th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.clpsych-1.8},
  url = {https://doi.org/10.18653/v1/2026.clpsych-1.8}
}

References

39 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yasmin Ali, Stefan Rennick-Egglestone, Joy Llewellyn-Beardsley, Fiona Ng, Caroline Yeo, Donna Franklin, Elvira Perez Vallejos, Dror Ben-Zeev, Yasuhiro Kotera, and Mike Slade. 2024. Perception and appropriation of a web-based recovery narratives intervention: qualitative interview study. Frontiers in… find
  2. Suyoung Bae, Gunhee Cho, Yun-Gyung Cheong, and Boyang Li. 2025. CharMoral: A character morality dataset for morally dynamic character analysis in long-form narratives. In Proceedings of the 31st International Conference on Computational Linguistics, pages 8809–8818, Abu Dhabi, UAE. Association for C… find
  3. Michele Banko and Eric Brill. 2001. Scaling to very very large corpora for natural language disambiguation. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics, pages 26–33, Toulouse, France. Association for Computational Linguistics.
  4. Ulya Bayram and Lamia Benhiba. 2022. Emotionally-informed models for detecting moments of change and suicide risk levels in longitudinal social media data. In Proceedings of the Eighth Workshop on Computational Linguistics and Clinical Psychology, pages 219–225, Seattle, USA. Association for Computa… find
  5. Ana-Maria Bucur, Hyewon Jang, and Farhana Ferdousi Liza. 2022. Capturing changes in mood over time in longitudinal data using ensemble methodologies. In Proceedings of the Eighth Workshop on Computational Linguistics and Clinical Psychology, pages 205–212, Seattle, USA. Association for Computational… find
  6. Callum Chan, Sunveer Khunkhun, Diana Inkpen, and Juan Antonio Lossio-Ventura. 2025. Prompt engineering for capturing dynamic mental health self states from social media posts. In Proceedings of the 10th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2025), pages 256–267, Albu… find
  7. Glen Coppersmith, Mark Dredze, Craig Harman, Kristy Hollingshead, and Margaret Mitchell. 2015. find
  8. CLPsych 2015 shared task: Depression and PTSD on Twitter. In Proceedings of the 2nd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, pages 31– 39, Denver, Colorado. Association for Computational Linguistics.
  9. DATAMIND. 2025a. Datamind: Better mental health through data-driven research. https://datamind.org.uk/. Accessed: 2026-01-22. ↗
  10. DATAMIND. 2025b. Lived experience advisory group. https://datamind.org.uk/patients-and-public/ lived-experience-advisory-group/. Accessed: 2026-01-22. ↗
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  12. Keith Harrigian, Carlos Aguirre, and Mark Dredze. 2021. On the state of social media data for mental health research. In Proceedings of the Seventh Workshop on Computational Linguistics and Clinical Psychology: Improving Access, pages 15–24, Online. Association for Computational Linguistics. find
  13. Haibo He and Edwardo A. Garcia. 2009. Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering, 21(9):1263–1284. find
  14. Joy Llewellyn-Beardsley, Yasmin Ali, Sylvia Bailey, Susie Booth, Paul Davis, Caroline Fox-Yeo, Donna Franklin, Julian Harrison, David King, Chris Newby, Fiona Ng, Scott Pomberth, Stefan Rennick-Egglestone, Dan Robotham, Roger Smith, Rianna Walcott, and Mike Slade. 2025. Lived experience narratives f… find
  15. Joy Llewellyn-Beardsley, Skye Barbic, Stefan Rennick-Egglestone, Fiona Ng, James Roe, Ada Hui, Donna Franklin, Emilia Deakin, Laurie Hare-Duke, and Mike Slade. 2020. INCRESE: Development of an inventory to characterize recorded mental health recovery narratives. J. Recovery Ment. Health, 3(2):25– 44… find
  16. Sean MacAvaney, Anjali Mittu, Glen Coppersmith, Jeff Leintz, and Philip Resnik. 2021. Community-level research on suicidality prediction in a secure environment: Overview of the CLPsych 2021 shared task. In Proceedings of the Seventh Workshop on Computational Linguistics and Clinical Psychology: Imp… find
  17. NEON Collection. Neon collection. https://www.researchintorecovery.com/research/ neon/neoncollection/. ↗
  18. Luke Paterson, Stefan Rennick-Egglestone, Sean P. Gavan, Mike Slade, Fiona Ng, Joy Llewellyn-Beardsley, Carmel Bond, Andrew Grundy, Joe Nicholson, Dania Quadri, Sylvia Bailey, and Rachel A. Elliott. 2022. Development and delivery cost of digital health technologies for mental health: Application to… find
  19. Bronwin Patrickson, Mike Musker, Dan Thorpe, Yasmin van Kasteren, Niranjan Bidargaddi, The Consumer, and Carer Advisory Group (CCAG). 2023. In-depth co-design of mental health monitoring technologies by people with lived experience. Future Internet, 15(6). find
  20. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, and et al. 2011. Scikit-learn: Machine learning in python. Journal of Machine Learning Research, 12:2825–2830. find
  21. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543.
  22. Research Into Recovery. 2025. Increse. find
  23. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
  24. Research Into Recovery. a. Increse – inventory of characteristics of recovery stories: measure to characterise mental health recovery narratives. https://www.researchintorecovery.com/measures/increse/. Research Into Recovery. b. Neon definitions: Key terms used in the narrative experiences online (… ↗
  25. Fabian Schmidt, Karin Hammerfald, Henrik Haaland Jahren, and Vladimir Vlassov. 2025. CFiCS: Graph-based classification of common factors and microcounseling skills. In Proceedings of the 10th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2025), pages 106–115, Albuquerque, Ne… find
  26. Sagarika Shreevastava and Peter Foltz. 2021. Detecting cognitive distortions from patient-therapist interactions. In Proceedings of the Seventh Workshop on Computational Linguistics and Clinical Psychology: Improving Access, pages 151–158, Online. Association for Computational Linguistics. find
  27. Mike Slade, Stefan Rennick-Egglestone, Joy Llewellyn-Beardsley, Caroline Yeo, James Roe, Sylvia Bailey, Roger Andrew Smith, Susie Booth, Julian Harrison, Adaresh Bhogal, Patricia Penas Morán, Ada Hui, Dania Quadri, Clare Robinson, Melanie Smuk, Marianne Farkas, Larry Davidson, Lian van der Krieke, E… find
  28. Mike Slade, Stefan Rennick-Egglestone, Felix Ng, Jenny Yiend, Nicola Jones, Eleanor Longden, Victoria J. Bird, Mary Leamy, Jesper Larsen, Shaz Ali, Larry Davidson, Adam Bell, and Sarah Byford. 2024. Effectiveness and cost-effectiveness of providing recorded mental health recovery narratives to peopl… find
  29. Miriah Steiger, Timir J. Bharucha, Sukrit Venkatagiri, Martin J. Riedl, and Matthew Lease. 2021. The psychological well-being of content moderators: The emotional labor of commercial moderation and avenues for improving support. In Proceedings of the 2021 CHI Conference on Human Factors in Computing… find
  30. Alexandra Thompson, Victoria Bartle, Elizabeth A Rem-fry, Duncan J Reynolds, Michael R Barnes, Nick J Reynolds, and Barbara Hanratty. 2025. Public and patient involvement in artificial intelligence and big data healthcare research: An exploration of issues and challenges within the AI-Multiply proje… find
  31. Linguistics and Clinical Psychology, pages 177–195, Abu Dhabi, UAE. Association for Computational Linguistics. find
  32. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Thomas Adams, Jeremy Howard, and Iacopo Poli. 2025. Smarter, better, faster, longer: A modern bidirectional encoder for…
  33. Caroline Yeo, Stefan Rennick-Egglestone, Victoria Armstrong, Marit Borg, Donna Franklin, Trude Kl-evan, Joy Llewellyn-Beardsley, Christopher Newby, Fiona Ng, Naomi Thorpe, Jijian Voronka, and Mike Slade. 2021. Uses and misuses of recorded mental health lived experience narratives in healthcare and c… find
  34. Swamy Rakshith Zanwar, Daniel Wiechmann, Yuan Qiao, and Elma Kerz. 2023. SMHD-GER: A large-scale benchmark dataset for automatic mental health detection from social media in German. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1538–1557, Dubrovnik, Croatia. Associat… find
  35. Ayah Zirikly, Philip Resnik, Özlem Uzuner, and Kristy Hollingshead. 2019. find
  36. CLPsych 2019 shared task: Predicting the degree of suicide risk in Reddit posts. In Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology, pages 24–33, Minneapolis, Minnesota. Association for Computational Linguistics.
  37. Yufei Tian, Tenghao Huang, Miri Liu, Derek Jiang, Alexander Spangher, Muhao Chen, Jonathan May, and Nanyun Peng. 2024. Are large language models capable of generating human-level narratives? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17659–17681,…
  38. Adam Tsakalidis, Jenny Chim, Iman Munire Bilal, Ayah Zirikly, Dana Atzil-Slonim, Federico Nanni, Philip Resnik, Manas Gaur, Kaushik Roy, Becky Inkster, Jeff Leintz, and Maria Liakata. 2022. Overview of the CLPsych 2022 shared task: Capturing moments of change in longitudinal user posts. In Proceedin… find
  39. Athanasia Tseriotou, Nikolaos Douzas, Molly Ireland, Ayah Zirikly, Dongji Yoo, Carlos Aguirre, Keith Harrigian, and Mark Dredze. 2025. Overview of the CLPsych 2025 shared task: Capturing mental health dynamics from social media timelines. In Proceedings of the Eleventh Workshop on Computational *Fig… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pandey-2026-automatic-annotation-mental-health-recovery/