Elamur Research⌕
☰
home / papers / 2026 / MindSET: Advancing Mental Health Benchmarking through Large-Scale Social Media Data

MindSET: Advancing Mental Health Benchmarking through Large-Scale Social Media Data

Summary

Reddit-based benchmarks for detecting mental health conditions, such as SMHD and RSDD, have become hard to use. Their collection relied on Pushshift, which Reddit has since restricted, so the data can no longer be shared. They also contain noise: reposted content, NSFW material and non-English text. MindSET is a replacement resource built from Reddit activity between January 2018 and September 2024, gathered through the Arctic Shift API. The authors find diagnosed users with high-precision self-report patterns that separate explicit diagnosis statements from tentative self-suspicion. They then pair each diagnosed user with control users who post in similar ways. Explicit mental health content is removed so the detection task does not become trivial. Cleaning has three parts: language filtering to keep English, NSFW removal and deduplication. The authors also run a LIWC analysis across the user groups. In binary diagnosis detection, both fine-tuned BERT and XGBoost on Bag-of-Words features beat earlier SMHD baselines for every condition. MindSET therefore offers a larger, cleaner and legally shareable foundation for social-media mental health research.

Key points

  • The dataset holds over 13M annotated posts covering seven conditions: Depression, OCD, Bipolar, ADHD, PTSD, Autism and Anxiety. Together with controls, these form eight groups, and the corpus is more than twice the size of earlier benchmarks.
  • Diagnosis labels come from modified self-report patterns based on Coppersmith et al. (2018), with roughly 96.4% reported pattern accuracy. These labels serve as gold standard because no clinical ground truth is available.
  • Control selection lowered the minimum post count from SMHD's 50 to 35. This yields about 4–5 controls per diagnosed user, compared with about 9 in SMHD.
  • XGBoost on BoW features reaches F1 scores of 85–89. Its largest gain over the previous best result (Dinu and Moldovan, 2021) is +18 F1 for Autism, rising from 71 to 89. BERT trained on MindSET scores 75–80 F1.
  • The mental health subreddit exclusion list was extended with forums such as r/AnxietyRecovery and r/schizoaffective.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4cdunjq3bziz

@inproceedings{mankarious2026mindset,
  title = {{MindSET: Advancing Mental Health Benchmarking through Large-Scale Social Media Data}},
  author = {Mankarious, Saad and Kempa, Edward and Wiechmann, Daniel and Kerz, Elma and Qiao, Yu and Zirikly, Ayah},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4cdunjq3bziz},
  url = {https://doi.org/10.63317/4cdunjq3bziz}
}

References

48 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Timothy Baldwin and Marco Lui. 2010. Language identification: The long and the short of the matter. pages 229–237. find
  2. BSI. 1973a. Natural Fibre Twines, 3rd edition. British Standards Institution, London. BS 2570. find
  3. BSI. 1973b. Natural fibre twines. BS 2570, British Standards Institution, London. 3rd. edn. find
  4. Wilma Bucci and Norbert Freedman. 1981. The language of depression. Bulletin of the Menninger Clinic, 45(4):334. find
  5. A. Castor and L. E. Pollux. 1992. The use of user modelling to guide inference and learning. Applied Intelligence, 2(1):37–53. find
  6. Stevie Chancellor and Munmun De Choudhury. 2020. Methods in predictive techniques for mental health status on social media: a critical review. NPJ digital medicine, 3(1):43. find
  7. J.L. Chercheur. 1994. Case-Based Reasoning, 2nd edition. Morgan Kaufman Publishers, San Mateo, CA. find
  8. N. Chomsky. 1973. Conditions on transformations. In A festschrift for Morris Halle, New York. Holt, Rinehart & Winston. find
  9. Arman Cohan, Bart Desmet, Andrew Yates, Luca Soldaini, Sean MacAvaney, and Nazli Goharian. 2018. Smhd: A large-scale resource for exploring online language usage for multiple mental health conditions. In Proceedings of the 27th International Conference on Computational Linguistics (COLING), pages 14… find
  10. Jacob Cohen. 2013. Statistical power analysis for the behavioral sciences. routledge. find
  11. Glen Coppersmith, Mark Dredze, and Craig Harman. 2015. Extracting signals of mental health disorders from twitter. ACL Workshop on Computational Linguistics and Clinical Psychology. find
  12. Glen Coppersmith, Rachel Leary, Patrick Crutchley, and Adam Fine. 2018. The smhd: A large-scale resource for exploring online language use in mental health. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologi… find
  13. Anca Dinu and Andreea-Codrina Moldovan. 2021. Automatic detection and classification of mental illnesses from general social media texts. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 358–366. find
  14. Umberto Eco. 1990. The Limits of Interpretation. Indian University Press. find
  15. Seraphina Fong, Alessandro Carollo, Giacomo Vivanti, Daniel S Messinger, Dagmara Dimitriou, and Gianluca Esposito. 2025. Autism spectrum disorders discourse on social media platforms: A topic modeling study of reddit posts. Autism research. find
  16. Sharath Chandra Guntuku, Mark S. Sherman, Margaret K. Stokes, and Lyle H. Ungar. 2020. Tracking mental health and symptom mentions on twitter during covid-19. Journal of General Internal Medicine, 35(9):2798–2800. find
  17. Paul Gerhard Hoel. 1971a. Elementary Statistics, 3rd edition. Wiley series in probability and mathematical statistics. Wiley, New York, Chichester. ISBN 0 471 40300. find
  18. Paul Gerhard Hoel. 1971b. Elementary Statistics, 3rd edition, Wiley series in probability and mathematical statistics, pages 19–33. Wiley, New York, Chichester. ISBN 0 471 40300. find
  19. Mohammad M. Hossain, Shazzat A. Ehsan, Victor H. Santos, and Frank A. de Andrade. 2020. Covid-19 and mental health: A review of the existing literature on social media data. Journal of Medical Internet Research, 22(6):e19607. find
  20. Molly E Ireland and James W Pennebaker. 2010. Language style matching predicts relationship initiation and stability. Journal of Personality and Social Psychology, 99(4):587–604. find
  21. Kayla Jordan James W. Pennebaker, Ryan L. Boyd and Kate Blackburn. 2015. The Development and Psychometric Properties of LIWC2015. University of Texas at Austin. find
  22. Otto Jespersen. 1922. Language: Its Nature, Development, and Origin. Allen and Unwin. find
  23. Hannah Rose Kirk, Abeba Birhane, Bertie Vidgen, and Leon Derczynski. 2022. Handling and presenting harmful text in nlp research. arXiv preprint arXiv:2204.14256. find
  24. Jason Lee, Jane Doe, and Michael Smith. 2021. The impact of deduplication on nlp model robustness and performance. Transactions of the Association for Computational Linguistics, 9:123–135. find
  25. David E. Losada, Fabio Crestani, and Javier Parapar. 2016. erisk 2016: A pilot task on early risk prediction on the internet: Experimental foundations. In Proceedings of the 7th International Conference of the CLEF Association (CLEF 2016). Springer. find
  26. David N. Milne, Gari Pink, and Ben Hachey. 2016. find
  27. Clpsych 2016 shared task: Triaging content in online peer-support forums. In Proceedings of the Third Workshop on Computational Linguistics and Clinical Psychology (CLPsych), pages 118– 127. find
  28. Jane M Murphy, Donald C Olivier, Richard R Monson, Arthur M Sobol, Elizabeth B Federman, and Alexander H Leighton. 1991. Depression and anxiety in relation to social status: A prospective epidemiologic study. Archives of General Psychiatry, 48(3):223–229. find
  29. Alexandra Olteanu, Carlos Castillo, Fernando Diaz, and Emre Kıcıman. 2019. Social data: Biases, methodological pitfalls, and ethical boundaries. Frontiers in big data, 2:13. find
  30. Minsu Park, David W. McDonald, and Meeyoung Cha. 2013. Examining user behaviors on social media for mental health status. In Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems, pages 3207–3216. ACM. find
  31. James W Pennebaker, Matthias R Mehl, and Kate G Niederhoffer. 2003. Psychological aspects of natural language use: Our words, our selves. Annual Review of Psychology, 54(1):547– 577. find
  32. Andrew G. Reece and Christopher M. Danforth. 2017. Instagram photos reveal predictive markers of bipolar disorder. EPJ Data Science, 6(1):1– 12. find
  33. Thomas Rodenberg and Jessica Davis. 2021. The impact of covid-19 on mental health: A comprehensive review. Journal of Social Psychiatry, 65(3):220–231. find
  34. Han-Chin Shing, Suraj Nair, Ayah Zirikly, Meir Friedenberg, Hal Daumé III, and Philip Resnik. 2018. Expert, crowdsourced, and machine assessment of suicide risk via online postings. In Proceedings of the Fifth Workshop on Computational Linguistics and Clinical Psychology: From Keyboard to Clinic, pa… find
  35. Charles Joseph Singer, E. J. Holmyard, and A. R. Hall, editors. 1954–58. A history of technology. Oxford University Press, London. 5 vol. find
  36. Jannik Strötgen and Michael Gertz. 2012. Temporal tagging on different domains: Challenges, strategies, and gold standards. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), pages 3746– 3753, Istanbul, Turkey. European Language Resource Association… find our card
  37. S. Superman, B. Batman, C. Catwoman, and S. Spiderman. 2000. Superheroes experiences with books, 20th edition. The Phantom Editors Associates, Gotham City. find
  38. Allison M Tackman, David A Sbarra, Alexander L Carey, et al. 2020. The utility of self-referential language in understanding personality, psychopathology, and mental health. Journal of Research in Personality, 85:103917. find
  39. Author Unknown. 2023. Problematic social internet use and associations with adhd symptoms in adolescents during covid-19. Oxford Academic. Accessed: 2024-12-03. find
  40. Diana Valdez, Emily E. Rhodes, Melissa S. Garcia, and Sophia S. Paredes. 2020. Social media insights into mental health recovery post-pandemic. Computers in Human Behavior, 120:106759. find
  41. Rianne Van der Zanden, Keshia Curie, Monique Van Londen, Jeannet Kramer, Gerard Steen, and Pim Cuijpers. 2014. Web-based depression treatment: Associations of clients word use with adherence and outcome. Journal of affective disorders, 160:10–13. find
  42. Edward Watkins and RG Brown. 2002. Rumination and executive function in depression: An experimental study. Journal of Neurology, Neurosurgery & Psychiatry, 72(3):400–402. find
  43. Bernard L Welch. 1947. The generalization of ‘student’s’problem when several different population varlances are involved. Biometrika, 34(1-2):28– 35. find
  44. Andrew Yates, Arman Cohan, and Nazli Goharian. 2017a. Depression and self-harm risk assessment in online forums. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2968–2978. ACL. find
  45. Andrew Yates, Arman Cohan, and Nazli Goharian. 2017b. Depression and self-harm risk assessment in online forums. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2968–2978. find
  46. Sourabh Zanwar, Daniel Wiechmann, Yu Qiao, and Elma Kerz. 2023. SMHD-GER: A large-scale benchmark dataset for automatic mental health detection from social media in German. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1526–1541, Dubrovnik, Croatia. Association for C… find
  47. Ayah Zirikly, Philip Resnik, Özlem Uzuner, and Kristy Hollingshead. 2019. find
  48. CLPsych 2019 shared task: Predicting the degree of suicide risk in Reddit posts. In Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology, pages 24–33, Minneapolis, Minnesota. Association for Computational Linguistics. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/mankarious-2026-mindset-advancing-mental-health-benchmarking/