Elamur Research⌕
☰
home / papers / 2018 / SMHD: a Large-Scale Resource for Exploring Online Language Usage for Multiple Mental Health Conditions

SMHD: a Large-Scale Resource for Exploring Online Language Usage for Multiple Mental Health Conditions

Paper · 2018 · Arman Cohan, Bart Desmet, Andrew Yates et al.

Summary

Studying how mental health conditions show up in everyday language requires large labeled collections of users, but expert annotation is costly and hard to arrange. Cohan et al. address this with SMHD (Self-reported Mental Health Diagnoses), a Reddit-based dataset built without manual labelling. They designed high-precision patterns, partly expanded with synonyms, to find users who state that they received a diagnosis. The patterns cover nine conditions aligned with DSM-5 categories. Each diagnosed user is paired with control users chosen by exclusion criteria. Posts that explicitly discuss mental health are removed, so the data reflects ordinary conversation rather than obvious signals. The authors compare diagnosed and control groups on linguistic and psychological variables measured with LIWC. They also benchmark several classifiers in binary and multi-label settings, and supervised FastText performs best overall. Compared with an earlier Twitter resource, SMHD has as much as a hundredfold more diagnosed users for each condition. It also includes users with more than one condition. This makes it suitable for data-hungry models and for studying conditions that co-occur.

Key points

  • The dataset covers nine conditions: ADHD, anxiety, autism, bipolar, depression, eating disorders, OCD, PTSD and schizophrenia.
    • Six of these are top-level DSM-5 disorders.
    • The other three (PTSD, autism, ADHD) sit one level lower in the taxonomy.
  • Diagnosed-user counts range from 14,139 for depression and 10,098 for ADHD down to 598 for eating disorders. The comparable Twitter dataset by Coppersmith et al. (2015a) has only about 100–400 users per condition.
  • SMHD extends the approach behind RSDD (Yates et al., 2017) in two ways:
    • RSDD covered depression only, while SMHD covers multiple conditions.
    • SMHD uses a wider set of matching patterns.
  • The baselines are logistic regression, XGBoost, linear SVM, supervised FastText and a Kim (2014)-style CNN.
    • The first three use tf-idf bag-of-words features.
    • The CNN reads each user's concatenated posts, truncated to 15,000 tokens.
  • FastText achieves the top F1 for most conditions and in the multi-label setting. The exceptions are bipolar, where SVM wins, and eating disorders, where the CNN wins. All models struggle on the smallest classes (OCD, schizophrenia, eating).

Interesting ideas

  • I1. Weak labels from self-disclosure. Explicit first-person statements (e.g., "I was diagnosed with X") can be matched with strict, precision-oriented patterns to produce labels at scale. This could be reused wherever people openly state an attribute about themselves, such as profession, location or native language.
  • I2. Synonym-expanded patterns. Growing the patterns with synonyms taken from reliable sources raises recall and captures more linguistic variety without much loss of label quality. This is a cheap way to widen any rule-based data collection step.
  • I3. Removing the giveaway signal. Deleting posts that directly discuss the target topic forces models to learn from general language rather than obvious keywords. This makes evaluation more honest and applies to any task where labels leak through topical words.
  • I4. Diagnosing the precision–recall balance. The authors observe that bag-of-words models favor precision while neural models are more balanced. Reporting this trade-off, not just F1, helps reveal overfitting to a few strongly indicative features.
  • I5. Testing whether sampling hurts. The CNN sees only part of each user's history and underperforms the simpler FastText, which sees all posts. This suggests that subsampling for efficiency should be checked explicitly as a possible cause of weaker results.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Georgetown UniversityWashington · United States69 works · 4,417 citations🔬Max Planck Institute for InformaticsSaarbrücken · Germany11 works · 4,289 citations🎓Ghent UniversityGhent · Belgium18 works · 2,888 citations🔬Max Planck SocietyMunich · Germany10 works · 42,142 citations

📚 Shelves 7 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Psychology · 798 works Experimental and Cognitive Psychology · 463 works Social Psychology · 129 works

Cite

Cite as Arman Cohan, Bart Desmet, Andrew Yates, Luca Soldaini, Sean MacAvaney and Nazli Goharian. 2018. SMHD: a Large-Scale Resource for Exploring Online Language Usage for Multiple Mental Health Conditions. Proceedings of the 27th International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{cohan2018smhd,
  title = {{SMHD: a Large-Scale Resource for Exploring Online Language Usage for Multiple Mental Health Conditions}},
  author = {Arman Cohan, Bart Desmet, Andrew Yates, Luca Soldaini, Sean MacAvaney and Nazli Goharian},
  booktitle = {Proceedings of the 27th International Conference on Computational Linguistics},
  year = {2018},
  url = {https://aclanthology.org/C18-1126/}
}

References

50 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tim Althoff, Kevin Clark, and Jure Leskovec. 2016. Large-scale analysis of counseling conversations: An application of natural language processing to mental health. In TACL. our card
  2. American Psychiatric Association. 2013. Diagnostic and Statistical Manual of Mental Disorders (DSM-5). American Psychiatric Association Publishing. find
  3. Adrian Benton, Margaret Mitchell, and Dirk Hovy. 2017. Multitask learning for mental health conditions with limited social media data. In EACL. find
  4. Wilma Bucci and Norbert Freedman. 1981. The language of depression. Bulletin of the Menninger Clinic, 45(4):334. find
  5. David Carmel, Eitan Farchi, Yael Petruschka, and Aya Soffer. 2002. Automatic query refinement using lexical affinities with maximal information gain. In Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, pages 283–290. ACM. find
  6. Tianqi Chen and Carlos Guestrin. 2016. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785–794. ACM. find
  7. Arman Cohan, Sydney Young, Andrew Yates, and Nazli Goharian. 2017. Triaging content severity in online mental health forums. Journal of the Association for Information Science and Technology (JASIST). find
  8. Jacob Cohen. 1988. Statistical power analysis for the behavioral sciences 2nd edition. find
  9. Glen Coppersmith, Mark Dredze, and Craig Harman. 2014a. Quantifying mental health signals in Twitter. In Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, pages 51–60. our card
  10. Glen Coppersmith, Craig Harman, and Mark Dredze. 2014b. Measuring post traumatic stress disorder in Twitter. In ICWSM. find
  11. Glen Coppersmith, Mark Dredze, Craig Harman, and Kristy Hollingshead. 2015a. From ADHD to SAD: Analyzing the language of mental health on Twitter through self-reported diagnoses. In Proceedings of the 2nd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinic… find
  12. Glen Coppersmith, Mark Dredze, Craig Harman, Kristy Hollingshead, and Margaret Mitchell. 2015b. find
  13. CLPsych 2015 shared task: Depression and PTSD on Twitter. In CLPsych HLT-NAACL. our card
  14. Glen Coppersmith, Casey Hilland, Ophir Frieder, and Ryan Leary. 2017. Scalable mental health analysis in the clinical whitespace via natural language processing. In Biomedical & Health Informatics (BHI), 2017 IEEE EMBS International Conference on, pages 393–396. IEEE. find
  15. Munmun De Choudhury and Emre Kıcıman. 2017. The language of social support in social media and its effect on suicidal ideation risk. In Proceedings of the... International AAAI Conference on Weblogs and Social Media. International AAAI Conference on Weblogs and Social Media, volume 2017, page 32. NI… find
  16. Munmun De Choudhury, Scott Counts, and Eric Horvitz. 2013a. Major life changes and behavioral markers in social media: case of childbirth. In Proceedings of the 2013 conference on Computer supported cooperative work, pages 1431–1442. ACM. find
  17. Munmun De Choudhury, Michael Gamon, Scott Counts, and Eric Horvitz. 2013b. Predicting depression via social media. In ICWSM. find
  18. Bart Desmet and Veronique Hoste. 2018. Online suicide prevention through optimised text classification. Information Sciences, 439-440:61–78. find
  19. Sindhu Kiranmai Ernala, Asra F. Rizvi, Michael L. Birnbaum, John M. Kane, and Munmun De Choudhury. 2017. Linguistic markers indicating therapeutic outcomes of social media disclosures of schizophrenia. PACMHCI, 1:43:1–43:27. find
  20. Dirk Hovy and Shannon L Spruit. 2016. The social impact of natural language processing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 591–598. our card
  21. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759. find
  22. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In EMNLP. our card
  23. Rohan Kshirsagar, Robert Morris, and Sam Bowman. 2017. Detecting and explaining crisis. arXiv preprint arXiv:1705.09585. find
  24. David E. Losada and Fabio Crestani. 2016. A test collection for research on depression and language use. In CLEF. find
  25. Sean MacAvaney, Bart Desmet, Arman Cohan, Luca Soldaini, Andrew Yates, Ayah Zirikly, and Nazli Goharian. 2018. RSDD-Time: Temporal annotation of self-reported mental health diagnoses. In Proceedings of the Fifth Workshop on Computational Linguistics and Clinical Psychology: From Keyboard to Clinic,… find
  26. David N. Milne, Glen Pink, Ben Hachey, and Rafael Alejandro Calvo. 2016. find
  27. CLPsych 2016 shared task: Triaging content in online peer-support forums. In CLPsych@HLT-NAACL. find
  28. Margaret Mitchell, Kristy Hollingshead, and Glen Coppersmith. 2015. Quantifying the language of schizophrenia in social media. In Proceedings of the 2nd workshop on Computational linguistics and clinical psychology: From linguistic signal to clinical reality, pages 11–20. find
  29. Danielle L Mowery, Albert Park, Craig Bryan, and Mike Conway. 2016. Towards automatically classifying depressive symptoms from Twitter data for population health. In Proceedings of the Workshop on Computational Modeling of People’s Opinions, Personality, and Emotions in Social Media (PEOPLES), pages… find
  30. Danielle Mowery, Craig Bryan, and Mike Conway. 2017a. Feature studies to inform the classification of depressive symptoms from Twitter data for population health. arXiv preprint arXiv:1701.08229. — [arXiv:1701.08229](https://arxiv.org/abs/1701.08229) find
  31. Danielle Mowery, Hilary Smith, Tyler Cheney, Greg Stoddard, Glen Coppersmith, Craig Bryan, and Mike Conway. 2017b. Understanding depressive symptoms and psychosocial stressors on Twitter: a corpus-based study. Journal of medical Internet research, 19(2). find
  32. Jane M Murphy, Donald C Olivier, Richard R Monson, Arthur M Sobol, Elizabeth B Federman, and Alexander H Leighton. 1991. Depression and anxiety in relation to social status: A prospective epidemiologic study. Archives of General Psychiatry, 48(3):223–229. find
  33. Matthew L Newman, James W Pennebaker, Diane S Berry, and Jane M Richards. 2003. Lying words: Predicting deception from linguistic styles. Personality and social psychology bulletin, 29(5):665–675. find
  34. Umashanthi Pavalanathan and Munmun De Choudhury. 2015. Identity management and mental health discourse in social media. In Proceedings of the 24th International Conference on World Wide Web, WWW ’15 Companion, pages 315–321, New York, NY, USA. ACM. find
  35. James W Pennebaker, Ryan L Boyd, Kayla Jordan, and Kate Blackburn. 2015. The development and psychometric properties of LIWC2015. Technical report. find
  36. Daniel Preotiuc-Pietro, Johannes C. Eichstaedt, Gregory J. Park, Maarten Sap, Laura Smith, Victoria Tobolsky, H. Andrew Schwartz, and Lyle H. Ungar. 2015. The role of personality, age, and gender in tweeting about mental illness. In CLPsych HLT-NAACL. find
  37. Philip Resnik, Anderson Garron, and Rebecca Resnik. 2013. Using topic modeling to improve prediction of neuroticism and depression in college students. In EMNLP. find
  38. Philip Resnik, William Armstrong, Leonardo Claudino, and Thang Nguyen. 2015. The University of Maryland CLPsych 2015 shared task system. In CLPsych HLT-NAACL. find
  39. Tetsuya Sakai. 2016. Statistical significance, power, and sample sizes: A systematic review of SIGIR and TOIS, 2006-2015. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, pages 5–14. ACM. find
  40. Luca Soldaini and Elad Yom-Tov. 2017. Inferring individual attributes from search engine queries and auxiliary information. In Proceedings of the 26th International Conference on World Wide Web (WWW ’17), pages 293– 301. find
  41. Tara W Strine, Ali H Mokdad, Lina S Balluz, Olinda Gonzalez, Raquel Crider, Joyce T Berry, and Kurt Kroenke. 2008. Depression and anxiety in the United States: findings from the 2006 behavioral risk factor surveillance system. Psychiatric services, 59(12):1383–1390. find
  42. Simon Šuster, Stéphan Tulkens, and Walter Daelemans. 2017. A short review of ethical challenges in clinical natural language processing. arXiv preprint arXiv:1703.10090. find
  43. Sho Tsugawa, Yusuke Kikuchi, Fumio Kishino, Kosuke Nakajima, Yuichi Itoh, and Hiroyuki Ohsaki. 2015. Recognizing depression from Twitter activity. In CHI. find
  44. R Jay Turner, B Gail Frankel, and Deborah M Levin. 1983. Social support: Conceptualization, measurement, and implications for mental health. Research in community & mental health. find
  45. Rianne Van der Zanden, Keshia Curie, Monique Van Londen, Jeannet Kramer, Gerard Steen, and Pim Cuijpers. 2014. Web-based depression treatment: Associations of clients word use with adherence and outcome. Journal of Affective Disorders, 160:10–13. find
  46. E Watkins and RG Brown. 2002. Rumination and executive function in depression: An experimental study. Journal of Neurology, Neurosurgery & Psychiatry, 72(3):400–402. find
  47. Bernard L Welch. 1947. The generalization ofstudent’s’ problem when several different population variances are involved. Biometrika, 34(1/2):28–35. find
  48. Andrew Yates and Nazli Goharian. 2013. ADRTrace: detecting expected and unexpected adverse drug reactions from user reviews on social media sites. In European Conference on Information Retrieval, pages 816–819. Springer. find
  49. Andrew Yates, Arman Cohan, and Nazli Goharian. 2017. Depression and self-harm risk assessment in online forums. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2968–2978, Copenhagen, Denmark, September. Association for Computational Linguistics. our card
  50. Elad Yom-Tov and Evgeniy Gabrilovich. 2013. Postmarket drug surveillance without trial costs: discovery of adverse drug reactions through large-scale analysis of web search queries. Journal of medical Internet research, 15(6). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cohan-2018-smhd-large-scale-resource-exploring/