Elamur Research⌕
☰
home / papers / 2017 / Depression and Self-Harm Risk Assessment in Online Forums

Depression and Self-Harm Risk Assessment in Online Forums

Summary

The paper addresses two linked mental-health tasks: spotting Reddit users who are likely depressed from their writing alone, and estimating how urgently a post on a support forum signals self-harm risk. The authors build RSDD (Reddit Self-reported Depression Diagnosis), a large corpus where users claiming a depression diagnosis are verified by human annotators. Each verified user is paired with many control users who have similar subreddit interests and posting volume. They also reuse the CLPsych 2016 Triage data from ReachOut.com, which has four risk levels. Their model is a CNN that encodes each post separately and then merges the post encodings into one user-level representation. It uses no hand-crafted resources such as LIWC, topic models, or lexicons. For the self-harm task they compare several loss functions, including ones that account for the ordered nature of risk labels. On both tasks the approach beats strong earlier systems. This suggests that text-only neural models can support moderators and large-scale public-health research.

Key points

  • RSDD contains 9,210 diagnosed users and 107,274 controls. Users average 969 posts (median 646), and posts average 148 tokens (median 74).
  • To enter RSDD, a candidate needed at least 100 posts before the diagnosis post and at least two of three layperson annotators confirming a genuine diagnosis claim. This filtered out hypotheticals, negations, and quoted speech.
  • Controls never posted in mental-health subreddits or used mental-health terms. Diagnosed users had such posts removed, so the task cannot be solved through obvious keywords.
  • On the CLPsych '16 Triage test set (65,024 ReachOut posts, 1,227 labeled green/amber/red/crisis), the models improved over the best prior systems:
    • up to 19% on non-green F1 (the official metric),
    • up to 11% on macro F1,
    • up to 5% on accuracy.
  • Categorical cross-entropy scored best on the test set. The ordinal-margin Class Metric loss was statistically tied with it and stayed strong in 10-fold cross-validation on the training set, making it the most robust variant. Training-set non-green F1 reached 0.87 versus 0.50 on test, which hints at a distribution shift.

Interesting ideas

  • I1. Matching controls on behaviour, not just sampling them. Each positive user is paired with 12 controls by Hellinger distance between subreddit-posting distributions, with post counts restricted to within ±10%. The same trick helps any author-profiling dataset avoid classifiers that learn topic or activity level instead of the target trait.
  • I2. Removing label-leaking content from positives. Positive users' posts that contain the obvious signal (mental-health terms or communities) are deleted. This forces the model to find indirect linguistic cues, a useful design choice wherever labels come from self-disclosure.
  • I3. Post-level encoder plus merge layer for user classification. Each document is encoded independently, then pooled into one profile. This scales naturally to users with hundreds of posts and transfers to tasks like author attribution or user-level stance detection.
  • I4. Ordinal-aware loss as a robustness choice. A margin loss that respects label order was not the top scorer, but it performed consistently on both test and cross-validation. That makes it worth trying for graded labels such as severity, toxicity levels, or ratings.
  • I5. Privacy-preserving error analysis. The authors found that even paraphrased posts could be traced through a web search. They therefore reported only the single highest-activation phrase per user, extracted via convolutional filter weights. This is a practical model for explaining classifiers trained on sensitive data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

78 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 73 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d17-1322

@inproceedings{yates2017depression,
  title = {{Depression and Self-Harm Risk Assessment in Online Forums}},
  author = {Yates, Andrew and Cohan, Arman and Goharian, Nazli},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1322},
  url = {https://doi.org/10.18653/v1/d17-1322}
}

References

51 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tim Althoff, Kevin Clark, and Jure Leskovec. 2016. Large-scale analysis of counseling conversations: An application of natural language processing to mental health. arXiv preprint arXiv:1605.04462. find
  2. Anxiety and Depression Association of America. 2015. Anxiety and depression, facts & statistics. find
  3. Adrian Benton, Glen Coppersmith, and Mark Dredze. 2017a. Ethical research protocols for social media health research. find our card
  4. Adrian Benton, Margaret Mitchell, and Dirk Hovy. 2017b. Multitask learning for mental health conditions with limited social media data. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 152–162, Valencia, Spai… find
  5. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python. O’Reilly Media. find
  6. Chris Brew. 2016. Classifying reachout posts with a radial basis function svm. In Proceedings of the Third Workshop on Computational Lingusitics and Clinical Psychology, pages 138–142, San Diego, CA, USA. Association for Computational Linguistics. find
  7. CDC. 2015. Suicide fact sheet, suicide facts at a glance. National Center for Injury Prevention and Control. find
  8. Munmun De Choudhury and Emre Kiciman. 2017. The language of social support in social media and its effect on suicidal ideation risk. In Proceedings of the International Conference onWeb and Social Media (ICWSM). AAAI. find
  9. Arman Cohan, Sydney Young, and Nazli Goharian. 2016. Triaging mental health forum posts. Proceedings of the 3rd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, pages 143–147. find
  10. Arman Cohan, Sydney Young, Andrew Yates, and Nazli Goharian. 2017. Triaging content severity in online mental health forums. Journal of the Association for Information Science and Technology. find
  11. Glen Coppersmith, Mark Dredze, and Craig Harman. 2014a. Quantifying mental health signals in Twitter. In Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality. find our card
  12. Glen Coppersmith, Mark Dredze, Craig Harman, and Kristy Hollingshead. 2015a. From adhd to sad: Analyzing the language of mental health on twitter through self-reported diagnoses. In CLPysch, pages 1–10. find
  13. Glen Coppersmith, Mark Dredze, Craig Harman, Kristy Hollingshead, and Margaret Mitchell. 2015b. find
  14. Clpsych 2015 shared task: Depression and ptsd on twitter. find our card
  15. Glen Coppersmith, Craig Harman, and Mark Dredze. 2014b. Measuring post traumatic stress disorder in twitter. In ICWSM. find
  16. Glen Coppersmith, Kim Ngo, Ryan Leary, and Anthony Wood. 2016. Exploratory analysis of social media prior to a suicide attempt. find
  17. Munmun De Choudhury and Sushovan De. 2014. Mental health discourse on reddit: Self-disclosure, social support, and anonymity. In ICWSM. find
  18. Munmun De Choudhury, Michael Gamon, Scott Counts, and Eric Horvitz. 2013. Predicting Depression via Social Media. AAAI. find
  19. Munmun De Choudhury, Emre Kiciman, Mark Dredze, Glen Coppersmith, and Mrinal Kumar. 2016. Discovering shifts to suicidal ideation from mental health content in social media. In Proceedings of the 34rd Annual ACM Conference on Human Factors in Computing Systems, CHI ’16. ACM. find
  20. Alize J Ferrari, Rosana E Norman, Greg Freedman, Amanda J Baxter, Jane E Pirkis, Meredith G Harris, Andrew Page, Emily Carnahan, Louisa Degenhardt, Theo Vos, et al. 2014. The burden attributable to mental and substance use disorders as risk factors for suicide: findings from the global burden of dis… find
  21. John F Gunn and David Lester. 2015. Twitter postings and suicide: An analysis of the postings of a fatal suicide in the 24 hours prior to death. Suicidologi, 17(3). find
  22. Kristy Hollingshead and Lyle Ungar, editors. 2016. Proceedings of the Third Workshop on Computational Linguistics and Clinical Psychology. Association for Computational Linguistics, San Diego, CA, USA. find
  23. Dirk Hovy and Shannon L Spruit. 2016. The social impact of natural language processing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, volume 2, pages 591– 598. find our card
  24. Jared Jashinsky, Scott H Burton, Carl L Hanson, Josh West, Christophe Giraud-Carrier, Michael D Barnes, and Trenton Argyle. 2014. Tracking suicide risk factors through Twitter in the US. Crisis. find
  25. Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 655–665. Association for Computational Linguistics. find our card
  26. Sunghwan Mac Kim, Yufei Wang, Stephen Wan, and Cecile Paris. 2016. Data61-csiro systems at the clpsych 2016 shared task. In Proceedings of the Third Workshop on Computational Lingusitics and Clinical Psychology, pages 128–132, San Diego, CA, USA. Association for Computational Linguistics. find
  27. H. Andrew Schwartz, and Lyle Ungar. 2015. The role of personality, age, and gender in tweeting about mental illness. In Proceedings of the 2nd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality. find
  28. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746–1751. find our card
  29. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprints, arXiv:1412.6980. find
  30. Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015. Skip-thought vectors. In NIPS, Cambridge, MA, USA. MIT Press. find
  31. Tibor Kiss and Jan Strunk. 2006. Unsupervised multilingual sentence boundary detection. Comput. Linguist., 32(4). find our card
  32. David E Losada and Fabio Crestani. 2016. A test collection for research on depression and language use. In International Conference of the Cross-Language Evaluation Forum for European Languages, pages 28–39. find
  33. Shervin Malmasi, Marcos Zampieri, and Mark Dras. 2016. Predicting post severity in mental health forums. In Proceedings of the Third Workshop on Computational Lingusitics and Clinical Psychology, pages 133–137, San Diego, CA, USA. Association for Computational Linguistics. find
  34. L Manikonda and M De Choudhury. 2017. Modeling and understanding visual attributes of mental health disclosures in social media. In CHI ’17. find
  35. David N. Milne, Glen Pink, Ben Hachey, and Rafael A. Calvo. 2016. find
  36. Clpsych 2016 shared task: Triaging content in online peer-support forums. Proceedings of the 3rd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, pages 118–127. find
  37. Margaret Mitchell, Kristy Hollingshead, and Glen Coppersmith. 2015. Quantifying the language of schizophrenia in social media. NAACL-HLT Workshop on CLPsych 2015, page 11. find
  38. Danielle Mowery, Albert Park, Mike Conway, and Craig Bryan. 2016. Towards automatically classifying depressive symptoms from twitter data for population health. In Proceedings of the Workshop on Computational Modeling of Peoples Opinions, Personality, and Emotions in Social Media, pages 182– 191. find
  39. Vinod Nair and Geoffrey E. Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pages 807–814. find
  40. James W Pennebaker, Ryan L Boyd, Kayla Jordan, and Kate Blackburn. 2015. The development and psychometric properties of liwc2015. UT Faculty/Researcher Works. find
  41. Daniel Preoţiuc-Pietro, Johannes Eichstaedt, Gregory Park, Maarten Sap, Laura Smith, Victoria Tobolsky, Philip Resnik, William Armstrong, Leonardo Claudino, and Thang Nguyen. 2015. The university of maryland clpsych 2015 shared task system. In Proceedings of the 2nd Workshop on Computational Lingui… find
  42. Philip Resnik, Anderson Garron, and Rebecca Resnik. 2013. Using topic modeling to improve prediction of neuroticism and depression. In EMNLP, pages 1348–1353. Association for Computational Linguistics. find
  43. Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958. find
  44. Jacopo Staiano and Marco Guerini. 2014. Depechemood: a lexicon for emotion analysis from crowd-annotated news. arXiv preprints, arXiv:1405.1605. — [arXiv:1405.1605](https://arxiv.org/abs/1405.1605) find
  45. Simon Šuster, Stéphan Tulkens, and Walter Daelemans. 2017. A short review of ethical challenges in clinical natural language processing. arXiv preprint arXiv:1703.10090. find
  46. Paul Thompson, Craig Bryan, and Chris Poulin. 2014. Predicting military and veteran suicide risk: Cultural aspects. In Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, pages 1–6, Baltimore, Maryland, USA. Association for Co… find
  47. Sho Tsugawa, Yusuke Kikuchi, Fumio Kishino, Kosuke Nakajima, Yuichi Itoh, and Hiroyuki Ohsaki. 2015. Recognizing depression from twitter activity. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM. find
  48. R Jay Turner, B Gail Frankel, and Deborah M Levin. 1983. Social support: Conceptualization, measurement, and implications for mental health. Research in Community & Mental Health. find
  49. Sida Wang and Christopher D Manning. 2012. Baselines and bigrams: Simple, good sentiment and topic classification. In ACL ’12. find our card
  50. WHO. 2010. World Health Organization, World Health Statistics 2010. World Health Organization. find
  51. Yijun Xiao and Kyunghyun Cho. 2016. Efficient character-level document classification by combining convolution and recurrent layers. arXiv preprints, arXiv:1602.00367. — [arXiv:1602.00367](https://arxiv.org/abs/1602.00367) find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yates-2017-depression-self-harm-risk-assessment/