Elamur Research⌕
☰
home / papers / 2018 / Expert, Crowdsourced, and Machine Assessment of Suicide Risk via Online Postings

Expert, Crowdsourced, and Machine Assessment of Suicide Risk via Online Postings

Summary

The paper asks whether clinicians can reliably judge suicide risk from social media text alone, without talking to the person. The authors built a dataset from a Reddit snapshot (2008–2015). It covers users who posted in the SuicideWatch subreddit, plus an equal number of control users who never posted in mental-health forums. Usernames were replaced with numeric IDs. Each user's SuicideWatch history was split into units of up to five posts. Crowd workers labeled these units on a four-level scale: None, Low, Moderate, Severe. Four suicide-assessment experts independently labeled a subset. Two of the experts used a brief category description, and two used a detailed rubric organized around families of risk factors. Expert agreement passed the usual reliability threshold, which the authors present as the first such evidence for social-media-based risk assessment. The detailed rubric produced more consistent judgments. The authors also compare crowd judgments with expert ones and report baseline machine classification experiments. The dataset will be distributed under governance by the American Association of Suicidology.

Key points

  • Data scale: 11,129 SuicideWatch users with ten or more Reddit posts, totaling about 1.56M posts. Of 1,097 sampled users, 934 were kept, yielding 982 annotation units. Full Reddit histories were extracted for these users and for the controls.
  • Risk scale: four levels, adapted from Corbitt-Hall et al. (2016). A user's overall label is the highest risk assigned to any of their units.
  • Rubric: the detailed version groups risk factors into four families:
    • thoughts, such as perceived burdensomeness;
    • feelings, such as hopelessness or agitation;
    • logistics, such as methods or access to lethal means;
    • context, such as prior attempts or isolation.
  • Expert agreement: four experts labeled 250 units from 245 users. Mean pairwise Krippendorff's α was 0.812. The pair using short instructions agreed noticeably less (0.768).
  • Consensus and access: consensus labels were inferred with the Dawid–Skene model. Researchers can request the data through AAS, with panel review covering IRB procedures and a ban on linking to other sites.

Interesting ideas

  • I1. Self-selection as a weak label. Treating participation in a topical forum as a candidate-positive signal, with matched controls, gets around missing clinical ground truth. This could work for any condition that has its own online community.
  • I2. Comparing instruction styles. Giving annotators brief versus detailed instructions, then comparing agreement, directly tests whether a guideline adds value. Any annotation project could run this check before scaling up.
  • I3. "Never downgrade" plus evidence selection. Annotators keep the highest risk seen across posts and mark the single post that best supports their label. This makes the labels conservative and also produces post-level evidence that could later supervise attention or explanation models.
  • I4. Capped annotation units. Splitting long user histories into chunks of at most five posts keeps each item manageable for crowd workers. The highest chunk label then becomes the user-level label.
  • I5. Gated release of sensitive data. Releasing the dataset through a domain organization that reviews access requests offers a practical model for sharing sensitive text corpora responsibly.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

80 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 75 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-0603

@inproceedings{shing2018expert,
  title = {{Expert, Crowdsourced, and Machine Assessment of Suicide Risk via Online Postings}},
  author = {Shing, Han-Chin and Nair, Suraj and Zirikly, Ayah and Friedenberg, Meir and III, Hal Daumé and Resnik, Philip},
  booktitle = {Proceedings of the Fifth Workshop on Computational Linguistics and Clinical Psychology: From Keyboard to Clinic},
  year = {2018},
  doi = {10.18653/v1/w18-0603},
  url = {https://doi.org/10.18653/v1/w18-0603}
}

References

57 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jonathan Anderson. 1983. Lix and rix: Variations on a little-known readability index. Journal of Reading 26(6):490–496. find
  2. Philip J Batterham, Maria Ftanou, Jane Pirkis, Jacqueline L Brewer, Andrew J Mackinnon, Annette Beau-trais, A Kate Fairweather-Schmidt, and Helen Christensen. 2015. A systematic review and evaluation of measures for suicidal ideation and behaviors in Glen Coppersmith, Kim Ngo, Ryan Leary, and Anthon… find
  3. Adrian Benton, Glen Coppersmith, and Mark Dredze. 2017. Ethical research protocols for social media health research. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing. pages 94–102. find our card
  4. Darcy J Corbitt-Hall, Jami M Gauthier, Margaret T Davis, and Tracy K Witte. 2016. College students’ responses to suicidal content on social networking sites: an examination using a simulated Facebook newsfeed. Suicide and life-threatening behavior 46(5):609–624. find
  5. Steven Bethard, Guergana Savova, Martha Palmer, and James Pustejovsky. 2017. Semeval-2017 task 12: Clinical tempeval. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). pages 565–572. find
  6. Alexander Philip Dawid and Allan M Skene. 1979. Maximum likelihood estimation of observer error-rates using the em algorithm. Applied statistics pages 20–28. find
  7. David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent Dirichlet allocation. Journal of machine Learning research 3(Jan):993–1022. find our card
  8. Munmun De Choudhury and Sushovan De. 2014. Mental health discourse on Reddit: Self-disclosure, social support, and anonymity. In ICWSM. find
  9. Bureau of Health Workforce. 2017. Designated health professional shortage areas: Statistics, first quarter of fiscal year 2018, designated HPSA quarterly summary. Health Resources and Services Administration (HRSA) U.S. Department of Health & Human Services, https://ersrs.hrsa.gov/ ReportServer?/HGD… ↗
  10. Munmun De Choudhury, Emre Kiciman, Mark Dredze, Glen Coppersmith, and Mrinal Kumar. 2016. Discovering shifts to suicidal ideation from mental health content in social media. In Proceedings of the 2016 CHI conference on human factors in computing systems. ACM, pages 2098–2110. find
  11. Rafael A Calvo, David N Milne, M Sazzad Hussain, and Helen Christensen. 2017. Natural language processing in mental health applications using non-clinical texts. Natural Language Engineering 23(5):649–685. find
  12. Katie G Egan, Rosalind N Koff, and Megan A Moreno. 2013. College students responses to mental health status updates on Facebook. Issues in mental health nursing 34(1):46–51. find
  13. James N Farr, James J Jenkins, and Donald G Paterson. 1951. Simplification of Flesch reading ease formula. Journal of applied psychology 35(5):333. find
  14. Kimberly M Christopherson. 2007. The positive and negative implications of anonymity in internet social interactions:on the internet, nobody knows youre a dog. Computers in Human Behavior 23(6):3038– 3056. find
  15. Ethan Fast, Binbin Chen, and Michael S Bernstein. 2016. Empath: Understanding topic signals in large-scale text. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, pages 4647–4657. find
  16. Meri Coleman and Ta Lin Liau. 1975. A computer readability formula designed for machine scoring. Journal of Applied Psychology 60(2):283. find
  17. Lucie Flekova and Iryna Gurevych. 2016. Supersense embeddings: A unified model for supersense interpretation, prediction, and utilization. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). volume 1, pages 2029–2041. find
  18. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research 12(Aug):2493–2537. find our card
  19. Mike Conway and Daniel O’Connor. 2016. Social media, big data, and mental health: current advances and ethical implications. Current opinion in psychology 9:77–82. find
  20. Rise B Goldstein, Donald W Black, Amelia Nasrallah, and George Winokur. 1991. The prediction of suicide: Sensitivity, specificity, and predictive value of a multivariate model applied to suicide among 1906 patients with affective disorders. Archives of general psychiatry 48(5):418–422. find
  21. Glen Coppersmith, Mark Dredze, and Craig Harman. 2014. Quantifying mental health signals in Twitter. In Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality. pages 51–60. find our card
  22. Kathleen M Griffiths, Louise Farrer, and Helen Christensen. 2010. The efficacy of internet interventions for depression and anxiety disorders: a review of randomised controlled trials. Medical Journal of Australia 192(11):S4. find
  23. Glen Coppersmith, Ryan Leary, Eric Whyne, and Tony Wood. 2015. Quantifying suicidal ideation via language usage on social media. In Joint Statistics Meetings Proceedings, Statistical Computing Section, JSM. find
  24. Robert Gunning. 1952. The technique of clear writing . find
  25. Linguistics and Clinical Psychology. Association for Computational Linguistics, San Diego, CA, USA, pages 118–127. http://www.aclweb.org/ anthology/W16-0312. ↗
  26. Sharath Chandra Guntuku, David B Yaden, Margaret L Kern, Lyle H Ungar, and Johannes C Eichstaedt. 2017. Detecting depression and mental illness on social media: an integrative review. Current Opinion in Behavioral Sciences 18:43–49. find
  27. D.N. Milne. 2017. Triaging content in online peer-support: an overview of the 2017 CLPsych shared task. Available online at http://clpsych.org/shared-task-2017. ↗
  28. Christopher Homan, Ravdeep Johar, Tong Liu, Megan Lytle, Vincent Silenzio, and Cecilia Ovesdotter Alm. 2014. Toward macro-insights for suicide prevention: Analyzing fine-grained distress at scale. In Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Si… find
  29. Saif M Mohammad and Peter D Turney. 2013. Crowdsourcing a word–emotion association lexicon. Computational Intelligence 29(3):436–465. find
  30. Jr Thomas E Joiner, Rheeda L Walker, Jeremy W Pettit, Marisol Perez, and Kelly C Cukrowicz. 2005. Evidence-based assessment of depression in adults. Psychological Assessment 17(3):267. find
  31. Kevin A Padrez, Lyle Ungar, Hansen Andrew Schwartz, Robert J Smith, Shawndra Hill, Tadas Antanavicius, Dana M Brown, Patrick Crutchley, David A Asch, and Raina M Merchant. 2015. Linking social media and medical record data: a study of adults presenting to an academic, urban emergency department. BMJ… find
  32. Jr Thomas E Joiner, Rheeda L Walker, M David Rudd, and David A Jobes. 1999. Scientizing and routinizing the assessment of suicidality in outpatient practice. Professional psychology: Research and practice 30(5):447. find
  33. Rebecca J Passonneau and Bob Carpenter. 2014. The benefits of a model of annotation. Transactions of the Association for Computational Linguistics 2:311–326. find our card
  34. Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882 . find our card
  35. Umashanthi Pavalanathan and Munmun De Choudhury. 2015. Identity management and mental health discourse in social media. In Proceedings of the 24th International Conference on World Wide Web. ACM, pages 315–321. find
  36. J Peter Kincaid, Jr Robert P Fishburne, Richard L Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Technical report, Naval Technical Training Command Millington TN Research B… find
  37. Bryan Klimt and Yiming Yang. 2004. The enron corpus: A new dataset for email classification research. In European Conference on Machine Learning. Springer, pages 217–226. find
  38. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12:2825… find
  39. Klaus Krippendorff. 2004. Reliability in content analysis. Human communication research 30(3):411– 433. find
  40. Alex D Pokorny. 1983. Prediction of suicide in psychiatric patients: report of a prospective study. Archives of general psychiatry 40(3):249–257. find
  41. Fabienne Lind, Maria Gruber, and Hajo G Boomgaarden. 2017. Content analysis by the crowd: Assessing the usability of crowdsourcing for coding latent constructs. Communication methods and measures 11(3):191–209. find
  42. Dina Popovic, Eduard Vieta, Jean-Michel Azorin, Jules Angst, Charles L Bowden, Sergey Mosolov, Allan H Young, and Giulio Perugi. 2015. Suicide attempts in major depressive episode: evidence from the bridge-ii-mix study. Bipolar disorders 17(7):795–803. find
  43. Tong Liu, Qijin Cheng, Christopher M Homan, and Vincent Silenzio. 2017. Learning from various labeling strategies for suicide-related messages on social media: An experimental study. ACM International Conference on Web Search and Data Mining Workshop on Mining Online Health Reports . find
  44. Reddit. 2018. Reddit privacy policy. Downloaded March 22, 2018, https://www.reddit.com/ help/privacypolicy/. ↗
  45. G Harry Mc Laughlin. 1969. Smog grading-a new readability formula. Journal of reading 12(8):639– 646. find
  46. Philip Resnik. 2017. The (in)ability to triangulate in data driven healthcare research. Presentation, SBS Decadal Survey - Workshop on Culture, Language, and Behavior, National Academies of Sciences, Engineering, and Medicine. find
  47. David N. Milne, Glen Pink, Ben Hachey, and Rafael A. Calvo. 2016. find
  48. CLPsych 2016 shared task: Triaging content in online peer-support forums. In Proceedings of the Third Workshop on Computational Philip Resnik, William Armstrong, Leonardo Claudino, and Thang Nguyen. 2015. The university of maryland clpsych 2015 shared task system. In CLPsych@ HLT-NAACL. pages 54–60. find
  49. Philip Resnik, Rebecca Resnik, and Margaret Mitchell, editors. 2014. Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality. Association for Computational Linguistics, Baltimore, Maryland, USA. http://www.aclweb.org/ anthology/W/W… ↗
  50. Rebecca Resnik. 2016. Psychological assessment: The not good enough state of the art. Presentation, Veterans Affairs Suicide Prevention Innovations Conference (VASPI). find
  51. RJ Senter and Edgar A Smith. 1967. Automated readability index. Technical report, CINCINNATI UNIV OH. find
  52. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y Ng. 2008. Cheap and fast—but is it good?: evaluating non-expert annotations for natural language tasks. In Proceedings of the conference on empirical methods in natural language processing. Association for Computational Linguistics, pages 25… find our card
  53. Yla R Tausczik and James W Pennebaker. 2010. The psychological meaning of words: LIWC and computerized text analysis methods. Journal of language and social psychology 29(1):24–54. find
  54. M Johnson Vioulès, Bilel Moulahi, Jérôme Azé, and Sandra Bringay. 2018. Detection of suicide-related posts in twitter data streams. IBM Journal of Research and Development 62(1):7–1. find
  55. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 14… find our card
  56. Andrew Yates, Arman Cohan, and Nazli Goharian. 2017. Depression and self-harm risk assessment in online forums. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Copenhagen, Denmark, pages 2968–2978. https://www.aclw… ↗ our card
  57. Ayah Zirikly, Varun Kumar, and Philip Resnik. 2016. The GW/UMD CLPsych 2016 shared task system. In CLPsych@ HLT-NAACL. pages 166–170. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shing-2018-expert-crowdsourced-machine-assessment-suicide/