Elamur Research⌕
☰
home / papers / 2019 / CLPsych 2019 Shared Task: Predicting the Degree of Suicide Risk in Reddit Posts

CLPsych 2019 Shared Task: Predicting the Degree of Suicide Risk in Reddit Posts

Summary

This paper describes the CLPsych 2019 shared task. The task asked whether Reddit language can show how much suicide risk a person faces, not just whether any risk exists. The organizers argue that binary screening would flag more people than clinicians can help, so a graded scale would make prioritization possible.

The data come from the Reddit collection of Shing et al. (2018). Annotators labelled users who had posted on r/SuicideWatch on a four-level scale: no, low, moderate or severe risk. The shared task offered three tracks:

  • Task A: assess risk from a user's SuicideWatch posts alone.
  • Task B: assess risk using those posts plus the user's other Reddit activity.
  • Task C: screen users from their everyday, non-SuicideWatch posts only.

Fifteen teams took part. The best systems were good at separating urgent cases and at telling apart users who need some attention from those at no risk. The middle categories (low and moderate) stayed hard for systems, as they are for human annotators. The task matters because it gives the community a shared, ethically managed benchmark for a sensitive problem where data are scarce.

Key points

  • Source data: 11,129 candidate SuicideWatch users with 1,556,194 posts across Reddit, plus a similar number of control users. Of these, 621 randomly chosen users were annotated for risk at the user level, not the post level.
  • Annotation scheme: built with suicide prevention experts. It groups risk factors into four families: Thoughts, Feelings, Logistics and Context.
  • Annotator agreement:
    • Experts reached Krippendorff's α = 0.81.
    • Crowdworkers reached α = 0.55, and their disagreements with experts tended to lean toward caution.
  • Labels used and de-identification: both training and test labels came from crowdworkers. De-identification steps:
    • user names were swapped for numeric IDs;
    • spaCy NER plus pattern matching replaced person names, emails, URLs and IPs with placeholder tokens;
    • posts containing Arabic were removed using langdetect.
  • Task C result: the top primary system by macro-F1 was CLaC (macro-F1 0.268, accuracy 0.673). For comparison, a binary "flagged" F1 of about 0.67 was reached, but the F1 for intermediate classes was often near zero.

Interesting ideas

  • I1. Grade risk instead of flagging it. Framing the problem as graded severity supports triage when intervention capacity is limited. The same framing could apply to any screening setting where the number of positives exceeds the resources to act on them.
  • I2. Report collapsed binary scores next to fine-grained ones. Alongside four-way macro-F1, the task reported coarser binary scores (flagged, urgent). This shows that a system can be useful for triage even when it cannot separate neighbouring classes, an evaluation habit worth copying for ordinal labels.
  • I3. Keep train and test labels from the same source. Both splits used crowdsourced labels, even though expert labels were available. This avoids a train/test annotation mismatch and makes it cheap to build fresh test sets for repeated editions. It is a pragmatic choice for any recurring benchmark.
  • I4. Make data access conditional on ethics safeguards. Participants had to show IRB approval, submit a data management plan and commit to published ethics guidelines. This is a reusable model for sharing sensitive corpora without fully open release.
  • I5. Bring researchers to the data. The authors plan secure infrastructure that lets researchers work on sensitive data where it is stored, rather than sending copies out. This pattern suits any project dealing with private or personal text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

101 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 98 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-3003

@inproceedings{zirikly2019clpsych,
  title = {{CLPsych 2019 Shared Task: Predicting the Degree of Suicide Risk in Reddit Posts}},
  author = {Zirikly, Ayah and Resnik, Philip and Uzuner, Özlem and Hollingshead, Kristy},
  booktitle = {Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology},
  year = {2019},
  doi = {10.18653/v1/w19-3003},
  url = {https://doi.org/10.18653/v1/w19-3003}
}

References

39 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Glen Coppersmith, Mark Dredze, and Craig Harman. 2014. Quantifying mental health signals in Twitter. In Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, pages 51–60. find our card
  2. Kristen Allen, Shrey Bagroy, Alex Davis, and Tamar Krishnamurti. 2019. ConvSent at CLPsych 2019 task a: Using post-level sentiment features for suicide risk prediction on Reddit. In Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology. find
  3. Glen Coppersmith, Ryan Leary, Patrick Crutchley, and Alex Fine. 2018. Natural language processing of social media as screening for suicide risk. Biomedical Informatics Insights, 10:1178222618792860. find
  4. Ashwin Karthik Ambalavanan, Pranjali Dileep Jagtap, Soumya Adhya, and Murthy Devarakonda. 2019. Using contextual representations for suicide risk assessment from internet forums. In Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology. find
  5. Darcy J Corbitt-Hall, Jami M Gauthier, Margaret T Davis, and Tracy K Witte. 2016. College students’ responses to suicidal content on social networking sites: an examination using a simulated Facebook newsfeed. Suicide and life-threatening behavior, 46(5):609–624. find
  6. Christos Baziotis, Nikos Pelekis, and Christos Doulk-eridis. 2017. Datastories at semeval-2017 task 4: Deep lstm with attention for message-level and topic-based sentiment analysis. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 747–754, Vancouver, Can… find our card
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
  8. Ethan Fast, Binbin Chen, and Michael S Bernstein. 2016. Empath: Understanding topic signals in large-scale text. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pages 4647–4657. ACM. find
  9. Adrian Benton, Glen Coppersmith, and Mark Dredze. 2017. Ethical research protocols for social media health research. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing, pages 94–102. find our card
  10. Joseph C Franklin, Jessica Ribeiro, Kathryn Fox, Kate Bentley, Evan Kleiman, Xieyining Huang, Katherine Musacchio, Adam Jaroszewski, Bernard Chang, and Matthew Nock. 2016. Risk factors for suicidal thoughts and behaviors: A meta-analysis of 50 years of research. Psychological bulletin, 143. DOI 10.1… — [doi:10.1037/bul0000084](https://doi.org/10.1037/bul0000084) find
  11. Semere Kiros Bitew, Giannis Bekoulis, Johannes Deleu, Lucas Sterckx, Klim Zaporojets, Thomas Demeester, and Chris Develder. 2019. Predicting suicide risk from online postings in Reddit the UGent-IDLab submission to the CLPysch 2019 shared task A. In Proceedings of the Sixth Workshop on Computational… find
  12. Devin Gaffney and J Nathan Matias. 2018. Caveat emptor, computational social science: Large-scale missing data in a widely-published Reddit corpus. PloS one, 13(7):e0200162. find
  13. David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993–1022. find our card
  14. Rachel Gutman. 2018. Reddit’s case for anonymity on the Internet. The Atlantic. find
  15. Olivier Bodenreider. 2004. The unified medical language system (umls): integrating biomedical terminology. Nucleic acids research, 32(suppl 1):D267– D270. find
  16. Alejandro González Hevia, Rebeca Cerezo Menéndez, and Daniel Gayo-Avello. 2019. Analyzing the use of existing systems for the CLPsych 2019 shared task. In Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology. find
  17. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175. find our card
  18. Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing. To appear. find
  19. Stevie Chancellor, Michael L Birnbaum, Eric D Caine, Vincent Silenzio, and Munmun De Choudhury. 2019. A taxonomy of ethical tensions in inferring mental health states from social media. In Proceedings of the 2nd ACM Conference on Fairness, Accountability, and Transparency (FAT* ’19). find
  20. Micah Iserman, Taleen Nalabandian, and Molly E. Ireland. 2019. Dictionaries and decision trees for the 2019 CLPsych shared task. In Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology. find
  21. Lushi Chen, Abeer Aldayel, Nikolay Bogoychev, and Tao Gong. 2019. Similar minds post alike: Assessment of suicide risk by hybrid language and behavioral model. In Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology. find
  22. Thomas E Joiner, Jr, Rheeda L Walker, M David Rudd, and David A Jobes. 1999. Scientizing and routinizing the assessment of suicidality in outpatient practice. Professional psychology: Research and practice, 30(5):447. find
  23. Victor Ruiz, Lingyun Shi, Jorge Guerra, Wei Quan, Neal Ryan, Candice Biernesser, David Brent, and Fuchiang Tsui. 2019. CLPsych2019 shared task: Predicting users suicide risk levels from their Reddit posts on multiple forums. In Proceedings of the Sixth Workshop on Computational Linguistics and Clini… find
  24. Matthew Matero, Akash Idnani, Youngseo Son, Salvatore Giorgi, Huy Vu, and Mohammadzaman Zamani. 2019. Suicide risk assessment with multi-level dual-context language and BERT. In Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology. find
  25. Catherine M McHugh, Amy Corderoy, Christopher James Ryan, Ian B Hickie, and Matthew Michael Large. 2019. Association between suicidal ideation and suicide: meta-analyses of odds ratios, sensitivity, specificity and positive predictive value. BJPsych open, 5(2). find
  26. Maarten Sap, Gregory Park, Johannes Eichstaedt, Margaret Kern, David Stillwell, Michal Kosinski, Lyle Ungar, and Hansen Andrew Schwartz. 2014. Developing age and gender predictive lexica over social media. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMN… find
  27. David N. Milne, Glen Pink, Ben Hachey, and Rafael A. Calvo. 2016. find
  28. CLPsych 2016 shared task: Triaging content in online peer-support forums. In Proceedings of the Third Workshop on Computational Linguistics and Clinical Psychology, pages 118–127, San Diego, CA, USA. Association for Computational Linguistics. find
  29. H Andrew Schwartz, Johannes C Eichstaedt, Margaret L Kern, Lukasz Dziurzynski, Stephanie M Ramones, Megha Agrawal, Achal Shah, Michal Kosinski, David Stillwell, Martin EP Seligman, et al. 2013. Personality, gender, and age in the language of social media: The open-vocabulary approach. PloS one, 8(9)… find
  30. D.N. Milne. 2017. Triaging content in online peer-support: an overview of the 2017 CLPsych shared task. Available online at http://clpsych.org/shared-task-2017. ↗
  31. Han-Chin Shing, Suraj Nair, Ayah Zirikly, Meir Friedenberg, Hal Daumé III, and Philip Resnik. 2018. Expert, crowdsourced, and machine assessment of suicide risk via online postings. In Proceedings of the Fifth Workshop on Computational Linguistics and Clinical Psychology: From Keyboard to Clinic, p… find our card
  32. Saif M Mohammad. 2017. Word affect intensities. arXiv preprint arXiv:1704.08798. find our card
  33. Elham Mohammadi, Hessam Amini, and Leila Kosseim. 2019. CLaC at clpsych 2019: Fusion of neural features and predicted class probabilities for suicide risk assessment based on online posts. In Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology. find
  34. Yla R Tausczik and James W Pennebaker. 2010. The psychological meaning of words: LIWC and computerized text analysis methods. Journal of language and social psychology, 29(1):24–54. find
  35. Michelle Morales, Danny Belitz, Natalia Chernova, Prajjalita Dey, and Thomas Theisen. 2019. An investigation of deep learning systems for suicide risk assessment. In Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology. find
  36. Matthew K. Nock, Franchesca Ramirez, and Osiris Rankin. 2019. Advancing Our Understanding of the Who, When, and Why of Suicide Risk. JAMA Psychiatry, 76(1):11–12. find
  37. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365. find
  38. Dina Popovic, Eduard Vieta, Jean-Michel Azorin, Jules Angst, Charles L Bowden, Sergey Mosolov, Allan H Young, and Giulio Perugi. 2015. Suicide attempts in major depressive episode: evidence from the bridge-ii-mix study. Bipolar disorders, 17(7):795–803. find
  39. Esteban A. Rı́ssola, Diana Ramı́rez-Cifuentes, Ana Freire, and Fabio Crestani. 2019. Suicide risk assessment on social media: USI-UPF at the CLPsych 2019 shared task. In Proceedings of the Sixth Workshop on Computational Linguistics and Clinical Psychology. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zirikly-2019-clpsych-2019-shared-task-predicting/