Elamur Research⌕
☰
home / papers / 2015 / CLPsych 2015 Shared Task: Depression and PTSD on Twitter

CLPsych 2015 Shared Task: Depression and PTSD on Twitter

Summary

This paper reports on the CLPsych 2015 shared and unshared tasks, which gave researchers a common benchmark for detecting mental-health signals in social media language. The data came from Twitter users who publicly said they had been diagnosed with depression or PTSD. A human annotator screened these statements and removed jokes and other insincere claims. Each diagnosed user was paired with a community control matched on age and gender, both estimated from language by the World Well-Being Project tool. The data were anonymized with a whitelist of allowed fields and hashing. The unshared task was a weekend hackathon at Johns Hopkins in November 2014. It helped refine the setup for the remote shared task, which had three user-level binary problems: depression vs. control, PTSD vs. control, and the new depression vs. PTSD. Teams submitted ranked lists, which were scored mainly by average precision. UMD's system performed best, and combining all submissions did not beat it. Topic models and even simple linguistic features proved useful, but none of the tasks were solved. The shared test set gives the community a common reference point.

Key points

  • Data split. The train set has 327 depression and 246 PTSD users, each with a matched control, for 1,146 users in total. The test set has 150 depression and 150 PTSD users plus their controls, for 600 users.
  • User filtering. Each user contributes up to their latest 3000 tweets. Users needed at least 25 tweets, with at least 75% of them in English according to the Compact Language Detector. The tweet containing the diagnosis statement was removed.
  • Evaluation. The main metric was average precision. Precision at 5%, 10%, and 20% false-alarm rates was also reported. The best of each team's multiple parameter settings counted as its official result. Which system ranked first depended on the metric used.
  • System combination. The organizers combined 47 submitted systems, with Duluth contributing 9, MIQ 12, UMD 16, and WWBP 10. They tried an SVM over normalized scores with 10-fold cross-validation and a simple average of ranks. Neither combination outperformed UMD.
  • Lessons from participants. Topic modeling (UMD, WWBP) produced strong and interpretable signals. Simple features (Duluth, MIQ) still had predictive power. Finer temporal resolution (UMD, MIQ) also showed promise.

Interesting ideas

  • I1. Matched controls from inferred demographics. Each positive user was paired with the control of the same predicted gender and nearest predicted age. This lets you control for confounders in any user-level classification task, even when demographic data is missing.
  • I2. Consistent, length-preserving hashing. Screen names and URLs were replaced with salted hashes that are identical across occurrences and have the same character length. URL domains were left intact. Frequency statistics and metadata character offsets still work, so this is a practical way to share sensitive text corpora.
  • I3. Removing the labeling artifact. The tweet used to assign the label was deleted from the data so that models cannot simply learn the selection signal. This is a general safeguard for any dataset where labels come from self-reported text.
  • I4. Ranked lists scored on relevant subsets. Teams submitted one ranking over all test users, and organizers filtered it down to the relevant pair of classes for each task. This keeps the submission format simple and supports several tasks from one set of outputs.
  • I5. Reporting precision at low false-alarm rates. The paper evaluates performance at low false-alarm rates alongside a single summary score. This matters because the real-world population is far more imbalanced than the balanced dataset, and the paper notes that the choice of metric changed which system came out on top.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

83 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 80 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/w15-1204

@inproceedings{coppersmith2015clpsych,
  title = {{CLPsych 2015 Shared Task: Depression and PTSD on Twitter}},
  author = {Coppersmith, Glen and Dredze, Mark and Harman, Craig and Hollingshead, Kristy and Mitchell, Margaret},
  booktitle = {Proceedings of the 2nd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality},
  year = {2015},
  doi = {10.3115/v1/w15-1204},
  url = {https://doi.org/10.3115/v1/w15-1204}
}

References

16 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Glen Coppersmith, Mark Dredze, and Craig Harman. 2014a. Quantifying mental health signals in Twitter. In Proceedings of the ACL Workshop on Computational Linguistics and Clinical Psychology. find our card
  2. Glen Coppersmith, Craig Harman, and Mark Dredze. 2014b. Measuring post traumatic stress disorder in Twitter. In Proceedings of the 8th International AAAI Conference on Weblogs and Social Media (ICWSM). find
  3. Glen Coppersmith, Mark Dredze, Craig Harman, and Kristy Hollingshead. 2015. From ADHD to SAD: Analyzing the language of mental health on Twitter through self-reported diagnoses. In Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Re… find
  4. Munmun De Choudhury, Michael Gamon, Scott Counts, and Eric Horvitz. 2013. Predicting depression via social media. In Proceedings of the 7th International AAAI Conference on Weblogs and Social Media (ICWSM). find
  5. Virgile Landeiro Dos Reis and Aron Culotta. 2015. Using matched samples to estimate the effects of exercise on mental health from Twitter. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence. find
  6. Paul McNamee and James Mayfield. 2004. Character n-gram tokenization for European language text retrieval. Information Retrieval, 7(1-2):73–97. find
  7. Margaret Mitchell, Glen Coppersmith, and Kristy Hollingshead, editors. 2015. Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality. North American Association for Computational Linguistics, Denver, Colorado, USA, June. find
  8. Ted Pedersen. 2015. Screening Twitter users for depression and PTSD with lexical decision lists. In Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, Denver, Colorado, USA, June. North American Chapter of the Association for… find
  9. Daniel Preotiuc-Pietro, Maarten Sap, H. Andrew Schwartz Schwartz, and Lyle Ungar. 2015. Mental illness detection at the World Well-Being Project for the CLPsych 2015 shared task. In Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical R… find
  10. Nairan Ramirez-Esparza, Cindy K. Chung, Ewa Kacewicz, and James W. Pennebaker. 2008. The psychology of word use in depression forums in English and in Spanish: Testing two text analytic approaches. In Proceedings of the 2nd International AAAI Conference on Weblogs and Social Media (ICWSM). find
  11. Philip Resnik, Anderson Garron, and Rebecca Resnik. 2013. Using topic modeling to improve prediction of neuroticism and depression. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1348–1353. find
  12. Philip Resnik, Rebecca Resnik, and Margaret Mitchell, editors. 2014. Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality. Association for Computational Linguistics, Baltimore, Maryland, USA, June. find
  13. Philip Resnik, William Armstrong, Leonardo Claudino, Thang Nguyen, Viet-An Nguyen, and Jordan Boyd-Graber. 2015. The University of Maryland CLPsych 2015 shared task system. In Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality… find
  14. J. Niels Rosenquist, James H. Fowler, and Nicholas A. Christakis. 2010. Social network determinants of depression. Molecular psychiatry, 16(3):273–281. find
  15. Stephanie S. Rude, Eva-Maria Gortner, and James W. Pennebaker. 2004. Language use of depressed and depression-vulnerable college students. Cognition & Emotion, 18(8):1121–1133. find
  16. Maarten Sap, Greg Park, Johannes C. Eichstaedt, Margaret L. Kern, David J. Stillwell, Michal Kosinski, Lyle H. Ungar, and H. Andrew Schwartz. 2014. Developing age and gender predictive lexica over social media. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMN… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/coppersmith-2015-clpsych-2015-shared-task-depression/