📄 Evaluating Document-Tuned Transformer Representations for Person-level Mental Health Assessment

Summary

The paper asks whether contrastively fine-tuned "sentence transformer" encoders produce better person-level representations for mental health assessment than standard masked-language-model encoders. Psychological outcomes are measured per person, but they must be inferred from many short texts. The authors compare two models that share the same architecture: roberta-large (base) and all-roberta-large-v1 (document-tuned). Embeddings are frozen, extracted layer by layer, aggregated across each person's messages by mean pooling or concatenation, and fed to the same downstream predictor. Accuracy is scored with Pearson r. Data come from two longitudinal EMA/survey collections, DS4UD and LMHD, covering depression, anxiety, stress, alcohol use, personality and related scales. Document-tuned embeddings were more accurate overall, by about 13.4% in Pearson r. The gain held across layers, datasets and outcome domains. Probes of word-level associations and noise perturbations did not explain the advantage. The authors therefore suggest document-tuned encoders as a reasonable default for language-based screening, while cautioning that better prediction does not imply clinical validity.

Key points

• Controlled model pair: roberta-large versus all-roberta-large-v1. The document-tuned model was further trained on roughly one billion sentence pairs, so the comparison is not fully matched on training data.
• Datasets:
  • DS4UD: 120 U.S. service-industry workers, 10,108 EMAs, about 84.9 per person.
  • LMHD: 1,307 Prolific participants, 7,207 responses, about 5.5 per person; the sample was enriched with people reporting prior MDD or GAD diagnoses.
• Overall result: document-tuned models improved Pearson r by 13.4% (p = .015).
  • On LMHD, mean pooling gave 0.572 versus 0.561 for the base model.
  • On DS4UD, concatenation gave 0.482 versus 0.411.
  • All of these differences were significant.
• Layer choice: document-tuned models performed best in later layers, with the last four to five layers roughly equivalent.
• Linguistic analysis: hedged words (e.g., "usually", "pretty") carried more outcome signal in document-tuned embeddings, while abundance words (e.g., "lot") carried more in base embeddings.
• Perturbation tests (deletion, typos, WordNet synonyms, back translation): the abstract claims document-tuned models were more robust, but the results section reports mostly comparable degradation for both models. The base model was slightly more robust under heavy synonym replacement.