Elamur Research⌕
☰
home / papers / 2026 / TiC-MuFormer: Time-Aware Caption-Integrated Multimodal Transformers for User-Level Mental Health Modeling

TiC-MuFormer: Time-Aware Caption-Integrated Multimodal Transformers for User-Level Mental Health Modeling

Summary

Inferring mental health risk from a person's social media history means combining text, images and post timing. These signals are hard to merge for three reasons: emotionally revealing posts are rare, people post at irregular intervals, and an image often carries meaning that its short caption-free text does not. The authors propose TiC-MuFormer, a multimodal transformer that addresses this in two steps. First, an instruction-tuned captioning model describes each picture in words, and that description is attached to the post text. Second, temporal embeddings are added before cross-modal attention, so posting rhythm can shape how the modalities are aligned. Each user is represented by a sampled window of consecutive posts. On the Twitter depression benchmark of Shen et al. (2017), the model reportedly outperforms earlier text-only and multimodal systems on every user-level metric. Ablations examine window length, batch size and text encoder. The study suggests that verbalising images and modelling time explicitly improve user-level risk detection. The authors caution that the system is not a clinical tool and has only been tested on one demographically narrow dataset.

Key points

  • Data and setup: experiments are run on the Twitter-based mental health corpus released by Shen et al. (2017). The ablation configuration combines CLIP image embeddings, Time2Vec temporal encoding and a RoBERTa or EmoBERTa text encoder.
  • Window size: longer post windows help steadily. F1 rises from 0.887 at K=32 to 0.971 at K=512 (RoBERTa, B=128).
  • Batch size: B=128 beats B=256. At K=512, F1 drops to 0.964 for RoBERTa and 0.927 for EmoBERTa with the larger batch.
  • Text encoder: general-purpose RoBERTa (0.971) outperforms emotion-tuned EmoBERTa (0.955). The authors attribute this to captions already bringing affective cues into the text stream.
  • Limitations:
    • The data over-represents young US users.
    • Labels are self-reported rather than clinically verified.
    • Generated captions may introduce bias or hallucinated content.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4pyjbjj5hsvt

@inproceedings{tsoumplekas2026tic,
  title = {{TiC-MuFormer: Time-Aware Caption-Integrated Multimodal Transformers for User-Level Mental Health Modeling}},
  author = {Tsoumplekas, Georgios and Spyridis, Yannis and Argyriou, Vasileios},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4pyjbjj5hsvt},
  url = {https://doi.org/10.63317/4pyjbjj5hsvt}
}

References

33 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Minghui An, Jingjing Wang, Shoushan Li, and Guodong Zhou. 2020. Multimodal topic-enriched auxiliary learning for depression detection. In proceedings of the 28th international conference on computational linguistics, pages 1078–1089. find
  2. Inci M Baytas, Cao Xiao, Xi Zhang, Fei Wang, Anil K Jain, and Jiayu Zhou. 2017. Patient subtyping via time-aware lstm networks. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pages 65–74. find
  3. David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993–1022. find our card
  4. Ana-Maria Bucur, Adrian Cosma, Paolo Rosso, and Liviu P Dinu. 2023. It’s just a matter of time: Detecting depression with time-enriched multimodal transformers. In European conference on information retrieval, pages 200–215. Springer. find
  5. Ju Chun Cheng and Arbee LP Chen. 2022. Multimodal time-aware attention networks for depression detection. Journal of Intelligent Information Systems, 59(2):319–339. find
  6. Glen Coppersmith, Mark Dredze, and Craig Harman. 2014. Quantifying mental health signals in twitter. In Proceedings of the workshop on computational linguistics and clinical psychology: From linguistic signal to clinical reality, pages 51–60. find our card
  7. Munmun De Choudhury, Scott Counts, and Eric Horvitz. 2013. Social media as a measurement tool of depression in populations. In Proceedings of the 5th annual ACM web science conference, pages 47–56. find
  8. Linus Ericsson, Henry Gouk, and Timothy M Hospedales. 2021. How well do self-supervised models transfer? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5414–5423. find
  9. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. find
  10. Tao Gui, Qi Zhang, Liang Zhu, Xu Zhou, Minlong Peng, and Xuanjing Huang. 2019a. Depression detection on social media with reinforcement learning. In China National Conference on Chinese Computational Linguistics, pages 613–624. Springer. find
  11. Tao Gui, Liang Zhu, Qi Zhang, Minlong Peng, Xu Zhou, Keyu Ding, and Zhigang Chen. 2019b. Cooperative multimodal approach to depression detection in twitter. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 110–117. find
  12. Keith Harrigian and Mark Dredze. 2022. Then and now: Quantifying the longitudinal validity of self-disclosed depression diagnoses. arXiv preprint arXiv:2206.11155. find
  13. Dirk Hovy, Shannon L Spruit, et al. 2016. The social impact of natural language processing. In The 54th Annual Meeting of the Association for Computational Linguistics Proceedings of the Conference, Vol. 2 (Short Papers). Association for Computational Linguistics. find our card
  14. Shaoxiong Ji, Tianlin Zhang, Luna Ansari, Jie Fu, Prayag Tiwari, and Erik Cambria. 2022. Mentalbert: Publicly available pretrained language models for mental healthcare. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 7184–7190. find
  15. Seyed Mehran Kazemi, Rishab Goel, Sepehr Eghbali, Janahan Ramanan, Jaspreet Sahota, Sanjay Thakur, Stella Wu, Cathal Smyth, Pascal Poupart, and Marcus Brubaker. 2019. Time2vec: Learning a vector representation of time. arXiv preprint arXiv:1907.05321. — [arXiv:1907.05321](https://arxiv.org/abs/1907.05321) find
  16. Taewoon Kim and Piek Vossen. 2021. Emoberta: Speaker-aware emotion recognition in conversation with roberta. arXiv preprint arXiv:2108.12009. — [arXiv:2108.12009](https://arxiv.org/abs/2108.12009) find
  17. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746–1751. find our card
  18. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  19. Xinping Long, Yifan Zhang, Xin Shu, and Jian Shu. 2023. Image-text fusion model for depression tendency detection based on attention. In 2023 6th International Conference on Artificial Intelligence and Big Data (ICAIBD), pages 730–734. IEEE. find
  20. Julia Ohse, Bakir Hadžić, Parvez Mohammed, Nicolina Peperkorn, Michael Danner, Akihiro Yorita, Naoyuki Kubota, Matthias Rätsch, and Youssef Shiban. 2024. Zero-shot strike: Testing the generalisation capabilities of out-of-the-box llm models for depression detection. Computer Speech & Language, 88:10… find
  21. Vrajkumar Patel, Aayush Modi, Harsh Mistry, Ab-hishesh Mishra, Rocky Upadhyay, and Apoorva Shah. 2025. From alt-text to real context: Revolutionizing image captioning using the potential of llm. IJS-CSEIT, 11(1):379–387. find
  22. James W Pennebaker, Ryan L Boyd, Kayla Jordan, and Kate Blackburn. 2015. The development and psychometric properties of liwc2015. find
  23. Gede Aditra Pradnyana, Wiwik Anggraeni, Eko Mulyanto Yuniarno, and Mauridhi Hery Purnomo. 2025a. An explainable ensemble model for revealing the level of depression in social media by considering personality traits and sentiment polarity pattern. Online Social Networks and Media, 46:100307. find
  24. Gede Aditra Pradnyana, Wiwik Anggraeni, Eko Mulyanto Yuniarno, and Mauridhi Hery Purnomo. 2025b. Revealing depression through social media via adaptive gated cross-modal fusion augmented with insights from personality traits. IEEE Access. find
  25. Wei Qin, Zetong Chen, Xun Yang, Lei Wang, Yunshi Lan, Weijieying Ren, and Richang Hong. 2025. Explainable and interactive llms-augmented depression detection in social media. IEEE Transactions on Computational Social Systems. find
  26. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. Pm… find our card
  27. Andrew G Reece and Christopher M Danforth. 2017. Instagram photos reveal predictive markers of depression. EPJ Data Science, 6(1):15. find
  28. Guangyao Shen, Jia Jia, Liqiang Nie, Fuli Feng, Cunjun Zhang, Tianrui Hu, Tat-Seng Chua, Wenwu Zhu, et al. 2017. Depression detection via harvesting social media: A multimodal dictionary learning solution. In IJCAI, volume 2017, pages 3838–3844. find
  29. Nikita Srivatsan, Sofia Samaniego, Omar Florez, and Taylor Berg-Kirkpatrick. 2024. Alt-text with context: Improving accessibility for images on twitter. In The Twelfth International Conference on Learning Representations. find
  30. Waleed Bin Tahir, Shah Khalid, Saied Alshahrani, Shuaa S Alharbi, and Haifa F Alhasson. 2025. Contextvecnet: A context-driven multimodal learning framework for depression detection. IEEE Journal of Biomedical and Health Informatics. find
  31. Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490. find our card
  32. Joel Philip Thekkekara, Sira Yongchareon, and Veronica Liesaputra. 2024. An attention-based cnn-bilstm model for depression detection on social media text. Expert systems with applications, 249:123834. find
  33. Anas Zafar, Danyal Aftab, Rizwan Qureshi, Yaofeng Wang, and Hong Yan. 2024. Multi-explainable temporalnet: An interpretable multimodal approach using temporal convolutional network for user-level depression detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogniti… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tsoumplekas-2026-tic-muformer-time-aware-caption/