Elamur Research⌕
☰
home / papers / 2026 / LENS: LLM-Enabled Narrative Synthesis for Mental Health by Aligning Multimodal Sensing with Language Models

LENS: LLM-Enabled Narrative Synthesis for Mental Health by Aligning Multimodal Sensing with Language Models

Paper · 2026 · Wenxuan Xu, Arvind Pillai, Subigya Nepal et al.

Summary

LENS addresses the challenge of converting long, multimodal sensor streams from phones and wearables into readable mental-health descriptions. Two obstacles make this hard: LLMs cannot directly process lengthy numeric time series, and few datasets pair raw sensor data with text. The authors use a 90-day study of 258 adults with major depressive disorder. Each EMA self-report on depression and anxiety symptoms is matched with the four hours of passive sensing data that preceded it. The numeric EMA scores are turned into templated narratives, which an LLM then polishes under quality control. This yields more than 100,000 item-level and summary-level QA pairs. A patch-level time-series encoder maps the raw signals into the embedding space of a Qwen LLM. Sensor and text embeddings are interleaved, and training follows a two-stage curriculum. LENS beats strong baselines on standard NLP metrics and on symptom-severity accuracy. Thirteen mental-health professionals rated its narratives as comprehensive and clinically meaningful. The work suggests LLMs could act as interfaces for behavioral sensing, for example in pre-consultation summaries for patients already in treatment.

Key points

  • Data collection: Participants wore a Garmin vivoactive 3 and ran an Android app for 90 days.
    • The analysis covers 50,957 EMAs from 258 participants, out of 300 recruited.
    • Each EMA has 14 items adapted from PHQ-9 and GAD-4, answered three times daily on a 0–100 scale.
  • Sensor signals: GPS, steps, accelerometer ZCR and energy, phone lock/unlock events, heart rate, stress, sleep, and conversation time.
    • All signals are resampled to fixed rates.
  • Narrative synthesis: Scores are mapped to frequency phrases (0–25 "not at all", 26–50 "sometimes", 51–75 "often", 76–100 "constantly").
    • Phrases are inserted into symptom templates, then refined with a Refine-n-Judge loop.
  • Model: A patch-level encoder aligns sensor data with Qwen-family LLMs, trained with a two-stage curriculum.
    • The authors did not test robustness across other model families.
  • Evaluation and availability: In a user study, 13 experts assessed 117 narratives.
    • Code is on GitHub (Wen-xuan-Xu/LENS).
    • Non-identifiable data will be released through the NIMH Data Archive under a Data Use Certification.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Harvard UniversityCambridge · United States126 works · 45,130 citations🎓University of VirginiaCharlottesville · United States40 works · 10,099 citations🎓Dartmouth CollegeHanover · United States41 works · 42,719 citations🏥Massachusetts General HospitalBoston · United States5 works · 5,421 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 worksPsychology · 1,649 worksExperimental and Cognitive Psychology · 930 worksSocial Psychology · 330 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksVision & language · 811 works

Cite

Cite as Wenxuan Xu, Arvind Pillai, Subigya Nepal, Amanda C. Collins, Daniel M Mackin, Michael V. Heinz, Tess Z Griffin, Nicholas C. Jacobson and Andrew Campbell. 2026. LENS: LLM-Enabled Narrative Synthesis for Mental Health by Aligning Multimodal Sensing with Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.793

@inproceedings{xu2026lens,
  title = {{LENS: LLM-Enabled Narrative Synthesis for Mental Health by Aligning Multimodal Sensing with Language Models}},
  author = {Wenxuan Xu, Arvind Pillai, Subigya Nepal, Amanda C. Collins, Daniel M Mackin, Michael V. Heinz, Tess Z Griffin, Nicholas C. Jacobson and Andrew Campbell},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.793},
  url = {https://doi.org/10.18653/v1/2026.acl-long.793}
}

References

45 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew G Wilson. 2023. Large language models are zero-shot time series forecasters. Advances in Neural Information Processing Systems, 36:19622–19635.
  2. Alaa Abd-Alrazaq, Rawan AlSaad, Farag Shuweihdi, Arfan Ahmed, Sarah Aziz, and Javaid Sheikh. 2023. Systematic review and meta-analysis of performance of wearable artificial intelligence in detecting and predicting depression. NPJ Digital Medicine, 6(1):84. find
  3. Sheikh Asif Imran, Mohammad Nur Hossain Khan, Subrata Biswas, and Bashima Islam. 2024. LLaSA: A Multimodal LLM for Human Activity Analysis Through Wearable and Smartphone Sensors. arXiv preprint. ArXiv:2406.14498 [cs] version: 2.
  4. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with im-Nicholas C Jacobson, Damien Lekkas, Raphael Huang, and Natalie Thomas. 2021. Deep learning paired with wearable passive sensing data predicts deterioration in anxiety disorder symptoms across 17–18 years.… find
  5. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  6. Johns Hopkins Medicine. 2023. Mental Health Disorder Statistics. https://www.hopkinsmedicine.org/health/wellness-and-prevention/ mental-health-disorder-statistics. Accessed: November 14, 2025. ↗
  7. Cosentino Justin, Belyaeva Anastasiya, Liu Xin, Furlotte Nicholas, A., Yang Zhun, Lee Chace, Schenck Erik, Patel Yojan, Cui Jian, Schneider Logan, Douglas, Bryant Robby, Gomes Ryan, G., Jiang Allen, Lee Roy, Liu Yun, Perez Javier, Rogers Jameson, K., Speed Cathy, Tailor Shyam, and 15 others. 2024. T… find
  8. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  9. Chunjiang Liu, Xiaoyuan Wang, Qingran Lin, Albert Xiao, Haoyu Chen, Shizheng Wen, Hao Zhang, Lu Qi, Ming-Hsuan Yang, Laszlo A. Jeni, Min Xu, and Yizhou Zhao. 2026. Mosiv: Multi-object system identification from videos. Preprint, arXiv:2603.06022. — [arXiv:2603.06022](https://arxiv.org/abs/2603.06022) find
  10. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Preprint, arXiv:2304.08485.
  11. Haoxin Liu, Chenghao Liu, and B. Aditya Prakash. 2025a. A picture is worth a thousand numbers: Enabling LLMs reason about time series via visualization. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Techn… find
  12. Yubin Kim, Xuhai Xu, Daniel McDuff, Cynthia Breazeal, and Hae Won Park. 2024. Health-llm: Large language models for health prediction via wearable sensor data. find
  13. Ji Liu, Beichen Ma, Qiaolin Yu, Ruoming Jin, Jingbo Zhou, Yang Zhou, Huaiyu Dai, Haixun Wang, Dejing Dou, and Patrick Valduriez. 2025b. Efficient federated learning with heterogeneous data and adaptive dropout. ACM Trans. Knowl. Discov. Data, 19(8). find
  14. Kurt Kroenke, Robert L Spitzer, and Janet BW Williams. 2001. The phq-9: validity of a brief depression severity measure. Journal of general internal medicine, 16(9):606–613. find
  15. Jin Ming, Wang Shiyu, Ma Lintao, Chu Zhixuan, Zhang James, Y., Shi Xiaoming, Chen Pin-Yu, Liang Yuxuan, Li Yuan-Fang, Pan Shirui, and Wen Qingsong. 2023. Time-LLM: Time series forecasting by reprogramming large language models.
  16. Patrick Langer, Thomas Kaar, Max Rosenblattl, Maxwell A. Xu, Winnie Chow, Martin Maritsch, Aradhana Verma, Brian Han, Daniel Seung Kim, Henry Chubb, Scott Ceresnak, Aydin Zahedivash, Alexander Tarlochan Singh Sandhu, Fatima Rodriguez, Daniel McDuff, Elgar Fleisch, Oliver Aalami, Filipe Barata, and P… find
  17. Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, and Huan Liu. 2025a. From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge. arXiv preprint. ArXiv:2411.16594 [cs]. find
  18. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. Preprint, arXiv:2201.12086.
  19. Zechen Li, Shohreh Deldari, Linyao Chen, Hao Xue, and Flora D. Salim. 2025b. SensorLLM: Human-Intuitive Alignment of Multivariate Sensor Data with LLMs for Activity Recognition. arXiv preprint. ArXiv:2410.10624 [cs]. — [arXiv:2410.10624](https://arxiv.org/abs/2410.10624) find
  20. Sehwan Moon, Aram Lee, Jeong Eun Kim, Hee-Ju Kang, Il-Seon Shin, Sung-Wan Kim, Jae-Min Kim, Min Jhon, and Ju-Wan Kim. 2025. Depressllm: Interpretable domain-adapted language model for depression detection from real-world narratives. find
  21. Subigya Nepal, Wenjun Liu, Arvind Pillai, Weichen Wang, Vlado Vojdanovski, Jeremy F Huckins, Courtney Rogers, Meghan L Meyer, and Andrew T Campbell. 2024a. Capturing the college experience: A four-year mobile sensing study of mental health, resilience and behavior of college students during the pand… find
  22. Subigya Nepal, Arvind Pillai, Emma M Parrish, Jason Holden, Colin Depp, Andrew T Campbell, and Eric L Granholm. 2024b. Social isolation and serious mental illness: the role of context-aware mobile interventions. IEEE pervasive computing, 23(1):46–56. find
  23. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
  24. Arvind Pillai, Subigya Nepal, and Andrew Campbell. 2023. Rare life event detection via mobile sensing using multi-task learning. In Conference on Health, Inference, and Learning, pages 279–293. PMLR. find
  25. Arvind Pillai, Dimitris Spathis, Subigya Nepal, Amanda C. Collins, Daniel M. Mackin, Michael V. Heinz, Tess Z. Griffin, Nicholas C. Jacobson, and Andrew Campbell. 2025. Beyond Prompting: Time2Lang – Bridging Time-Series Foundation Models and Large Language Models for Health Sensing. arXiv preprint.… — [arXiv:2502.07608](https://arxiv.org/abs/2502.07608) find
  26. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. find
  27. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. Preprint, arXiv:2103.00020.
  28. Mahsa Sheikh, Meha Qassem, and Panicos A Kyriacou. 2021. Wearable, environmental, and smartphone-based passive sensing for mental health monitoring. Frontiers in digital health, 3:662811. find
  29. Dimitris Spathis and Fahim Kawsar. 2024. The first step is the hardest: Pitfalls of representing and tokenizing temporal data for large language models. Journal of the American Medical Informatics Association, 31(9):2151–2158. find
  30. Robert L Spitzer, Kurt Kroenke, Janet BW Williams, and Bernd Löwe. 2006. A brief measure for assessing generalized anxiety disorder: the gad-7. Archives of internal medicine, 166(10):1092–1097. find
  31. Mingtian Tan, Mike A. Merrill, Vinayak Gupta, Tim Althoff, and Thomas Hartvigsen. 2025. Are language models actually useful for time series forecasting? In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA. Curran Associates Inc. find
  32. Rui Wang, Fanglin Chen, Zhenyu Chen, Tianxing Li, Gabriella Harari, Stefanie Tignor, Xia Zhou, Dror Ben-Zeev, and Andrew T Campbell. 2014. Stu-dentlife: assessing mental health, academic performance and behavioral trends of college students using smartphones. In Proceedings of the 2014 ACM internati… find
  33. Xiaoyuan Wang, Yizhou Zhao, Botao Ye, Xiaojun Shan, Weijie Lyu, Lu Qi, Kelvin CK Chan, Yinxiao Li, and Ming-Hsuan Yang. 2025. Holigs: Holistic gaussian splatting for embodied view synthesis. arXiv preprint arXiv:2506.19291. — [arXiv:2506.19291](https://arxiv.org/abs/2506.19291) find
  34. Eric W Weisstein. 2004. Bonferroni correction. https://mathworld.wolfram. com/. ↗
  35. Zhe Xie, Zeyan Li, Xiao He, Longlong Xu, Xidao Wen, Tieying Zhang, Jianjun Chen, Rui Shi, and Dan Pei. 2025. ChatTS: Aligning Time Series with LLMs via Synthetic Data for Enhanced Understanding and Reasoning. arXiv preprint. ArXiv:2412.03104 [cs]. — [arXiv:2412.03104](https://arxiv.org/abs/2412.03104) find
  36. Xuhai Xu, Xin Liu, Han Zhang, Weichen Wang, Subigya Nepal, and 1 others. 2023. Globem: Cross-dataset generalization of longitudinal human behavior modeling. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 6(4). find
  37. Hao Xue and Flora D Salim. 2023. Promptcast: A new prompt-based learning paradigm for time series forecasting. IEEE Transactions on Knowledge and Data Engineering, 36(11):6851–6864. find
  38. Kun Yan, Lei Ji, Zeyu Wang, Yuntao Wang, Nan Duan, and Shuai Ma. 2023. Voila-a: Aligning vision-language models with user’s gaze attention. Preprint, arXiv:2401.09454. — [arXiv:2401.09454](https://arxiv.org/abs/2401.09454) find
  39. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report.
  40. Kailai Yang, Tianlin Zhang, Ziyan Kuang, Qianqian Xie, Jimin Huang, and Sophia Ananiadou. 2024. Mental-lama: Interpretable mental health analysis on social media with large language models. In Proceedings of the ACM Web Conference 2024, WWW ’24, page 4489–4500, New York, NY, USA. Association for Com… find
  41. Jihan Yao, Peter Jin, Ke Bao, Qiaolin Yu, Khushi Bhardwaj, Chang Su, Jialei Wang, Yikai Zhu, Sugam Devare, Damon Mosk-Aoyama, Zhen Dong, Venkat Krishna Srinivasan, Yineng Zhang, Oleksii Kuchaiev, Jiantao Jiao, and Banghua Zhu. 2025. The measure of all measures: Quantifying LLM benchmark quality. In… find
  42. Hyungjun Yoon, Biniyam Aschalew Tolera, Taesik Gong, Kimin Lee, and Sung-Ju Lee. 2024. By my eyes: Grounding multimodal large language models with sensor data via visual prompting. arXiv preprint arXiv:2407.10385. find
  43. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. Preprint, arXiv:1904.09675. find
  44. Yunkai Zhang, Yawen Zhang, Ming Zheng, Kezhen Chen, Chongyang Gao, Ruian Ge, Siyuan Teng, Amine Jelloul, Jinmeng Rao, Xiaoyuan Guo, Chiang-Wei Fang, Zeyu Zheng, and Jie Yang. 2023. Insight miner: A large-scale multimodal model for insight mining from time series. find
  45. Yuzhen Zhou, Jiajun Li, Yusheng Su, Gowtham Ramesh, Zilin Zhu, Xiang Long, Chenyang Zhao, Jin Pan, Xiaodong Yu, Ze Wang, Kangrui Du, Jialian Wu, Ximeng Sun, Jiang Liu, Qiaolin Yu, Hao Chen, Zicheng Liu, and Emad Barsoum. 2025. April: Active partial rollouts in reinforcement learning to tame long-tai… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2026-lens-llm-enabled-narrative-synthesis/