Elamur Research⌕
☰
home / papers / 2026 / Chat-TS: Enhancing Multi-Modal Reasoning Over Time-Series and Natural Language Data

Chat-TS: Enhancing Multi-Modal Reasoning Over Time-Series and Natural Language Data

Paper · 2026 · Paul Quinlan, Qingguo Li and Xiaodan Zhu

Summary

Time-series data such as prices, sensor readings and ECGs are common in many fields, yet LLMs are still weak at reasoning jointly over numeric series and accompanying text. Existing approaches also tend to sacrifice the model's general language skills or never measure them. Chat-TS addresses this by adding discrete time-series tokens directly to a pretrained LLM's vocabulary (Llama-3.1-8B), so that text and series form a single token stream trained with ordinary next-token prediction. No connector module is needed. The authors compare two ways of initializing the new embeddings: averaging existing text embeddings, or a pre-training stage that updates only the embedding and output layers. They then fine-tune on their new TS Instruct conversational data mixed with a subset of OpenOrca to retain text abilities. They also release a human-validated multiple-choice benchmark and a small probing set. Chat-TS outperforms text-encoding and learned-encoder baselines on time-series reasoning while preserving language quality. This makes it a practical, reproducible baseline for multimodal reasoning over time series and text.

Key points

  • Released resources:

    • the TS Instruct Training Dataset, with instruction conversations grounded in real, non-overlapping series;
    • the TS Instruct QA Gold benchmark, with 1,056 human-verified multiple-choice questions;
    • a TS Instruct Quantitative Probing Set covering QA, math and decision-making;
    • the models, data and code at github.com/quinlanp/Chat-TS-Multi-Modal-Reasoning.

  • Overall gain: Chat-TS improves average accuracy on QA Gold by roughly 13% over the prior state-of-the-art text-encoding baseline.

  • Matched-setting comparison (Table 6), using Llama-3.1-8B and identical OpenOrca+TS-Instruct training throughout:

    ModelQA Gold (%)MCQ2TS (%)
    Untrained base model54.236.5
    Text encoding63.050.8
    MLP encoder67.543.1
    Transformer encoder65.945.0
    Chat-TS69.456.4

  • Text-quality ratings: GPT-4o scored responses from 1 to 5 on helpfulness, relevance, accuracy and detail. Variants that mixed OpenOrca text with time-series data (OrcaTS, PreOrcaTS) ranked highest.

  • Limitations: forecasting is not yet reliable, and zero-shot time-series classification remains weak.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Queen's UniversityKingston · Canada11 works · 1,637 citations🎓Queens UniversityDhaka · Bangladesh4 works · 579 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksSignal Processing · 452 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksVision & language · 657 works

Cite

Cite as Paul Quinlan, Qingguo Li and Xiaodan Zhu. 2026. Chat-TS: Enhancing Multi-Modal Reasoning Over Time-Series and Natural Language Data. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.263

@inproceedings{quinlan2026chat,
  title = {{Chat-TS: Enhancing Multi-Modal Reasoning Over Time-Series and Natural Language Data}},
  author = {Paul Quinlan, Qingguo Li and Xiaodan Zhu},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.263},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.263}
}

References

40 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, Jasper Zschiegner, Danielle C. Maddix, Hao Wang, Michael W. Mahoney, Kari Torkkola, Andrew Gordon Wilson, Michael Bohlke-Sc… — [arXiv:2403.07815](https://arxiv.org/abs/2403.07815) find
  2. Defu Cao, Furong Jia, Sercan O Arik, Tomas Pfister, Yixiang Zheng, Wen Ye, and Yan Liu. 2024. Tempo: Prompt-based generative pre-trained transformer for time series forecasting. Preprint, arXiv:2310.04948. — [arXiv:2310.04948](https://arxiv.org/abs/2310.04948) find
  3. Ching Chang, Wei-Yao Wang, Wen-Chih Peng, and Tien-Fu Chen. 2024. Llm4ts: Aligning pre-trained llms as data-efficient time-series forecasters. Preprint, arXiv:2308.08469. — [arXiv:2308.08469](https://arxiv.org/abs/2308.08469) find
  4. Yongjian Chen, Tao Guan, and Cheng Wang. 2010. Approximate nearest neighbor search by residual vector quantization. Sensors (Basel), 10(12):11259–11273. find
  5. Mingyue Cheng, Yiheng Chen, Qi Liu, Zhiding Liu, and Yucong Luo. 2024. Advancing time series classification with multimodal language modeling. Preprint, arXiv:2403.12371. — [arXiv:2403.12371](https://arxiv.org/abs/2403.12371) find
  6. Winnie Chow, Lauren Gardiner, Haraldur T. Hallgríms-son, Maxwell A. Xu, and Shirley You Ren. 2024. Towards time series reasoning with llms. Preprint, arXiv:2409.11376. — [arXiv:2409.11376](https://arxiv.org/abs/2409.11376) find
  7. Kevin Crowston. 2012. Amazon mechanical turk: A research tool for organizations and information systems scholars. In Shaping the Future of ICT Research. Methods and Approaches, pages 210–221, Berlin, Heidelberg. Springer Berlin Heidelberg. find
  8. Angus Dempster, François Petitjean, and Geoffrey I. Webb. 2020. Rocket: exceptionally fast and accurate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery, 34(5):1454–1495. find
  9. Abhimanyu Dubey, Abhinav Jauhri, and Abhinav Pandey et al. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  10. Rakshitha Godahewa, Christoph Bergmeir, Geoffrey I. Webb, Rob J. Hyndman, and Pablo Montero-Manso. 2021. Monash time series forecasting archive. In Neural Information Processing Systems Track on Datasets and Benchmarks. find
  11. Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew Gordon Wilson. 2024. Large language models are zero-shot time series forecasters. Preprint, arXiv:2310.07820. find
  12. Ori Hadary, Luke Marshall, Ishai Menache, Abhisek Pan, David Dion, Esaias E Greeff, Star Dorminey, Shailesh Joshi, Yang Chen, Mark Russinovich, and Thomas Moscibroda. 2020. Protean: Vm allocation service at scale. In OSDI. USENIX. find
  13. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… — Training Compute-Optimal Large Language Models
  14. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  15. Yushan Jiang, Zijie Pan, Xikun Zhang, Sahil Garg, Anderson Schneider, Yuriy Nevmyvaka, and Dongjin Song. 2024. Empowering time series analysis with large language models: A survey. Preprint, arXiv:2402.03182. — [arXiv:2402.03182](https://arxiv.org/abs/2402.03182)
  16. Nataliia Kashpruk, Cezary Piskor-Ignatowicz, and Jerzy Baranowski. 2023. Time series prediction in industry 4.0: A comprehensive review and prospects for future advancements. Applied Sciences, 13(22). find
  17. Yaxuan Kong, Yiyuan Yang, Yoontae Hwang, Wenjie Du, Stefan Zohren, Zhangyang Wang, Ming Jin, and Qingsong Wen. 2025. Time-MQA: Time series multi-task question answering with context enhancement. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon…
  18. Yang Li, Yangyang Yu, Haohang Li, Zhi Chen, and Khaldoun Khashanah. 2023. Tradinggpt: Multi-agent system with layered memory and distinct characters for enhanced financial trading performance. Preprint, arXiv:2309.03736. — [arXiv:2309.03736](https://arxiv.org/abs/2309.03736) find
  19. Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". 2023. Openorca: An open dataset of gpt augmented flan reasoning traces. https://https://huggingface.co/Open-Orca/OpenOrca. find
  20. Spyros Makridakis, Steven C. Wheelwright, and Victor E. McGee. 1977. Forecasting: Methods and Applications. find
  21. Jyoti Aneja Marah Abdin and Hany Awadalla et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. Preprint, arXiv:2404.14219.
  22. Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, Anton Belyi, Haotian Zhang, Karanjeet Singh, Doug Kang, Ankur Jain, Hongyu Hè, Max Schwarzer, Tom Gunter, Xiang Kong, and 13 others. 2024. Mm1: Methods, an… — [arXiv:2403.09611](https://arxiv.org/abs/2403.09611) find
  23. Mike A. Merrill, Mingtian Tan, Vinayak Gupta, Tom Hartvigsen, and Tim Althoff. 2024. Language models still struggle to zero-shot reason about time series. Preprint, arXiv:2404.11757. find
  24. Matthew Middlehurst, Patrick Schäfer, and Anthony Bagnall. 2024. Bake off redux: a review and experimental evaluation of recent time series classification algorithms. Data Mining and Knowledge Discovery, 38(4):1958–2031. find
  25. Mohammad Amin Morid, Olivia R. Liu Sheng, and Joseph Dunbar. 2023. Time series prediction using deep learning methods in healthcare. ACM Trans. Manage. Inf. Syst., 14(1). find
  26. Zabir Al Nazi and Wei Peng. 2024. Large language models in healthcare and medical domain: A review. Informatics, 11(3). find
  27. OpenAI, Josh Achiam, Steven Adler, and Sandhini Agarwal et al. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774.
  28. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2023. Gpqa: A graduate-level google-proof q&a benchmark. Preprint, arXiv:2311.12022.
  29. Chenxi Sun, Hongyan Li, Yaliang Li, and Shenda Hong. 2024. Test: Text prototype aligned embedding to activate llm’s ability for time series. Preprint, arXiv:2308.08241. — [arXiv:2308.08241](https://arxiv.org/abs/2308.08241) find
  30. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. 2022. Challenging big-bench tasks and whether chain-of-thought can solve them. Preprint, arXiv:2210.09261.
  31. Gemini Team, Rohan Anil, Sebastian Borgeaud, and Jean-Baptiste Alayrac et al. 2024a. Gemini: A family of highly capable multimodal models. Preprint, arXiv:2312.11805.
  32. Gemma Team, Thomas Mesnard, and Cassidy Hardin et al. 2024b. Gemma: Open models based on gemini research and technology. Preprint, arXiv:2403.08295. find
  33. Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. 2023. Large language models in medicine. Nature Medicine, 29(8):1930– 1940. find
  34. Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. 2016. Wavenet: A generative model for raw audio. Preprint, arXiv:1609.03499.
  35. Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. 2018. Neural discrete representation learning. Preprint, arXiv:1711.00937. find
  36. Zhe Xie, Zeyan Li, Xiao He, Longlong Xu, Xidao Wen, Tieying Zhang, Jianjun Chen, Rui Shi, and Dan Pei. 2025. Chatts: Aligning time series with llms via synthetic data for enhanced understanding and reasoning. Preprint, arXiv:2412.03104. — [arXiv:2412.03104](https://arxiv.org/abs/2412.03104) find
  37. Lijun Yu, Jose Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A Ross, and Lu Jiang. 2024. Language model beats diffusion - tokenizer is key to visual generation.… find
  38. Cheng Zhang, Nilam Nur Amir Sjarif, and Roslina Ibrahim. 2024a. Deep learning models for price forecasting of financial time series: A review of recent advancements: 2020–2022. WIREs Data Mining and Knowledge Discovery, 14(1):e1519. find
  39. Zijian Zhang, Yujie Sun, Zepu Wang, Yuqi Nie, Xiaobo Ma, Peng Sun, and Ruolin Li. 2024b. Large language models for mobility in transportation systems: A survey on forecasting tasks. Preprint, arXiv:2405.02357. — [arXiv:2405.02357](https://arxiv.org/abs/2405.02357) find
  40. Tian Zhou, PeiSong Niu, Xue Wang, Liang Sun, and Rong Jin. 2023. One fits all:power general time series analysis by pretrained lm. Preprint, arXiv:2302.11939. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/quinlan-2026-chat-ts-enhancing-multi-modal/