Elamur Research⌕
☰
home / papers / 2026 / CaTS-Bench: Can Language Models Describe Time Series?

CaTS-Bench: Can Language Models Describe Time Series?

Paper · 2026 · Luca Zhou, Pratham Yashwante, Marshall Fisher et al.

Summary

Time series captioning (TSC) means producing natural-language descriptions of numeric sequences. Existing benchmarks for it mostly rely on synthetic or template-like captions and rarely include metadata or plots. CaTS-Bench addresses this gap with a multimodal benchmark built from 11 real-world source datasets, covering areas such as climate, health, crime, sales, and agriculture. Each sample combines three inputs: the raw series, a JSON file with context and basic statistics, and a randomly styled line plot. Data is split by time to avoid leakage.

The evaluation core is a set of 1746 captions. Humans heavily rewrote these from drafts produced by four different LLMs. For training, the authors use Gemini 2.0 Flash as an oracle grounded in metadata to generate validated synthetic captions at scale. They also provide multiple-choice diagnostic questions and metrics focused on numeric fidelity.

Experiments show that zero-shot VLMs, including proprietary ones, often hallucinate statistics. Fine-tuning open-source models on the synthetic captions improves results substantially. Models also tend to rely on numbers printed on plots rather than reading the curve itself. CaTS-Bench therefore offers a testbed for numerically grounded multimodal text generation.

Key points

  • The human-rewritten test set has 1746 captions, with an average Word Error Rate of 0.60 versus the drafts and a Numeric Jaccard of 0.74. Drafts came from Gemini 2.0 Flash, GPT-4o, Gemma 27B, and Llama 3 70B.
  • The synthetic data comprises about 16k training and 4k test captions from a single oracle, Gemini 2.0 Flash. The authors list reliance on one oracle as a limitation.
  • The benchmark spans 11 domains grouped into 7 macro domains and defines six tasks, with captioning as the main one. It also includes 910 diagnostic MCQs.
  • Each series is split temporally: the first 80% is used for training and augmentation, and the last 20% for benchmark data.
  • Removing numeric annotations from plots sharply reduces accuracy on Max, Min, and Mean. This suggests weak visual grounding. The dataset is available on Hugging Face at mhfisher/CaTSBench.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Sapienza University of RomeRome · Italy57 works · 9,564 citations🎓University of California San DiegoSan Diego · United States73 works · 51,948 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksSignal Processing · 452 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Luca Zhou, Pratham Yashwante, Marshall Fisher, Alessio Sampieri, Zihao Zhou, Fabio Galasso and Rose Yu. 2026. CaTS-Bench: Can Language Models Describe Time Series?. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1722

@inproceedings{zhou2026cats,
  title = {{CaTS-Bench: Can Language Models Describe Time Series?}},
  author = {Luca Zhou, Pratham Yashwante, Marshall Fisher, Alessio Sampieri, Zihao Zhou, Fabio Galasso and Rose Yu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1722},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1722}
}

References

66 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, and 8 oth… — Phi-4 Technical Report
  2. Abdul Fatir Ansari, Lorenzo Stella, Ali Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, Jasper Zschiegner, Danielle C. Maddix, Hao Wang, Michael W. Mahoney, Kari Torkkola, Andrew Gordon Wilson, Michael Bohlk… find
  3. Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku. Model card, Anthropic. find
  4. Saad Aziz. Population collapse time series data of the world. find
  5. Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. 2018. The uea multivariate time series classification archive, 2018. arXiv preprint arXiv:1811.00075. — [arXiv:1811.00075](https://arxiv.org/abs/1811.00075) find
  6. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966.
  7. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72.
  8. Yifu Cai, Arvind Srinivasan, Mononito Goswami, Arjun Choudhry, and Artur Dubrawski. 2024. Jolt: Jointly learned representations of language and time-series for clinical time-series interpretation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 23447–23448. AAAI St… find
  9. California Department of Public Health. n.d. Road traffic injuries narrative. find
  10. Rui Cao and Qiao Wang. 2024. An evaluation of standard statistical models and llms on time series forecasting. arXiv preprint arXiv:2408.04867. — [arXiv:2408.04867](https://arxiv.org/abs/2408.04867) find
  11. Ching Chang, Wen-Chih Peng, and Tien-Fu Chen. 2023. Llm4ts: Two-stage fine-tuning for time-series forecasting with pre-trained llms. arXiv preprint arXiv:2308.08469. — [arXiv:2308.08469](https://arxiv.org/abs/2308.08469) find
  12. Georgios Chatzigeorgakidis, Konstantinos Lentzos, and Dimitrios Skoutas. 2024. Multicast: Zero-shot multivariate time series forecasting using llms. In 2024 IEEE 40th International Conference on Data Engineering Workshops (ICDEW), pages 119–127. IEEE. find
  13. Daqing Chen. 2015. Online Retail. UCI Machine Learning Repository. find
  14. Mouxiang Chen, Lefei Shen, Zhuo Li, Xiaoyun Joy Wang, Jianling Sun, and Chenghao Liu. 2024. Visionts: Visual masked autoencoders are free-lunch zero-shot time series forecasters. arXiv preprint arXiv:2408.17253. — [arXiv:2408.17253](https://arxiv.org/abs/2408.17253) find
  15. Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yiming Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, and 23 others. 2025. Expanding performance boundaries of open-source mult…
  16. Lin Chin-Yew. 2004. Rouge: A package for automatic evaluation of summaries. In Proceedings of the Workshop on Text Summarization Branches Out, 2004.
  17. City of Los Angeles. n.d. Crime data from 2020 to present. find
  18. Hoang Anh Dau, Eamonn Keogh, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, Yanping, Bing Hu, Nurjahan Begum, Anthony Bagnall, Abdullah Mueen, Gustavo Batista, and Hexagon-ML. 2018. The ucr time series classification archive. find
  19. Kota Dohi, Aoi Ito, Harsh Purohit, Tomoya Nishida, Takashi Endo, and Yohei Kawaguchi. 2025. Domain-independent automatic generation of descriptive texts for time-series data. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. find
  20. Jiaxiang Dong, Haixu Wu, Yuxuan Wang, Li Zhang, Jianmin Wang, and Mingsheng Long. 2024. Metadata matters for time series: Informative forecasting with transformers. arXiv preprint arXiv:2410.03806. — [arXiv:2410.03806](https://arxiv.org/abs/2410.03806) find
  21. European Centre for Disease Prevention and Control. 2024. Download today’s data on the geographic distribution of covid-19 cases worldwide. find
  22. Elizabeth Fons, Rachneet Kaur, Soham Palande, Zhen Zeng, Tucker Balch, Manuela Veloso, and Svitlana Vyetrenko. 2024a. Evaluating large language models on time series feature understanding: A comprehensive taxonomy and benchmark. In Proceedings of the 2024 Conference on Empirical Methods in Natural L… find
  23. Elizabeth Fons, Rachneet Kaur, Zhen Zeng, Soham Palande, Tucker Balch, Svitlana Vyetrenko, and Manuela Veloso. 2024b. Tadacap: Time-series adaptive domain-aware captioning. In Proceedings of the 5th ACM International Conference on AI in Finance, pages 54–62. find
  24. Food and Agriculture Organization of the United Nations. 2024. Faostat - food balance sheets. find
  25. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Pal: Program-aided language models. Preprint, arXiv:2211.10435. find
  26. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Empirical Methods in Natural Language Processing (EMNLP).
  27. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazari… — Gemini: A Family of Highly Capable Multimodal Models
  28. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penche… find
  29. Rakshitha Godahewa, Christoph Bergmeir, Geoffrey I. Webb, Rob J. Hyndman, and Pablo Montero-Manso. 2021. Monash time series forecasting archive. In Neural Information Processing Systems Track on Datasets and Benchmarks. find
  30. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  31. Yasser Hassan. 2021. Walmart dataset. find
  32. Abhishek S. Jha. 2023. Time series air quality data of india (2010–2023). find
  33. Harsh Jhamtani and Taylor Berg-Kirkpatrick. 2021. Truth-conditional captioning of time series data. In EMNLP. find
  34. Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, and Qingsong Wen. 2024. Time-LLM: Time series forecasting by reprogramming large language models. In International Conference on Learning Representations (ICLR). find
  35. Kai Kim, Howard Tsai, Rajat Sen, Abhimanyu Das, Zihao Zhou, Abhishek Tanpure, Mathew Luo, and Rose Yu. 2024. Multi-modal forecaster: Jointly predicting time series and textual data. arXiv preprint arXiv:2411.06735. — [arXiv:2411.06735](https://arxiv.org/abs/2411.06735) find
  36. Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. 2024. What matters when building vision-language models? Advances in Neural Information Processing Systems, 37:87874–87907.
  37. Chen Liu, Shibo He, Qihang Zhou, Shizhong Li, and Wenchao Meng. 2024a. Large language model guided knowledge distillation for time series anomaly detection. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, pages 2162–2170. find
  38. Chenxi Liu, Qianxiong Xu, Hao Miao, Sun Yang, Lingzheng Zhang, Cheng Long, Ziyue Li, and Rui Zhao. 2024b. Timecma: Towards llm-empowered time series forecasting via cross-modality alignment. CoRR. find
  39. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning.
  40. Haoxin Liu, Harshavardhan Kamarthi, Zhiyuan Zhao, Shangqing Xu, Shiyu Wang, Qingsong Wen, Tom Hartvigsen, Fei Wang, and B Aditya Prakash. 2025. How can time series analysis benefit from multiple modalities? a survey and outlook. arXiv preprint arXiv:2503.11835. — [arXiv:2503.11835](https://arxiv.org/abs/2503.11835) find
  41. Peiyuan Liu, Hang Guo, Tao Dai, Naiqi Li, Jigang Bao, Xudong Ren, Yong Jiang, and Shu-Tao Xia. 2024c. Calf: Aligning llms for time series forecasting via cross-modal fine-tuning. arXiv preprint arXiv:2403.07300. — [arXiv:2403.07300](https://arxiv.org/abs/2403.07300) find
  42. Andrés Marafioti, Orr Zohar, Miquel Farré, Merve Noyan, Elie Bakouch, Pedro Cuenca, Cyril Zakka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Vaibhav Srivastav, Joshua Lochner, Hugo Larcher, Mathieu Morlon, Lewis Tunstall, Leandro von Werra, and Thomas Wolf. 2025. Smolvlm: Redefining small and ef… find
  43. Mike Merrill, Mingtian Tan, Vinayak Gupta, Thomas Hartvigsen, and Tim Althoff. 2024. Language models still struggle to zero-shot reason about time series. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3512–3533. find
  44. OpenAI. 2025. Gpt-5.1 technical report and system card addendum. Accessed: 2026-01-05. find
  45. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others.… — GPT-4 Technical Report
  46. Zijie Pan, Yushan Jiang, Sahil Garg, Anderson Schneider, Yuriy Nevmyvaka, and Dongjin Song. 2024. s 2 ip-llm: Semantic space informed prompt learning with llm for time series forecasting. In Forty-first International Conference on Machine Learning. find
  47. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
  48. Hannah Ritchie. 2021. Many countries have decoupled economic growth from co2 emissions, even if we take offshored production into account. Our World in Data. https://ourworldindata.org/co2-gdp -decoupling. ↗
  49. Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. 2023. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 11121– 11128. find
  50. Medhasweta Sen, Zachary Gottesman, Jiaxing Qiu, C. Bayan Bruss, Nam Nguyen, and Tom Hartvigsen. 2025. Bedtime: A unified benchmark for automatically describing time series. Preprint, arXiv:2509.05215. — [arXiv:2509.05215](https://arxiv.org/abs/2509.05215) find
  51. Jingyang Zhang. 2024. Visualizing the attention of vision-language models. find
  52. Chenxi Sun, Hongyan Li, Yaliang Li, and Shenda Hong. 2023. Test: Text prototype aligned embedding to activate llm’s ability for time series. In The Twelfth International Conference on Learning Representations. find
  53. Mingtian Tan, Mike A Merrill, Vinayak Gupta, Tim Althoff, and Thomas Hartvigsen. 2024. Are language models actually useful for time series forecasting? In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  54. Benny J Tang, Angie Boggust, and Arvind Satyanarayan. 2023. Vistext: A benchmark for semantically rich chart captioning. arXiv preprint arXiv:2307.05356. find
  55. Hua Tang, Chong Zhang, Mingyu Jin, Qinkai Yu, Zhenting Wang, Xiaobo Jin, Yongfeng Zhang, and Mengnan Du. 2025. Time series forecasting with llms: Understanding and enhancing model capabilities. ACM SIGKDD Explorations Newsletter, 26(2):109–118. find
  56. U.S. Department of Transportation, Bureau of Transportation Statistics. n.d. Border crossing entry data. find
  57. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. find
  58. Xiyuan Zhang, Ranak Roy Chowdhury, Rajesh K Gupta, and Jingbo Shang. 2024. Large language models for time series: a survey. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, pages 8335–8343. find
  59. Tian Zhou, Peisong Niu, Xue Wang, Liang Sun, and Rong Jin. 2023. One Fits All: Power general time series analysis by pretrained lm. In NeurIPS. find
  60. Zihao Zhou and Rose Yu. 2025. Can LLMs understand time series anomalies? In The Thirteenth International Conference on Learning Representations. find
  61. Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, and 32 others. 2025. Internvl3: Exploring advanced training and test-…
  62. USDA Economic Research Service. 2024. International agricultural productivity. find
  63. Xinlei Wang, Maike Feng, Jing Qiu, Jinjin Gu, and Junhua Zhao. 2024. From news to forecast: Integrating event analysis in llm-based time series forecasting with reflection. Advances in Neural Information Processing Systems, 37:58118–58153. find
  64. J. Nan Wilkenfeld, Bei Yan, Jujun Huang, Guirong Luo, and Kristina Algas. 2022. “AI Love You”: Linguistic Convergence in Human-Chatbot Relationship Development. In Academy of Management Proceedings, volume 2022, page 17063. Academy of Management. find
  65. Andrew Robert Williams, Arjun Ashok, Étienne Marcotte, Valentina Zantedeschi, Jithendaraa Subramanian, Roland Riachi, James Requeima, Alexandre Lacoste, Irina Rish, Nicolas Chapados, and Alexandre Drouin. 2024. Context is key: A benchmark for forecasting with essential textual information. Preprint,… — [arXiv:2410.18959](https://arxiv.org/abs/2410.18959) find
  66. Hao Xue and Flora D Salim. 2023. Promptcast: A new prompt-based learning paradigm for time series forecasting. IEEE Transactions on Knowledge and Data Engineering, 36(11):6851–6864. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2026-cats-bench-can-language-models/