Elamur Research⌕
☰
home / papers / 2026 / Which Way Does Time Flow? A Psychophysics-Grounded Evaluation for Vision–Language Models

Which Way Does Time Flow? A Psychophysics-Grounded Evaluation for Vision–Language Models

Summary

The paper asks whether current vision–language models (VLMs) can tell if a short video runs forward or in reverse. This arrow-of-time (AoT) judgment is simple for people but depends on intuitions about gravity, entropy and causality. The authors build AoT-PsyPhyBENCH from an earlier human study by Hanyu et al. (2023), which used everyday clips with measured human responses. They drop cyclic-motion clips, which humans find ambiguous, and keep only items with clear human agreement, so model scores can be compared directly with human scores. They test proprietary and open-weight VLMs, with and without reasoning, under four setups: zero-shot, few-shot, chain-of-thought prompting and supervised fine-tuning. Most models score close to chance and tend to answer "forward." Even the strongest configuration trails humans by roughly 29 percentage points, and more elaborate prompting or fine-tuning brings only small, inconsistent gains. The authors conclude that the limiting factor is weak temporal and physical understanding rather than poor instruction following. They release the benchmark, code and model outputs.

Key points

  • Source data: The original study used 360 three-second clips from the Moments in Time dataset (29.97 FPS). Ten participants viewed each clip both forward and reversed. Humans showed a forward bias: they misjudged 39% of the reversed presentations but only 9% of those played forward.
  • Filtering: The Reciprocal category reached only 61.1% human accuracy, compared with 81.3–85.1% for the other categories. Removing it raised overall human accuracy from 77.6% to 89.0%. The final benchmark has 212 clips, giving 424 videos.
  • Categories kept: Proceed, Fall, Diffusion, Division and Put. Each describes an irreversible or causal process, and the categories may overlap.
  • Models tested:
    • Proprietary non-reasoning: GPT-4o, GPT-4.1.
    • Proprietary reasoning: o3, o4-mini, GPT-5, Gemini-2.5-Pro.
    • Open-weight non-reasoning: Qwen2-VL, Qwen2.5-VL.
    • Open-weight reasoning: Cosmos-Reason1-7B (trained with explicit AoT supervision), QVQ-72B-Preview.
  • Result: Models perform near chance with a bias toward answering forward. The best setting remains about 29 points below humans, even on physically irreversible events.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/58hxjifrhvw7

@inproceedings{matta2026which,
  title = {{Which Way Does Time Flow? A Psychophysics-Grounded Evaluation for Vision–Language Models}},
  author = {Matta, Shiho and Pereira, Lis Kanashiro and Han, Peitao and Kitazawa, Shigeru and Cheng, Fei},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/58hxjifrhvw7},
  url = {https://doi.org/10.63317/58hxjifrhvw7}
}

References

23 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. 2025. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575. — [arXiv:2501.03575](https://arxiv.org/abs/2501.03575) find
  2. Alisson Azzolini, Junjie Bai, Hannah Brandon, Jiaxin Cao, Prithvijit Chattopadhyay, Huayu Chen, Jinju Chu, Yin Cui, Jenna Diamond, Yifan Ding, et al. 2025. Cosmos-reason1: From physical common sense to embodied reasoning. arXiv preprint arXiv:2503.15558. — [arXiv:2503.15558](https://arxiv.org/abs/2503.15558) find
  3. Piyush Bagad, Makarand Tapaswi, and Cees GM Snoek. 2023. Test of time: Instilling video-language models with a sense of time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2503–2516. find
  4. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. find our card
  5. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv prep… find our card
  6. Yang Du, Yuqi Liu, and Qin Jin. 2024. Reversed in time: A novel temporal-emphasized benchmark for cross-modal video-text retrieval. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 5260–5269. find
  7. Arthur Eddington. 2019. The nature of the physical world: THE GIFFORD LECTURES 1927, volume 23. BoD–Books on Demand. find
  8. Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Caifeng Shan, Ran He, and Xing Sun. 2025. Video-mme: The first-ever comprehensive… find our card
  9. Nao Hanyu, Kei Watanabe, and Shigeru Kitazawa. 2023. Ready to detect a reversal of time’s arrow: a psychophysical study using short video clips in daily scenes. Royal Society open science, 10(4):230036. find
  10. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. find
  11. Haowei Liu, Xi Zhang, Haiyang Xu, Yaya Shi, Chaoya Jiang, Ming Yan, Ji Zhang, Fei Huang, Chunfeng Yuan, Bing Li, and Weiming Hu. 2024. MIBench: Evaluating multimodal large language models over multiple images. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,… find
  12. Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfreund, Carl Vondrick, et al. 2019. Moments in time dataset: one million videos for event understanding. IEEE transactions on pattern analysis and machine intelligence, 42(2):50… find
  13. OpenAI. 2024. Hello gpt-4o. https://openai.com/index/hello-gpt-4o/. ↗
  14. OpenAI. 2025a. Introducing gpt-4.1 in the api. find
  15. OpenAI. 2025b. Introducing gpt-5. find
  16. OpenAI. 2025c. Introducing openai o3 and o4-mini. https://openai.com/index/introducing-o3-and-o4-mini/. ↗
  17. Qwen Team. 2024. Qvq: Qwen2-vl with visual question answering. Technical report, Alibaba Cloud. find
  18. Fei Wang, Xingyu Fu, James Y. Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, Tianyi Lorena Yan, Wenjie Jacky Mo, Hsiang-Hui Liu, Pan Lu, Chunyuan Li, Chaowei Xiao, Kai-Wei Chang, Dan Roth, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2024a. Muirbench: A compr… find
  19. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024b. Qwen2-vl: Enhancing vision-language model’s perception of the… find our card
  20. Zhenhailong Wang, Ansel Blume, Sha Li, Genglin Liu, Jaemin Cho, Zineng Tang, Mohit Bansal, and Heng Ji. 2023. Paxion: Patching action knowledge in video-language foundation models. Advances in Neural Information Processing Systems, 36:20729–20749. find
  21. Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. 2024. Star: A benchmark for situated reasoning in real-world videos. find
  22. Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. 2021. Next-qa:next phase of question-answering to explaining temporal actions. find
  23. Zihui Xue, Mi Luo, and Kristen Grauman. 2025. Seeing the arrow of time in large multimodal models. arXiv preprint arXiv:2506.03340. — [arXiv:2506.03340](https://arxiv.org/abs/2506.03340) find

↑ back to top

This page in Markdown →