Elamur Research⌕
☰
home / papers / 2026 / Similar Predictions, Different Processes: A Multi-Level Comparison of Human and Multimodal LLM Language Prediction

Similar Predictions, Different Processes: A Multi-Level Comparison of Human and Multimodal LLM Language Prediction

Summary

This study asks whether a multimodal LLM that hears speech and sees images resembles humans only in which referent it anticipates, or also in when that anticipation arises. The authors tested native Mandarin speakers and Qwen2.5-Omni-7B on Mandarin dative sentences in a visual world paradigm (VWP). The design crossed word order (double-object vs. prepositional-object) with the placement of contrastive stress (none, on the theme, or on the recipient). In an offline cloze-in-VWP experiment, each system heard a sentence cut off after the connective "but not" and picked which of four pictures would come next. An online experiment then compared human eye movements with the model's audio-to-image cross-modal attention. Both humans and the model were sensitive to structure and prosody. In both, stress had its clearest effect on constituents that lacked default focus. However, the model leaned more heavily on structure and showed some atypical prosody effects. Online, humans showed structural effects before the connective, while the model showed them only afterwards. Similar outputs therefore need not reflect similar processing, which constrains arguments that multimodal LLMs are cognitively plausible models of human processing.

Key points

  • Materials: 36 items (selected from 50 after norming), arranged in a 2 × 3 Structure × Stress design across six Latin-square lists, with four-image displays showing the mentioned and alternative theme and recipient.
  • Experiment 1 involved 118 native speakers, tested online via Gorilla. The model side used Qwen2.5-Omni-7B with 20 seeded generations per item, each analysed as a separate trial in a GLMM.
  • Experiment 2 tracked 58 participants with an EyeLink 1000 Plus at 1000 Hz, using 50 ms bins aligned to target-word onset. The model-side measure, cross-modal attention, is treated as a proxy rather than an equivalent of gaze.
  • The model's responses fell outside the task-defined alternatives far more often than humans' did, with exclusion rates of 33.3% vs. 5.4%.
  • Stimuli, audio, model outputs, the attention-extraction pipeline and anonymized human data are released at github.com/wang-shuqi/conll2026-vwp-qwen-prediction.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.conll-main.6

@inproceedings{wang2026similar,
  title = {{Similar Predictions, Different Processes: A Multi-Level Comparison of Human and Multimodal LLM Language Prediction}},
  author = {Wang, Shuqi and Cai, Zhenguang},
  booktitle = {Proceedings of the 30th Conference on Computational Natural Language Learning},
  year = {2026},
  doi = {10.18653/v1/2026.conll-main.6},
  url = {https://doi.org/10.18653/v1/2026.conll-main.6}
}

References

48 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gerry TM Altmann and Yuki Kamide. 1999. Incremental interpretation at verbs: Restricting the domain of subsequent reference. Cognition, 73(3):247–264. find
  2. Alexander L. Anwyl-Irvine, Jessica Massonnié, Adam Flitton, Natasha Kirkham, and Jo K. Evershed. 2020. Gorilla in our midst: An online behavioral experiment builder. Behavior Research Methods, 52(1):388–407. find
  3. Manabu Arai, Roger P.G. van Gompel, and Christoph Scheepers. 2007. Priming ditransitive structures in comprehension. Cognitive Psychology, 54(3):218– 250. find
  4. Tiwalayo Eisape, Noga Zaslavsky, and Roger Levy. 2020. Cloze distillation: Improving neural language models with human next-word prediction. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 609–619, Online. Association for Computational Linguistics. find
  5. Behrooz Azarkhalili and Maxwell W. Libbrecht. 2025. Generalized attention flow: Feature attribution for transformer models via maximum flow. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 19954–19974, Vienna, Austria. Associa… find
  6. Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019. Neural language models as psycholinguistic subjects: Representations of syntactic state. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Lingui… find our card
  7. Dale J. Barr. 2008. Analyzing ‘visual world’ eyetracking data using multilevel logistic regression. Journal of Memory and Language, 59(4):457–474. find
  8. Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2015. Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1):1– 48. find our card
  9. Changjiang Gao, Zhengwu Ma, Jiajun Chen, Ping Li, Shujian Huang, and Jixing Li. 2025. Increasing alignment of large language models with language processing in the human brain. Nature Computational Science, 5(11):1080–1090. find
  10. Zhenguang G. Cai, Martin J. Pickering, and Patrick Sturt. 2013. Processing verb-phrase ellipsis in Mandarin Chinese: Evidence against the syntactic account. Language and Cognitive Processes, 28(6):810–828. find
  11. Ariel Goldstein, Eric Ham, Mariano Schain, Samuel A. Nastase, Bobbi Aubrey, Zaid Zada, Avigail Grinstein-Dabush, Harshvardhan Gazula, Amir Feder, Werner Doyle, Sasha Devore, Patricia Dugan, Daniel Friedman, Michael Brenner, Avinatan Hassidim, Yossi Matias, Orrin Devinsky, Noam Siegelman, Adeen Flink… find
  12. Zhenguang G. Cai, Nan Zhao, and Martin J. Pickering. 2022. How do people interpret implausible sentences? Cognition, 225:105101. find
  13. Sasha Calhoun. 2010. The centrality of metrical structure in signaling information structure: A probabilistic perspective. Language, 86(1):1–42. find
  14. Charlotte Caucheteux and Jean-Rémi King. 2022. Brains and algorithms partially converge in natural language processing. Communications Biology, 5(1):134. find
  15. Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A. Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, Aren Jansen, Harshvardhan Gazula, Gina Choe, Aditi Rao, Catherine Kim, Colton Casto, Lora Fanda, Werner Doyle, Daniel Friedman, and 13 others. 2022. Shared co… find
  16. Charles Clifton, Jr. and Lyn Frazier. 2016. Focus in corrective exchanges: Effects of pitch accent and syntactic form. Language and Speech, 59(4):544–561. find
  17. Ruth E. Corps, Charlotte Brooke, and Martin J. Pickering. 2022. Prediction involves two stages: Evidence from visual-world eye-tracking. Journal of Memory and Language, 122:104298. find
  18. Olivia Guest and Andrea E. Martin. 2023. On logical inference over brains, behaviour, and artificial neural networks. Computational Brain & Behavior, 6(2):213–227. find
  19. Ruth E. Corps, Meijian Liao, and Martin J. Pickering. 2023. Evidence for two stages of prediction in non-native speakers: A visual-world eye-tracking study. Bilingualism: Language and Cognition, 26(1):231– 243. find
  20. Jesse A. Harris. 2023. The enduring effects of default focus in let alone ellipsis: Evidence from pupillometry. In Proceedings of the Second Experiments in Linguistic Meaning Conference (ELM 2), volume 2, pages 117–128, Philadelphia, PA. Linguistic Society of America. find
  21. Delphine Dahan, Michael K. Tanenhaus, and Craig G. Chambers. 2002. Accent and reference resolution in spoken-language comprehension. Journal of Memory and Language, 47(2):292–314. find
  22. Jesse A. Harris and Katy Carlson. 2018. Information structure preferences in focus-sensitive ellipsis: How defaults persist. Language and Speech, 61(3):480– 512. find
  23. Oliver Eberle, Stephanie Brandl, Jonas Pilot, and Anders Søgaard. 2022. Do transformer models show similar attention patterns to task-specific human gaze? In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4295–4309, Dublin, Irel… find
  24. Falk Huettig, Joost Rommers, and Antje S. Meyer. 2011. Using the visual world paradigm to study language processing: A review and critical evaluation. Acta Psychologica, 137(2):151–171. find
  25. Aine Ito and Pia Knoeferle. 2023. Analysing data from the psycholinguistic visual-world paradigm: Comparison of different analysis methods. Behavior Research Methods, 55(7):3461–3493. find
  26. Ekta Sood, Simon Tannert, Diego Frassinelli, Andreas Bulling, and Ngoc Thang Vu. 2020. Interpreting attention models with human visual attention in machine reading comprehension. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 12–25, Online. Association for Co… find
  27. Aine Ito, Martin J. Pickering, and Martin Corley. 2018. Investigating the time-course of phonological prediction in native and non-native speakers of English: A visual world eye-tracking study. Journal of Memory and Language, 98:1–11. find
  28. Michael K. Tanenhaus, Michael J. Spivey-Knowlton, Kathleen M. Eberhard, and Julie C. Sedivy. 1995. Integration of visual and linguistic information in spoken language comprehension. Science, 268(5217):1632–1634. find
  29. Kiwako Ito and Shari R. Speer. 2008. Anticipatory effects of intonation: Eye movements during instructed visual search. Journal of Memory and Language, 58(2):541–573. find
  30. Wilson L. Taylor. 1953. “Cloze procedure”: A new tool for measuring readability. Journalism Quarterly, 30(4):415–433. find
  31. Gina R. Kuperberg and T. Florian Jaeger. 2016. What do we mean by prediction in language comprehension? Language, Cognition and Neuroscience, 31(1):32–59. find
  32. Jacolien van Rij, Petra Hendriks, Hedderik van Rijn, R. Harald Baayen, and Simon N. Wood. 2019. Analyzing the time course of pupillometric data. Trends in Hearing, 23:2331216519832483. find
  33. Weijun Li, Nali Deng, Yufang Yang, and Lin Wang. 2018. Process focus and accentuation at different positions in dialogues: An ERP study. Language, Cognition and Neuroscience, 33(2):255–274. find
  34. Jacolien van Rij, Martijn Wieling, R. Harald Baayen, and Hedderik van Rijn. 2015. itsadug: Interpreting Time Series and Autocorrelated Data Using GAMMs. R package. find
  35. Zhicheng Lin. 2025. Six fallacies in substituting large language models for human participants. Advances in Methods and Practices in Psychological Science, 8(3):25152459251357566. find
  36. Andrea Weber, Martine Grice, and Matthew W. Crocker. 2006. The role of prosody in the interpretation of structural ambiguities: A study of anticipatory eye movements. Cognition, 99(2):B63–B72. find
  37. Tal Linzen and Marco Baroni. 2021. Syntactic structure from deep learning. Annual Review of Linguistics, 7(1):195–212. find
  38. Ethan Gotlieb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger P. Levy. 2020. On the predictive power of neural language models for human real-time comprehension behavior. In Proceedings of the 42nd Annual Meeting of the Cognitive Science Society, pages 1707–1713. find
  39. Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2024. Dissociating language and thought in large language models. Trends in Cognitive Sciences, 28(6):517–540. find
  40. Simon N. Wood. 2017. Generalized Additive Models: An Introduction with R, 2nd edition. Chapman and Hall/CRC. find
  41. David Marr. 1982. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. W.H. Freeman, San Francisco. find
  42. Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. 2025. Qwen2.5-omni technical report. Preprint, arXiv:2503.20215. find our card
  43. Evelyn Milburn, Tessa Warren, and Michael Walsh Dickey. 2016. World knowledge affects prediction as quickly as selectional restrictions: Evidence from the visual world paradigm. Language, Cognition and Neuroscience, 31(4):536–548. find
  44. Liejiong Xu. 2004. Manifestation of informational focus. Lingua, 114(3):277–299. find
  45. Martin J. Pickering and Chiara Gambi. 2018. Predicting while comprehending language: A theory and review. Psychological Bulletin, 144(10):1002–1044. find
  46. Mengzhu Yan and Sasha Calhoun. 2020. Rejecting false alternatives in Chinese and English: The interaction of prosody, clefting, and default focus position. Laboratory Phonology, 11(1):17. find
  47. Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A. Hosseini, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2021. The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 118(45)… find
  48. Jayden Ziegler and Jesse Snedeker. 2019. The use of syntax and information structure during language comprehension: Evidence from structural priming. Language, Cognition and Neuroscience, 34(3):365– 384. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-similar-predictions-different-processes-multi/