Elamur Research⌕
☰
home / papers / 2026 / This One or That One? A Study on Accessibility via Demonstratives with Multimodal Large Language Models

This One or That One? A Study on Accessibility via Demonstratives with Multimodal Large Language Models

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC 4.0 (ELRA, LREC proceedings)" does not allow redistributing the text itself. Original — aclanthology.org/2026.lrec-1.763.pdf

Summary

The paper examines whether multimodal large language models (MLLMs) can pick out a physical object among several look-alike items using only demonstratives such as "this"/"that" (Chinese 这/那). Here, accessibility means how easily a speaker can reach an object, and it depends on perspective. The same object can be "this" for one person and "that" for someone facing them. The authors argue that existing embodied and referring-expression benchmarks, such as RefCOCO and PentoRef, rely on visually distinctive targets. They built a small, carefully controlled bilingual dataset of first-person scene images paired with single-choice questions. The questions come in pairs, so a model must answer both members of a pair correctly to show genuine understanding rather than a lucky guess. Advanced models, including GPT-4.1 and Gemini 2.5 Pro, almost entirely fail on the items that require a perspective shift, while human participants exceed 80%. The authors conclude that MLLMs lean on textual co-occurrence and lack a self–other distinction and embodied grounding. They see this as a concern for embodied AI deployment.

Key points

  • Dataset size: 200 items, with 100 visual questions each in English and Chinese, built around 10 situations (pictures) per cue type.
  • Cue types and counts: Accessibility has 80 items (4 questions per pair). Physical Uniqueness, Qualitative Characteristics and Frame-based knowledge have 40 items each.
  • Question format: Each question gives a situation, a context, a character's instruction and a question. There are four answer options: red box, blue box, green box, or "Not here".
  • Perspective-shift results: Model accuracy is about 2.33% for Chinese and 1.83% for English, against over 80% for humans.
  • Other weaknesses: Models also struggle with qualitative-characteristic and frame-based cues. They seem to apply implicit contextual rules only when those rules are explicitly present in training data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

7 papers of our library cite it (of the 140673 whose bibliographies we read).

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/29f29zththay

@inproceedings{wang2026this,
  title = {{This One or That One? A Study on Accessibility via Demonstratives with Multimodal Large Language Models}},
  author = {Wang, Yu and Chersoni, Emmanuele and Huang, Chu-Ren},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/29f29zththay},
  url = {https://doi.org/10.63317/29f29zththay}
}

References

39 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alibaba DAMO Academy. 2025. Qwen2.5-vl-32b. Accessed: 2025-05-25. find
  2. Meta AI. 2025. Llama 4: Multimodal intelligence. Accessed: 2025-05-25. find
  3. Anthropic. 2024. The Claude 3 Model Family: Opus, Sonnet, Haiku. find
  4. Mira Ariel. 1990. Accessing Noun-phrase Antecedents. Routledge, New York, USA. find
  5. Shuanghao Bai, Wanqi Zhou, Pengxiang Ding, Wei Zhao, Donglin Wang, and Badong Chen. 2025. Rethinking Latent Redundancy in Behavior Cloning: An Information Bottleneck Approach for Robot Manipulation. arXiv preprint arXiv:2502.02853. — [arXiv:2502.02853](https://arxiv.org/abs/2502.02853) find
  6. Emily M Bender and Alexander Koller. 2020. Climbing Towards NLU: On Meaning, Form, and Understanding in the Age of Data. In Proceedings of ACL. find our card
  7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Christ… find
  8. Zhili Cheng, Yuge Tu, Ran Li, Shiqi Dai, Jinyi Hu, Shengding Hu, Jiahao Li, Yang Shi, Tianyu Yu, Weize Chen, Lei Shi, and Maosong Sun. 2025. EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agents. arXiv preprint arXiv:2501.11858. — [arXiv:2501.11858](https://arxiv.org/abs/2501.11858) find
  9. Ronghao Dang, Yuqian Yuan, Wenqi Zhang, Yifei Xin, Boqiang Zhang, Long Li, Liuyi Wang, Qinyang Zeng, Xin Li, and Lidong Bing. 2025. ECBench: Can Multi-modal Foundation Models Understand the Egocentric World? A Holistic Embodied Cognition Benchmark. In Proceedings of the Computer Vision and Pattern R… find
  10. Mingyu Du, Bowen Wu, Zhengyuan Li, Xuancheng Huang, and Zhengjue Wei. 2024. EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Models. In Proceedings of ACL. find
  11. Jiale Duan, Shuai Yu, Hao L Tan, Hongbo Zhu, and Cheston Tan. 2022. A Survey of Embodied AI: From Simulators to Research Tasks. arXiv preprint arXiv:2103.04918. — [arXiv:2103.04918](https://arxiv.org/abs/2103.04918) find
  12. Charles J Fillmore. 2006. Frame Semantics. In Cognitive Linguistics: Basic Readings, volume 34, pages 373–400. find
  13. Google DeepMind. 2025. Gemini 2.5 Pro. https://deepmind.google/models/gemini/pro/. Accessed: 2025-05-25. ↗
  14. Jennifer Hu, Felix Sosa, and Tomer Ullman. 2025. Re-evaluating Theory of Mind evaluation in Large Language Models. Philosophical Transactions B, 380(1932):20230499. find
  15. Chu-Ren Huang. 2010. Ontology and the Lexicon: A Natural Language Processing Perspective. Cambridge University Press, Cambridge, United Kingdom. find
  16. Cory R Jones and Sean Trott. 2024. Multimodal Language Models Show Evidence of Embodied Simulation. In Proceedings of LREC-COLING. find
  17. Carina Kauf, Emmanuele Chersoni, Alessandro Lenci, Evelina Fedorenko, and Anna A Ivanova. 2024. Log Probabilities Are a Reliable Estimate of Semantic Plausibility in Base and Instruction-Tuned Language Models. In Proceedings of the EMNLP Workshop on Analyzing and Interpreting Neural Networks for NLP… find
  18. Carina Kauf, Anna A Ivanova, Giulia Rambelli, Emmanuele Chersoni, Jingyuan Selena She, Zawad Chowdhury, Evelina Fedorenko, and Alessandro Lenci. 2023. Event Knowledge in Large Language Models: The Gap Between the Impossible and the Unlikely. Cognitive Science, 47(11):e13386. find
  19. Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. 2014. ReferItGame: Referring to Objects in Photographs of Natural Scenes. In Proceedings of EMNLP. find
  20. Sotaro Kita. 2003. Pointing: Where Language, Culture, and Cognition Meet. Psychology Press, Hove, East Sussex, England, UK. find
  21. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large Language Models Are Zero-Shot Reasoners. In Advances in Neural Information Processing Systems. find our card
  22. Aoxue Kong, Shuofei Zhao, Hong Chen, Qian Li, Yujia Qin, Ruoxi Sun, Yuxian Zhang, and Xin Dong. 2023. Better Zero-Shot Reasoning with Role-Play Prompting. arXiv preprint arXiv:2308.07702. find
  23. Michal Kosinski. 2023. Theory of Mind May Have Spontaneously Emerged in Large Language Models. arXiv preprint arXiv:2302.02083, 4:169. find our card
  24. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The Winograd Schema Challenge. In Proceedings of the International Conference on the Principles of Knowledge Representation and Reasoning. find
  25. Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Mart’ın-Mart’ın, Chengzhi Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, et al. 2023. BEHAVIOR-1K: A Benchmark for Embodied AI with 1,000 Everyday Activities and Realistic Simulation. In Proceedings of the Confer… find
  26. Xun Liu and Zhengwei Ni. 2024. Role-playing Prompt Framework: Generation and Evaluation. arXiv preprint arXiv:2406.00627. — [arXiv:2406.00627](https://arxiv.org/abs/2406.00627) find
  27. Arjun Majumdar, Anurag Ajay, Xingyu Zhang, Pranav Putta, Santhosh Yenamandra, Mikael Henaff, Saksham Silwal, Patrick McVay, Oleksandr Maksymets, and Simone Arnaud. 2024. OpenEQA: Embodied Question Answering in the Era of Foundation Models. In Proceedings of the IEEE/CVF Conference on Computer Vision… find
  28. Ken McRae and Kazunaga Matsuki. 2009. People Use their Knowledge of Common Events to Understand Language, and Do So as Quickly as Possible. Language and Linguistics Compass, 3(6):1417–1429. find
  29. OpenAI. 2023. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774. find our card
  30. OpenAI. 2025. Introducing GPT-4.1 in the API. Accessed: 2025-05-25. find
  31. Paolo Pedinotti, Giulia Rambelli, Emmanuele Chersoni, Enrico Santus, Alessandro Lenci, and Philippe Blache. 2021. Did the Cat Drink the Coffee? Challenging Transformers with Generalized Event Knowledge. In Proceedings of the Tenth Joint Conference on Lexical and Computational Semantics (*SEM 2021). find
  32. James Pustejovsky. 1998. The Generative Lexicon. MIT Press, Cambridge, Massachusetts, USA. find
  33. Paula Rubio-Fernandez. 2022. Demonstrative Systems: From Linguistic Typology to Social Cognition. Cognitive Psychology, 139:101519. find
  34. James WA Strachan, Dalila Albergo, Giulia Borghini, Oriana Pansardi, Eugenio Scaliti, Saurabh Gupta, Krati Saxena, Alessandro Rufo, Stefano Panzeri, Guido Manzi, et al. 2024. Testing Theory of Mind in Large Language Models and Humans. Nature Human Behaviour, 8(7):1285–1295. find
  35. Tao Wu, Chuhao Zhou, Yen Heng Wong, Lin Gu, and Jianfei Yang. 2024. NoisyEQA: Benchmarking Embodied Question Answering Against Noisy Queries. arXiv preprint arXiv:2412.10726. — [arXiv:2412.10726](https://arxiv.org/abs/2412.10726) find
  36. Bingyu Yan, Zhibo Zhou, Litian Zhang, Lian Zhang, Ziyi Zhou, Dezhuang Miao, Zhoujun Li, Chaozhuo Li, and Xiaoming Zhang. 2025. Beyond Self-talk: A Communication-centric Survey of LLM-based Multi-agent Systems. arXiv preprint arXiv:2502.14321. — [arXiv:2502.14321](https://arxiv.org/abs/2502.14321) find
  37. Yulin Yuan. 2013. Research on Semantic Knowledge System Based on Generative Lexicon Theory and Argument Structure Theory. Journal of Chinese Information Processing, 27(6):23–30. find
  38. Sina Zarrieß, Julian Hough, Casey Kennington, Ramesh Manuvinakurike, David DeVault, Raquel Fernández, and David Schlangen. 2016. PentoRef: A Corpus of Spoken References in Task-oriented Dialogues. In Proceedings of LREC. find
  39. Yong Zhao, Kai Xu, Zhengqiu Zhu, Yue Hu, Zhiheng Zheng, Yingfeng Chen, Yatai Ji, Chen Gao, Yong Li, and Jincai Huang. 2025. CityEQA: A Hierarchical LLM Agent on Embodied Question Answering Benchmark in City Space. arXiv preprint arXiv:2502.12532. find

↑ back to top