Elamur Research⌕
☰
home / papers / 2026 / Common Objects Out of Context (COOCo): Investigating Multimodal Context and Semantic Scene Violations in Referential Communication

Common Objects Out of Context (COOCo): Investigating Multimodal Context and Semantic Scene Violations in Referential Communication

Paper · 2026 · Filippo Merlo, Ece Takmaz, Wenkai Chen et al.

Summary

The paper asks how much vision-language models (VLMs) draw on surrounding scene context when naming an object, and whether this depends on how well the object fits the scene. The authors built COOCo from COCO-Search18 photos. They used an inpainting pipeline to replace a target object with alternatives whose relatedness to the scene was graded as low, medium or high, plus same-category replacements as controls. Several VLMs were tested on object naming, with Gaussian noise applied either to the target or to the context. Models relied on context selectively. When the object fit the scene, context helped and made models robust to a degraded target. When the object clashed with the scene, context acted as a distractor, and hiding it improved performance. An attention analysis of LLaVA-OneVision showed that correct naming went with stronger mid-layer attention to the target. Attention also varied with semantic fit in a U-shaped way, resembling human eye-tracking findings. This matters for building and interpreting robust referring-expression systems.

Key points

  • COOCo has 18,395 images built from 2,241 retained COCO-Search18 images. These include 1,862 originals with object-removed versions, 5,480 low-relatedness images, 5,572 medium-relatedness images and 1,771 high-relatedness images.
  • Scene types were assigned by a ViT finetuned on SUN-397, reduced to 29 labels.
  • Replacement objects came from THINGSplus norms, filtered for typicality and real-world size. Their scene relatedness was scored by cosine similarity of ConceptNet Numberbatch embeddings.
  • In LLaVA-OneVision-0.5B, correct categorisation was associated with higher attention to the target in layers 7–15. Quadratic fits of attention against semantic fit were convex, with the minimum near 0.764 on COOCo.
  • The findings were replicated on the human-validated VISIONS dataset (1,136 images), evaluated with RefCLIPScore and accuracy. Moderate target noise (0.5) increased attention to the target, while full noise (1.0) pushed reliance onto context.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TrentoTrento · Italy85 works · 15,524 citations🎓Utrecht UniversityUtrecht · Netherlands39 works · 1,603 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 657 works

Cite

Cite as Filippo Merlo, Ece Takmaz, Wenkai Chen and Albert Gatt. 2026. Common Objects Out of Context (COOCo): Investigating Multimodal Context and Semantic Scene Violations in Referential Communication. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl.a.701

@article{merlo2026common,
  title = {{Common Objects Out of Context (COOCo): Investigating Multimodal Context and Semantic Scene Violations in Referential Communication}},
  author = {Filippo Merlo, Ece Takmaz, Wenkai Chen and Albert Gatt},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.1162/tacl.a.701},
  url = {https://doi.org/10.1162/tacl.a.701}
}

References

59 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Elena Allegretti, Giorgia D’Innocenzo, and Moreno I. Coco. 2025. The Visual Integration of Semantic and Spatial Information of Objects in Naturalistic Scenes (VISIONS) database: attentional, conceptual, and perceptual norms. Behavior Research Methods, 57(1):42. find
  2. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Ha… — Qwen2.5-VL Technical Report
  3. Moshe Bar. 2004. Visual objects in context. Nature Reviews Neuroscience, 5(8):617–629. find
  4. Irving Biederman, Robert J. Mezzanotte, and Jan C. Rabinowitz. 1982. Scene perception: Detecting and judging objects undergoing relational violations. Cognitive Psychology, 14(2):143–177. find
  5. Sage E. P. Boettcher, Dejan Draschkow, Eric Di-enhart, and Melissa L.-H. Võ. 2018. Anchoring visual search in scenes: Assessing the role of anchor objects on eye movements during visual search. Journal of Vision, 18(13):11. find
  6. Philipp Bomatter, Mengmi Zhang, Dimitar Karev, Spandan Madan, Claire Tseng, and Gabriel Kreiman. 2021. When Pigs Fly: Contextual Reasoning in Synthetic and Natural Scenes. ArXiv:2104.02215 [cs]. — [arXiv:2104.02215](https://arxiv.org/abs/2104.02215) find
  7. Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. 2022. WebQA: Multihop and Multimodal QA. ArXiv:2109.00590 [cs]. — [arXiv:2109.00590](https://arxiv.org/abs/2109.00590) find
  8. Yupei Chen, Zhibo Yang, Seoyoung Ahn, Dimitris Samaras, Minh Hoai, and Gregory Zelinsky. 2021. COCO-Search18 fixation dataset for predicting goal-directed attention control. Scientific Reports, 11(1):8776. find
  9. Alasdair D F Clarke, Moreno I. Coco, and Frank Keller. 2013. The impact of attentional, linguistic, and visual features during object naming. Frontiers in Psychology, 4(DEC):1–12. find
  10. Moreno I. Coco, Frank Keller, and George L. Malcolm. 2016. Anticipation in real-world scenes: The role of visual context and visual memory. Cognitive Science, 40(8):1995–2024. find
  11. Claudia Damiano, Maarten Leemans, and Johan Wagemans. 2024. Exploring the Semantic-Inconsistency Effect in Scenes Using a Continuous Measure of Linguistic-Semantic Similarity. Psychological Science, 35(6):623–634. find
  12. Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, Yen-Sung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, Andrew Head, Rose Hen… — Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
  13. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ArXiv:201…
  14. Haley C. Dresang, Michael Walsh Dickey, and Tessa C. Warren. 2019. Semantic memory for objects, actions, and events: A novel test of event-related conceptual semantic knowledge. Cognitive Neuropsychology, 36(7-8):313–335. find
  15. John M. Henderson, Taylor R. Hayes, Candace E. Peacock, and Gwendolyn Rehrig. 2019. Meaning and Attentional Guidance in Scenes: A Review of the Meaning Map Approach. Vision, 3(2):19. find
  16. Albert End and Matthias Gamer. 2017. Preferential processing of social features and their interplay with physical saliency in complex naturalistic scenes. Frontiers in Psychology, 8(March):418. find
  17. Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2022. CLIPScore: A Reference-free Evaluation Metric for Image Captioning. ArXiv:2104.08718.
  18. Giorgio Ganis and Marta Kutas. 2003. An electrophysiological study of scene effects on object identification. Cognitive Brain Research, 16(2):123–144. find
  19. Michal Golovanevsky, William Rudman, Vedant Palit, Carsten Eickhoff, and Ritambhara Singh. 2025. What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of… find
  20. Floor de Groot, Falk Huettig, and Christian N. L. Olivers. 2015. When Meaning Matters: The Temporal Dynamics of Semantic Influences on Visual Attention. Journal of Experimental Psychology: Human Perception and Performance. find
  21. Floor de Groot, Falk Huettig, and Christian N.L. Olivers. 2017. Language-induced visual and semantic biases in visual search are subject to task requirements. Visual Cognition, 0(0):1–16. Publisher: Taylor & Francis. find
  22. Taylor R. Hayes and John M. Henderson. 2021. Looking for semantic similarity: What a vector-space model of semantics can tell us about attention in real-world scenes. Psychological Science, 32(8):1262–1270. PMID: 34252325. find
  23. Martin N. Hebart, Adam H. Dickter, Alexis Kidder, Wan Y. Kwok, Anna Corriveau, Caitlin Van Wicklin, and Chris I. Baker. 2019. THINGS: A database of 1,854 object concepts Alex D. Hwang, Hsueh-Cheng Wang, and Marc Pomplun. 2011. Semantic guidance of eye movements in real-world scenes. Vision Research,… find
  24. Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. 2017. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference. ArXiv:1712.05877 [cs]. find
  25. Simeon Junker and Sina Zarrieß. 2024. Resilience through scene context in visual referring expression generation. In Proceedings of the 17th International Natural Language Generation Conference, pages 344–357, Tokyo, Japan. Association for Computational Linguistics. find
  26. Simeon Junker and Sina Zarrieß. 2025. Scene-Gram: Conceptualizing and Describing Tangrams in Scene Context. arXiv preprint arXiv:2506.11631. — [arXiv:2506.11631](https://arxiv.org/abs/2506.11631) find
  27. Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. 2014. ReferItGame: Referring to Objects in Photographs of Natural Scenes. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 787–798, Doha, Qatar. Association for Computational Ling…
  28. Emiel Krahmer and Kees Van Deemter. 2012. Computational Generation of Referring Expressions: A Survey. Computational Linguistics, 38(1):173–218.
  29. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Fei-Fei Li. 2016. Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations. ArXiv:1602.07332 [c…
  30. Tim Lauer, Tim H. W. Cornelissen, Dejan Draschkow, Verena Willenbockel, and Melissa L.-H. Võ. 2018. The role of scene summary statistics in object recognition. Scientific Reports, 8(1):14666. find
  31. Tim Lauer, Filipp Schmidt, and Melissa L.-H. Võ. 2021. The role of contextual materials in object recognition. Scientific Reports, 11(1):21988. find
  32. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. ArXiv:2103.00020 [cs].
  33. Simeon Schüz, Albert Gatt, and Sina Zarrieß. 2023. Rethinking symbolic and visual context in Referring Expression Generation. Frontiers in Artificial Intelligence, 6:1067125. find
  34. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2025. LLaVA-onevision: Easy visual task transfer. Transactions on Machine Learning Research.
  35. Simeon Schüz and Sina Zarrieß. 2023. Keeping an Eye on Context: Attention Allocation over Input Partitions in Referring Expression Generation. In Proceedings of the Workshop on Multimodal, Multilingual Natural Language Generation and Multilingual WebNLG Challenge (MM-NLG 2023), pages 20–27, Prague,… find
  36. Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. 2015. Microsoft COCO: Common Objects in Context. ArXiv:1405.0312 [cs].
  37. Yarden Shir, Naphtali Abudarham, and Liad Mu-drik. 2021. You won’t believe what this guy is doing with the potato: The ObjAct stimulus-set depicting human actions on congruent and incongruent objects. Behavior Research Methods, 53(5):1895–1909. find
  38. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruction Tuning. ArXiv:2304.08485 [cs].
  39. Robyn Speer, Joshua Chin, and Catherine Havasi. 2018. ConceptNet 5.5: An Open Multilingual Graph of General Knowledge. ArXiv:1612.03975 [cs]. find
  40. Sabine Öhlschläger and Melissa L.-H. Võ. 2016. Scegram: An image database for semantic and syntactic inconsistencies in scenes. Behavior Research Methods, 49:1780–1791. find
  41. Aude Oliva and Antonio Torralba. 2007. The role of context in object recognition. Trends in Cognitive Sciences, 11(12):520–527. find
  42. Laura M. Stoinski, Jonas Perkuhn, and Martin N. Hebart. 2023. THINGSplus: New norms and metadata for the THINGS database of 1854 object concepts and 26,107 natural object images. Behavior Research Methods. find
  43. Marius V. Peelen, Eva Berlot, and Floris P. De Lange. 2023. Predictive processing of scenes and objects. Nature Reviews Psychology, 3(1):13–26. find
  44. Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. 2021. Resolution-robust Large Mask Inpainting with Fourier Convolutions. ArXiv:2109.07161 [cs]. — [arXiv:2109.07161](https://arxiv.org/abs/2109.07161) find
  45. Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, Qixiang Ye, and Furu Wei. 2024. Grounding multimodal large language models to the world. In The Twelfth International Conference on Learning Representations. find
  46. Antonio Torralba, Aude Oliva, Monica S. Castel-hano, and John M. Henderson. 2006. Contextual guidance of eye movements and attention in real-world scenes: The role of global features in object search. Psychological Review, 113(4):766–786. find
  47. Jesse Vig and Yonatan Belinkov. 2019. Analyzing the Structure of Attention in a Transformer Language Model. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 63–76, Florence, Italy. Association for Computational Linguistics.
  48. Melissa Le-Hoa Võ. 2021. The meaning and structure of scenes. Vision Research, 181:10– 20. find
  49. Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. 2010. SUN database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 3485–3492, San Francisco, CA, USA. IEEE. find
  50. Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S Ryoo, Shrikant Kendre, Jieyu Zhang, Can Qin, Shu Zhang, Chia-Chih Chen, Ning Yu, Juntao Tan, Tulika Manoj Awalgaonkar, Shelby Heinecke, Huan Wang, Yejin Choi, Ludwig Schmidt, Ze… find
  51. Tianze Yang, Tyson Jordan, Ninghao Liu, and Jin Sun. 2025. Common Inpainted Objects In-N-Out of Context. ArXiv:2506.00721 [cs]. — [arXiv:2506.00721](https://arxiv.org/abs/2506.00721) find
  52. Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. 2016. Modeling Context in Referring Expressions. In ECCV. ArXiv: 1608.00272.
  53. Woo-han Yun, Taewoo Kim, Jaeyeon Lee, Jaehong Kim, and Junmo Kim. 2021. Cut-and-Paste Dataset Generation for Balancing Domain Gaps in Object Instance Detection. IEEE Access, 9:14319–14329. ArXiv:1909.11972 [cs]. — [arXiv:1909.11972](https://arxiv.org/abs/1909.11972) find
  54. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid Loss for Language Image Pre-Training. ArXiv:2303.15343 [cs].
  55. Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, and Filip Ilievski. 2025a. MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs. ArXiv:2502.17422 [cs]. — [arXiv:2502.17422](https://arxiv.org/abs/2502.17422) find
  56. Zhi Zhang, Srishti Yadav, Fengze Han, and Ekaterina Shutova. 2025b. Cross-modal Information Flow in Multimodal Large Language Models. ArXiv:2411.18620 [cs]. — [arXiv:2411.18620](https://arxiv.org/abs/2411.18620) find
  57. Junhao Zhuang, Yanhong Zeng, Wenran Liu, Chun Yuan, and Kai Chen. 2024. A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpainting. ArXiv:2312.03594 [cs]. — [arXiv:2312.03594](https://arxiv.org/abs/2312.03594) find
  58. Pelin Çelikkol, Jochen Laubrock, and David Schlangen. 2023. TF-IDF based Scene-Object Relations Correlate With Visual Attention. In 2023 Symposium on Eye Tracking Research and Applications, pages 1–6, Tubingen Germany. ACM. find
  59. Sabine Öhlschläger and Melissa Le-Hoa Võ. 2017. SCEGRAM: An image database for semantic and syntactic inconsistencies in scenes. Behavior Research Methods, 49(5):1780–1791. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/merlo-2026-common-objects-out-context-cooco/