Elamur Research⌕
☰
home / papers / 2023 / The Vector Grounding Problem

The Vector Grounding Problem

Paper · 2023 · Dimitri Coelho Mollo and Raphael Milliere

Summary

Harnad's 1990 Symbol Grounding Problem asked whether symbols in an AI system can mean anything beyond what human readers attribute to them. Coelho Mollo and Millière restate this question for LLMs, which compute over continuous embeddings learned only from text, and call it the Vector Grounding Problem. The work is philosophical. It introduces no dataset, benchmark or model, and its method is conceptual analysis based on teleosemantic theories of content. The authors separate several senses of "grounding" and argue that only referential grounding is essential: the link between a representation and its worldly target. A system achieves this when its internal states carry information causally tied to the world and have acquired, through a selection history, the function of carrying that information. They argue that text-only LLMs can meet both conditions. This is clearest after preference tuning that rewards factuality, and it may also happen in narrow domains through pre-training or in-context learning. The upshot is that grounding neither requires multimodality or embodiment nor is guaranteed by them, while questions of understanding and mentality remain open.

Key points

  • Referential grounding requires two conditions: (1) causal-informational relations with the world, and (2) a selection history that gives internal states the function of carrying that information.
  • Text-only LLMs get indirect causal links to the world through human-produced text. Preference tuning adds an extra-linguistic accuracy standard (truthfulness or factuality) that selects and stabilises world-tracking states.
  • Without preference tuning, grounding may still arise in limited domains, such as input sequences that describe game moves.
  • The authors judge CLIP (image + text) and SayCan (a frozen LLM on a robot) to be poor candidates for referential grounding, because their training does not select states for world-involving functions.
  • Vision-language-action models such as RT-2, RT-X and OpenVLA may inherit grounding from their pre-trained language components. Their embodied fine-tuning may add further grounding.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Arts and Humanities · 16,039 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 worksNeuroscience · 165 worksCognitive Neuroscience · 164 worksHistory and Philosophy of Science · 53 works

Cite

Cite as Dimitri Coelho Mollo and Raphael Milliere. 2023. The Vector Grounding Problem. arXiv:2304.01481.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2304.01481 · arXiv 2304.01481

@misc{mollo2023vector,
  title = {{The Vector Grounding Problem}},
  author = {Dimitri Coelho Mollo and Raphael Milliere},
  year = {2023},
  doi = {10.48550/arxiv.2304.01481},
  eprint = {2304.01481},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.01481}
}

References

89 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Ho, D., Hsu, J., Ibarz, J., Ichter, B., Irpan, A., Jang, E., Ruano, R. J., Jeffrey, K., Jesmonth, S., Joshi, N. J., Julian, R., Kalashnikov, D., Kuang, Y., Lee, K.-H.,… — Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
  2. Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Kernion, J., Ndousse, K., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C. & Kaplan, J. (2021), ‘A general language… find
  3. Barsalou, L. W. (1999), ‘Perceptual symbol systems’, Behavioral and Brain Sciences 22, 577–660. find
  4. Bender, E. M., Gebru, T., McMillan-Major, A. & Shmitchell, S. (2021), On the dangers of stochastic parrots, in ‘Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency’, ACM. find
  5. Bender, E. M. & Koller, A. (2020), Climbing towards nlu: On meaning, form, and understanding in the age of data, in ‘Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics’, Association for Computational Linguistics.
  6. Block, N. (1986), ‘Advertisement for a semantics for psychology’, Midwest Studies in Philosophy 10, 615–678. find
  7. Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N., Chen, A., Creel, K., Davis, J. Q., Demszky, D., Donahue, C., Doumbouya, M., Durmus, E., Ermon, S.,… — On the Opportunities and Risks of Foundation Models
  8. Bordes, F., Pang, R. Y., Ajay, A., Li, A. C., Bardes, A., Petryk, S., Mañas, O., Lin, Z., Mahmoud, A., Jayaraman, B., Ibrahim, M., Hall, M., Xiong, Y., Lebensold, J., Ross, C., Jayakumar, S., Guo, C., Bouchacourt, D., Al-Tahan, H., Padthe, K., Sharma, V., Xu, H., Tan, X. E., Richards, M., Lavoie, S.… find
  9. Borg, E. (2025), ‘Llms, turing tests and chinese rooms: The prospects for meaning in large language models’, Inquiry pp. 1–31. find
  10. Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., Driessche, G. B. V. D., Lespiau, J.-B., Damoc, B., Clark, A., Casas, D. D. L., Guy, A., Menick, J., Ring, R., Hennigan, T., Huang, S., Maggiore, L., Jones, C., Cassirer, A., Brock, A., Paganini, M., Irving, G., Vinyals, O… find
  11. Brandom, R. (1994), Making It Explicit: Reasoning, Representing, and Discursive Commitment, Harvard University Press. find
  12. Brennan, S. E. (1998), The grounding problem in conversations with and through computers, in S. R. Fussell & R. J. Kreuz, eds, ‘Social and Cognitive Approaches to Interpersonal Communication’, Lawrence Erlbaum, Hillsdale, NJ, pp. 201–225. find
  13. Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., Nori, H., Palangi, H., Ribeiro, M. T. & Zhang, Y. (2023), ‘Sparks of artificial general intelligence: Early experiments with gpt-4’.
  14. Butlin, P. (2021), ‘Sharing our concepts with machines’, Erkenntnis 88(7), 3079–3095. find
  15. Cappelen, H. & Dever, J. (2021), Making AI Intelligible: Philosophical Foundations, Oxford University Press, Oxford, New York. find
  16. Chalmers, D. J. (2023), ‘Does thought require sensory grounding? from pure thinkers to large language models’, Proceedings and Addresses of the American Philosophical Association 97, 22–45. find
  17. Chalmers, D. J. (2025), ‘Propositional interpretability in artificial intelligence’. find
  18. Chandu, K. R., Bisk, Y. & Black, A. W. (2021), Grounding ‘grounding’ in nlp, in ‘Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021’, Association for Computational Linguistics, Online, pp. 4283–4305. find
  19. Chomsky, N. (1957), Syntactic Structures, Mouton. find
  20. Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S. & Amodei, D. (2017), Deep reinforcement learning from human preferences, in ‘Advances in Neural Information Processing Systems’, Vol. 30, Curran Associates, Inc. find
  21. Clark, H. H. & Brennan, S. E. (1991), Grounding in communication, in ‘Perspectives on Socially Shared Cognition’, American Psychological Association, Washington, DC, US, pp. 127–149. find
  22. Coelho Mollo, D. (2015), ‘Being clear on content’, Philosophia 43(3), 687–699. find
  23. Coelho Mollo, D. (2022), ‘Deflationary realism: Representation and idealisation in cognitive science’, Mind & Language 37(5), 1048–1066. de Saussure, F. (1916), Cours de Linguistique Générale, Payot, Paris. find
  24. Delétang, G., Ruoss, A., Duquenne, P.-A., Catt, E., Genewein, T., Mattern, C., Grau-Moya, J., Wenliang, L. K., Aitchison, M., Orseau, L., Hutter, M. & Veness, J. (2024), ‘Language modeling is compression’.
  25. Di Maro, M. (2021), ‘Computational grounding: An overview of common ground applications in conversational agents’, IJCoL. Italian Journal of Computational Linguistics 7(1 \| 2), 133–156. find
  26. Firth, J. R. (1957), Papers in Linguistics, 1934-1951, Oxford University Press, London. find
  27. Fodor, J. A. (1990), A Theory of Content and Other Essays, MIT Press. find
  28. Garson, J. (2019), What Biological Functions Are and Why They Matter, Cambridge University Press. find
  29. Godfrey-Smith, P. (1994), ‘A modern history theory of functions’, Noûs 28(3), 344–362. find
  30. Goldstein, S. & Levinstein, B. A. (2024), ‘Does chatgpt have a mind?’. find
  31. Grindrod, J. (2024), ‘Large language models and linguistic intentionality’, Synthese 204(2), 71. find
  32. Grzankowski, A. (2024), ‘Real sparks of artificial intelligence and the importance of inner interpretability’, Inquiry 0(0), 1–27. find
  33. Haiman, J. (1985), Iconicity in Syntax, Vol. 6, John Benjamins Publishing. find
  34. Harding, J. (2023), ‘Operationalising representation in natural language processing’. find
  35. Harnad, S. (1990), ‘The symbol-grounding problem’, Physica D 42, 335–346. find
  36. Harris, Z. S. (1954), ‘Distributional structure’, Word 10, 146–162. find
  37. Heyes, C. M. (2018), Cognitive gadgets, The Belknap Press of Harvard University Press, Cambridge, Massachusetts. Includes bibliographical references and index. jylin04, JackS, Karvonen, A. & Can (2024), ‘Othellogpt learned a bag of heuristics’. find
  38. Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., Vuong, Q., Kollar, T., Burchfiel, B., Tedrake, R., Sadigh, D., Levine, S., Liang, P. & Finn, C. (2024), ‘Openvla: An open-source vision-language-action model’. find
  39. LeCun, Y., Bengio, Y. & Hinton, G. (2015), ‘Deep learning’, Nature 521(7553), 436–444. find
  40. Lederman, H. & Mahowald, K. (2024), ‘Are language models more like libraries or like librarians? bibliotechnism, the novel reference problem, and the attitudes of llms’.
  41. Li, K., Hopkins, A. K., Bau, D., Viégas, F., Pfister, H. & Wattenberg, M. (2023), ‘Emergent world representations: Exploring a sequence model trained on a synthetic task’. find
  42. Mandelkern, M. & Linzen, T. (2024), ‘Do language models’ words refer?’, Computational Linguistics pp. 1–10. find
  43. Mcclelland, J. L., Rumelhart, D. E. & Group, P. R. (1986), Parallel Distributed Processing, Explorations in the Microstructure of Cognition, MIT Press, Cambridge, MA. find
  44. McCoy, R. T., Yao, S., Friedman, D., Hardy, M. D. & Griffiths, T. L. (2024), ‘Embers of autoregression show how large language models are shaped by the problem they are trained to solve’, Proceedings of the National Academy of Sciences 121(41). find
  45. Merrill, W., Wu, Z., Naka, N., Kim, Y. & Linzen, T. (2024), Can you learn semantics through next-word prediction? the case of entailment, in L.-W. Ku, A. Martins & V. Srikumar, eds, ‘Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and Virtual Meeting, August 11… find
  46. Millière, R. (forthcoming), Language models as models of language, in R. Nefdt, G. Dupre & K. Stanton, eds, ‘The Oxford Handbook of the Philosophy of Linguistics’, Oxford University Press. find
  47. Millière, R. & Buckner, C. (2024), ‘A philosophical introduction to language models – part i: Continuity with classic debates’. find
  48. Millière, R. & Buckner, C. (2025), Interventionist methods for interpreting deep neural networks, in G. Piccinini, ed., ‘Neurocognitive Foundations of Mind’, Routledge. find
  49. Millikan, R. G. (1989), ‘In defense of proper functions’, Philosophy of Science 56(2), 288–302. find
  50. Millikan, R. G. (2017), Beyond Concepts: Unicepts, Language, and Natural Information, Oxford University Press. find
  51. Miracchi Titus, L. (2024), ‘Does chatgpt have semantic understanding? a problem with the statistics-of-occurrence strategy’, Cognitive Systems Research 83, 101174. find
  52. Montague, R. (1970), ‘Universal grammar’, Theoria 36(3), 373–398. find
  53. Nanda, N., Lee, A. & Wattenberg, M. (2023), Emergent linear representations in world models of self-supervised sequence models, in Y. Belinkov, S. Hao, J. Jumelet, N. Kim, A. McCarthy & H. Mohebbi, eds, ‘Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP,… find
  54. Neander, K. (1991), ‘Functions as selected effects: The conceptual analyst’s defence’, Philosophy of Science 58(2), 168–184. find
  55. Neander, K. (2017), A Mark of the Mental: in defense of informational semantics, The MIT Press. find
  56. O’Neill, A., Rehman, A., Gupta, A., Maddukuri, A., Gupta, A., Padalkar, A., Lee, A., Pooley, A., Gupta, A., Mandlekar, A., Jain, A., Tung, A., Bewley, A., Herzog, A., Irpan, A., Khazatsky, A., Rai, A., Gupta, A., Wang, A., Kolobov, A., Singh, A., Garg, A., Kembhavi, A., Xie, A., Brohan, A., Raffin,… find
  57. OpenAI (2023), ‘Gpt-4 technical report’.
  58. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J. & Lowe, R. (2022), ‘Training language models to follow instruction…
  59. Partee, B. H. (1981), Montague grammar, mental representations, and reality, in S. Kanger & S. Ōhman, eds, ‘Philosophy and Grammar: Papers on the Occasion of the Quincentennial of Uppsala University’, Synthese Library, Springer Netherlands, Dordrecht, pp. 59–78. find
  60. Pavlick, E. (2022), ‘Semantic structure in deep learning’, Annual Review of Linguistics 8(1), 447–471. find
  61. Pepp, J. (2025), ‘Reference without intentions in large language models’, Inquiry pp. 1–19. find
  62. Perniss, P. & Vigliocco, G. (2014), ‘The bridge of iconicity: From a world of experience to the experience of language’, Philosophical Transactions of the Royal Society B: Biological Sciences 369(1651), 20130300. find
  63. Petroni, F., Rocktäschel, T., Lewis, P., Bakhtin, A., Wu, Y., Miller, A. H. & Riedel, S. (2019), ‘Language models as knowledge bases?’.
  64. Piantadosi, S. T. & Hill, F. (2022), ‘Meaning without reference in large language models’. find
  65. Poon, H. (2013), Grounded unsupervised semantic parsing, in ‘Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)’, Association for Computational Linguistics, Sofia, Bulgaria, pp. 933–943. find
  66. Pulvermüller, F. (1999), ‘Words in the brain’s language’, Behavioural and Brain Sciences 22, 253–336. find
  67. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G. & Sutskever, I. (2021), ‘Learning transferable visual models from natural language supervision’, arXiv:2103.00020 [cs] .
  68. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D. & Finn, C. (2024), ‘Direct preference optimization: Your language model is secretly a reward model’.
  69. Searle, J. R. (1980), ‘Minds, brains and programs’, Behavioral and Brain Sciences 3(3), 417–457. find
  70. Shea, N. (2012), Millikan?s isomorphism requirement, in D. Ryder, J. Kingsbury & K. Williford, eds, ‘Millikan and her critics’, Wiley, pp. 63–86. find
  71. Shea, N. (2018), Representation in Cognitive Science, Oxford University Press. find
  72. Shortliffe, E. H. (1977), ‘Mycin: A knowledge-based computer program applied to infectious diseases’, Proceedings of the Annual Symposium on Computer Application in Medical Care pp. 66–69. find
  73. Simon, H. A. & Newell, A. (1971), ‘Human problem solving: The state of the theory in 1970’, American Psychologist 26(2), 145–159. find
  74. Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., Kluska, A., Lewkowycz, A., Agarwal, A., Power, A., Ray, A., Warstadt, A., Kocurek, A. W., Safaya, A., Tazarv, A., Xiang, A., Parrish, A., Nie, A., Hussain, A., Askell… — Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
  75. Stalnaker, R. (2002), ‘Common ground’, Linguistics and Philosophy 25(5-6), 701–721. find
  76. Sterelny, K. (2014), The evolved apprentice, The Jean Nicod lec tures, 1st paperback edition edn, The MIT Press, Cambridge, Massachusetts. find
  77. Stoljar, D. & Zhang, Z. V. (2025), ‘Why chatgpt doesn’t think: An argument from rationality’, Inquiry pp. 1–29. find
  78. Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., Li, Y., Lee, H., Zheng, H. S., Ghafouri, A., Menegali, M., Huang, Y., Krikun, M., Lepikhin, D., Qin, J., Chen, D., Xu, Y., Chen, Z., Roberts, A., Bosma, M., Zhao, V., Zhou, Y.,… — LaMDA: Language Models for Dialog Applications
  79. Tomasello, M. (2009), Cultural Origins of Human Cognition, Harvard University Press, Cambridge. Description based upon print version of record. find
  80. Traum, D. R. (1994), A Computational Theory of Grounding in Natural Language Conversation, Ph.d. thesis, University of Rochester, Rochester, NY. find
  81. Tsai, C.-T. & Roth, D. (2016), ‘Concept grounding to multiple knowledge bases via indirect supervision’, Transactions of the Association for Computational Linguistics 4(0), 141–154. find
  82. Van Langendonck, W. (2010), Iconicity, in D. Geeraerts & H. Cuyckens, eds, ‘The Oxford Handbook of Cognitive Linguistics’, Oxford University Press, p. 0. find
  83. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. & Polosukhin, I. (2017), ‘Attention is all you need’. von Oswald, J., Schlegel, M., Meulemans, A., Kobayashi, S., Niklasson, E., Zucchet, N., Scherrer, N., Miller, N., Sandler, M., y Arcas, B. A., Vladymyrov, M.…
  84. Williams, I. (2025), ‘Can structural correspondences ground real world representational content in large language models?’. URL: https://arxiv.org/abs/2506.16370 — [arXiv:2506.16370](https://arxiv.org/abs/2506.16370) ↗
  85. Wright, L. (1973), ‘Functions’, The Philosophical Review 82(2), 139–168. find
  86. Wu, X. & Varshney, L. R. (2024), A meta-learning perspective on transformers for causal language modeling, in L.-W. Ku, A. Martins & V. Srikumar, eds, ‘Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and Virtual Meeting, August 11-16, 2024’, Association for Com… find
  87. Yildirim, I. & Paul, L. A. (2023), ‘From task structures to world models: What do llms know?’, Trends in Cognitive Sciences . find
  88. Zheng, C., Huang, W., Wang, R., Wu, G., Zhu, J. & Li, C. (2024), On mesa-optimization in autoregressively trained transformers: Emergence and capability, in A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomczak & C. Zhang, eds, ‘Advances in Neural Information Processing Systems 38:… find
  89. Zitkovich, B., Yu, T., Xu, S., Xu, P., Xiao, T., Xia, F., Wu, J., Wohlhart, P., Welker, S., Wahid, A., Vuong, Q., Vanhoucke, V., Tran, H. T., Soricut, R., Singh, A., Singh, J., Sermanet, P., Sanketi, P. R., Salazar, G., Ryoo, M. S., Reymann, K., Rao, K., Pertsch, K., Mordatch, I., Michalewski, H., L… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mollo-2023-vector-grounding-problem/