Elamur Research⌕
☰
home / papers / 2024 / The Platonic Representation Hypothesis

The Platonic Representation Hypothesis

Paper · 2024 · Minyoung Huh, Brian Cheung, Tongzhou Wang et al.

Summary

The authors ask whether neural networks that differ in architecture, training objective, data and even modality end up encoding the world in similar ways, and if so, why. They describe each representation by the kernel it induces, meaning the pairwise similarities between inputs. Agreement between two models is scored with a mutual nearest-neighbor metric: the average overlap of each datapoint's k-nearest-neighbor sets, divided by k.

Combining a literature review (model stitching, kernel-similarity studies) with their own experiments, they show two trends. Stronger vision models cluster together. Language and vision models also become more aligned with each other as they grow.

From this they propose the Platonic Representation Hypothesis: training pushes models toward a single shared statistical model of the reality that generates observed data, with scale and task diversity as the driving pressures. This matters for anyone reusing foundation models across tasks or modalities, including Ukrainian NLP work. It suggests that embeddings from separately trained systems may become increasingly interchangeable, although the authors acknowledge clear limits.

Key points

  • 78 vision models were compared using kernel alignment on Places-365, with competence measured on VTAB (19 tasks). Models that solve more tasks form a tight cluster, while weak models diverge in different directions.
  • The cross-modal experiments show image–text alignment rising with model size. However, the observed mutual nearest-neighbor score peaks at only 0.16 (maximum 1), and whether this counts as strong alignment is left as an open question.
  • Supporting prior work includes:
    • Lenc & Vedaldi (2015), who stitched ImageNet- and Places-365-trained networks and found early layers the most interchangeable.
    • Bansal et al. (2021), who found that self-supervised and supervised models align.
    • Moschella et al. (2022), who used zero-shot stitching to link an English encoder with a French decoder.
  • The formal argument holds only for bijective projections of reality. Lossy modalities, abstract concepts and narrow special-purpose systems may not converge, and the choice of alignment metric remains debated.
  • The paper was published at ICML 2024 (PMLR 235) by MIT authors. Code is at github.com/minyoungg/platonic-rep.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Minyoung Huh, Brian Cheung, Tongzhou Wang and Phillip Isola. 2024. The Platonic Representation Hypothesis. arXiv:2405.07987.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2405.07987 · arXiv 2405.07987

@misc{huh2024platonic,
  title = {{The Platonic Representation Hypothesis}},
  author = {Minyoung Huh, Brian Cheung, Tongzhou Wang and Phillip Isola},
  year = {2024},
  doi = {10.48550/arxiv.2405.07987},
  eprint = {2405.07987},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.07987}
}

References

134 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. BigScience, Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon, F., et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022. find
  2. Abdou, M., Kulmizev, A., Hershcovich, D., Frank, S., Pavlick, E., and Søgaard, A. Can language models encode perceptual structure without grounding? a case study in color. arXiv preprint arXiv:2109.06129, 2021.
  3. Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
  4. Ainsworth, S. K., Hayase, J., and Srinivasa, S. Git re-basin: Merging models modulo permutation symmetries. arXiv preprint arXiv:2209.04836, 2022. — [arXiv:2209.04836](https://arxiv.org/abs/2209.04836) find
  5. Antonello, R. and Huth, A. Predictive coding or just feature discovery? an alternative account of why language models fit brain data. Neurobiology of Language, 5(1):64–79, 2024. find
  6. Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023. find
  7. Aronszajn, N. Theory of reproducing kernels. Transactions of the American mathematical society, 68(3):337–404, 1950. find
  8. Cao, R. and Yamins, D. Explanatory models in neuroscience: Part 2–constraint-based intelligibility. Cognitive Systems Research, 85, 2024. find
  9. Arora, S., Cohen, N., Hu, W., and Luo, Y. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems, 32, 2019a. find
  10. Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650–9660, 2021.
  11. Arora, S., Khandeparkar, H., Khodak, M., Plevrakis, O., and Saunshi, N. A theoretical analysis of contrastive Chambers, N. and Jurafsky, D. Unsupervised learning of narrative event chains. In Proceedings of ACL-08: HLT, pp. 789–797, 2008. find
  12. Gao, T., Yao, X., and Chen, D. SimCSE: Simple contrastive learning of sentence embeddings. In Empirical Methods in Natural Language Processing (EMNLP), 2021.
  13. Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597–1607. PMLR, 2020.
  14. Garipov, T., Izmailov, P., Podoprikhin, D., Vetrov, D. P., and Wilson, A. G. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems, 31, 2018. find
  15. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
  16. Gell-Mann, M. The Quark and the Jaguar: Adventures in the Simple and the Complex. Macmillan, 1995. find
  17. Geng, X. and Liu, H. OpenLLaMA: An open reproduction of LLaMA, May 2023. URL https://github.com/ openlm-research/open llama. ↗
  18. Conwell, C., Prince, J. S., Kay, K. N., Alvarez, G. A., and Konkle, T. What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines? BioRxiv, pp. 2022–03, 2022. find
  19. Gokaslan, A. and Cohen, V. Openwebtext corpus. http://Skylion007.github.io/OpenWebTextCorpus, 2019. ↗
  20. Goldblum, M., Finzi, M., Rowan, K., and Wilson, A. G. The no free lunch theorem, Kolmogorov complexity, and the role of inductive biases in machine learning. arXiv preprint arXiv:2304.05366, 2023. — [arXiv:2304.05366](https://arxiv.org/abs/2304.05366) find
  21. Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:30318–30332, 2022. find
  22. Google. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
  23. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  24. Gretton, A., Bousquet, O., Smola, A., and Schölkopf, B. Measuring statistical dependence with hilbert-schmidt norms. In International conference on algorithmic learning theory, pp. 63–77. Springer, 2005. find
  25. Diamond, J. M. Guns, germs and steel: a short history of everybody for the last 13,000 years. Vintage London, 1998. find
  26. Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y., et al. Olmo: Accelerating the science of language models. arXiv preprint arXiv:2402.00838, 2024. find
  27. Dingle, K., Camargo, C. Q., and Louis, A. A. Input–output maps are strongly biased towards simple outputs. Nature communications, 9(1):761, 2018. find
  28. Gunasekar, S., Lee, J. D., Soudry, D., and Srebro, N. Implicit bias of gradient descent on linear convolutional networks. In Advances in Neural Information Processing Systems, pp. 9461–9471, 2018. find
  29. Doppelt, G. Reconstructing scientific realism to rebut the pessimistic meta-induction. Philosophy of Science, 74(1): 96–118, 2007. find
  30. Gutmann, M. and Hyvärinen, A. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 297–304. JMLR Workshop and Conference Proceedings, 2010. find
  31. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  32. Dravid, A., Gandelsman, Y., Efros, A. A., and Shocher, A. Rosetta neurons: Mining the common units in a model zoo. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1934–1943, 2023. find
  33. Ha, D. and Schmidhuber, J. World models. arXiv preprint arXiv:1803.10122, 2018. find
  34. Hall, M., van der Maaten, L., Gustafson, L., Jones, M., and Adcock, A. A systematic study of bias amplification. arXiv preprint arXiv:2201.11706, 2022. — [arXiv:2201.11706](https://arxiv.org/abs/2201.11706) find
  35. Driess, D., Xia, F., Sajjadi, M. S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023. find
  36. Hardin, C. L. and Rosenberg, A. In defense of convergent realism. Philosophy of Science, 49(4):604–615, 1982. find
  37. He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729–9738, 2020.
  38. Kabsch, W. A solution for the best rotation to relate two sets of vectors. Acta Crystallographica Section A: Crystal Physics, Diffraction, Theoretical and General Crystallography, 32(5):922–923, 1976. find
  39. Kabsch, W. A discussion of the solution for the best rotation to relate two sets of vectors. Acta Crystallographica Section A: Crystal Physics, Diffraction, Theoretical and General Crystallography, 34(5):827–828, 1978. find
  40. He, K., Chen, X., Xie, S., Li, Y., Doll’ar, P., and Girshick, R. B. Masked autoencoders are scalable vision learners. 2022 ieee. In CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15979–15988, 2021.
  41. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
  42. Held, R., Ostrovsky, Y., de Gelder, B., Gandhi, T., Ganesh, S., Mathur, U., and Sinha, P. The newly sighted fail to match seen with felt. Nature neuroscience, 14(5):551– 553, 2011. find
  43. Klabunde, M., Schumacher, T., Strohmaier, M., and Lemmerich, F. Similarity of neural network models: A survey of functional and representational measures. arXiv preprint arXiv:2305.06329, 2023. — [arXiv:2305.06329](https://arxiv.org/abs/2305.06329) find
  44. Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., and Zhou, Y. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017.
  45. Koh, J. Y., Salakhutdinov, R., and Fried, D. Grounding language models to images for multimodal inputs and outputs. In International Conference on Machine Learning, pp. 17283–17300. PMLR, 2023. find
  46. Hooker, S. The hardware lottery. Communications of the ACM, 64(12):58–65, 2021. find
  47. Huh, M., Mobahi, H., Zhang, R., Cheung, B., Agrawal, P., and Isola, P. The low-rank simplicity bias in deep networks. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/forum?id=bCiNWDmlY2. ↗
  48. Kornblith, S., Norouzi, M., Lee, H., and Hinton, G. Similarity of neural network representations revisited. In International conference on machine learning, pp. 3519– 3529. PMLR, 2019.
  49. Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009. find
  50. Isola, P. The discovery of perceptual structure from visual co-occurrences in space and time. In MIT Ph.D. Thesis, 2015. find
  51. Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84–90, 2017. find
  52. Isola, P., Zoran, D., Krishnan, D., and Adelson, E. H. Crisp boundary detection using pointwise mutual information. In ECCV, 2014. find
  53. Lample, G., Ott, M., Conneau, A., Denoyer, L., and Ranzato, M. Phrase-based & neural unsupervised machine translation. In Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, J. (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October… find
  54. Isola, P., Zoran, D., Krishnan, D., and Adelson, E. H. Learning visual groups from co-occurrences in space and time. In ICLR, Workshop paper, 2016. find
  55. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. find
  56. Lenc, K. and Vedaldi, A. Understanding image representations by measuring their equivariance and equivalence. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 991–999, 2015. find
  57. Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. find
  58. Li, T., Katabi, D., and He, K. Return of unconditional generation: A self-supervised representation generation method. arXiv:2312.03701, 2023. find
  59. Jordan, K., Sedghi, H., Saukh, O., Entezari, R., and Neyshabur, B. Repair: Renormalizing permuted activations for interpolation repair. arXiv preprint arXiv:2211.08403, 2022. — [arXiv:2211.08403](https://arxiv.org/abs/2211.08403) find
  60. Lian, L., Li, B., Yala, A., and Darrell, T. LLM-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models. arXiv preprint arXiv:2305.13655, 2023a. find
  61. Lian, L., Shi, B., Yala, A., Darrell, T., and Li, B. LLM-grounded video diffusion models. arXiv preprint arXiv:2309.17444, 2023b. find
  62. Mirchandani, S., Xia, F., Florence, P., Ichter, B., Driess, D., Arenas, M. G., Rao, K., Sadigh, D., and Zeng, A. Large language models as general pattern machines. arXiv preprint arXiv:2307.04721, 2023.
  63. Lindsey, D. T. and Brown, A. M. The color lexicon of american english. Journal of vision, 14(2):17–17, 2014. find
  64. Mirza, M. and Osindero, S. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014. find
  65. Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. In NeurIPS, 2023.
  66. Moschella, L., Maiorca, V., Fumero, M., Norelli, A., Locatello, F., and Rodolà, E. Relative representations enable zero-shot latent space communication. arXiv preprint arXiv:2209.15430, 2022. — [arXiv:2209.15430](https://arxiv.org/abs/2209.15430) find
  67. Liu, S., Wang, T., Bau, D., Zhu, J.-Y., and Torralba, A. Diverse image generation via self-conditioned GANs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. find
  68. Nagarajan, V. and Kolter, J. Z. Uniform convergence may be unable to explain generalization in deep learning. Advances in Neural Information Processing Systems, 32, 2019. find
  69. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.
  70. Nettleship, R. L. Lectures on the ‘Republic’ of Plato, volume 2. Macmillan, 1897. find
  71. Locke, J. An Essay Concerning Human Understanding. 1690. Newton-Smith, W. The Rationality of Science. International Library of Philosophy, Psychology, and Scientific Method. Routledge & Kegan Paul, 1981. ISBN 9780710009135. find
  72. López-Cifuentes, A., Escudero-Vinolo, M., Bescós, J., and Garcı́a-Martı́n, Á. Semantic-aware scene recognition. Pattern Recognition, 102:107256, 2020. find
  73. Ng, E., Subramanian, S., Klein, D., Kanazawa, A., Darrell, T., and Ginosar, S. Can language models learn to listen? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10083–10093, 2023. find
  74. Lu, K., Grover, A., Abbeel, P., and Mordatch, I. Pretrained transformers as universal computation engines. arXiv preprint arXiv:2103.05247, 1, 2021. — [arXiv:2103.05247](https://arxiv.org/abs/2103.05247) find
  75. Ngo, J. and Kim, Y. What do language models hear? probing for auditory representations in language models, 2024. find
  76. Lubana, E. S., Bigelow, E. J., Dick, R. P., Krueger, D., and Tanaka, H. Mechanistic mode connectivity. In International Conference on Machine Learning, pp. 22965– 23004. PMLR, 2023. find
  77. Olshausen, B. A. and Field, D. J. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381(6583):607–609, 1996. find
  78. Ma, J., He, Y., Li, F., Han, L., You, C., and Wang, B. Segment anything in medical images. Nature Communications, 15(1):654, 2024. find
  79. Olshausen, B. A. and Field, D. J. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision research, 37(23):3311–3325, 1997. find
  80. Maniparambil, M., Akshulakov, R., Djilali, Y. A. D., El Amine Seddik, M., Narayan, S., Mangalam, K., and O’Connor, N. E. Do vision and language encoders represent the world similarly? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14334–14343, 2024. find
  81. Oord, A. v. d., Li, Y., and Vinyals, O. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.
  82. OpenAI. GPT-4 technical report. arXiv:2303.08774, 2023.
  83. McInnes, L., Healy, J., and Melville, J. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426, 2018. arXiv preprint Oquab, M., Darcet, T., Moutakanni, T., Vo, H. V., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby,…
  84. Merullo, J., Castricato, L., Eickhoff, C., and Pavlick, E. Linearly mapping from image to text space. arXiv preprint arXiv:2209.15162, 2022. find
  85. Meta. Meta LLaMA 3, 2024. URL https://ai.meta.com/blog/meta-llama-3/. ↗
  86. Oron, S., Dekel, T., Xue, T., Freeman, W. T., and Avidan, S. Best-buddies similarity—robust template matching using mutual nearest neighbors. IEEE transactions on pattern analysis and machine intelligence, 40(8):1799– 1813, 2017. find
  87. Schrimpf, M., Kubilius, J., Hong, H., Majaj, N. J., Rajaling-ham, R., Issa, E. B., Kar, K., Bashivan, P., Prescott-Roy, J., Geiger, F., et al. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv, pp. 407007, 2018. find
  88. Papyan, V., Han, X., and Donoho, D. L. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):24652–24663, 2020. find
  89. Sharma, P., Rott Shaham, T., Baradad, M., Fu, S., Rodriguez-Munoz, A., Duggal, S., Isola, P., and Torralba, A. A vision check-up for language models. In arXiv preprint, 2024. find
  90. Park, Y.-J., Wang, H., Ardeshir, S., and Azizan, N. Quantifying representation reliability in self-supervised learning models. In Conference on Uncertainty in Artificial Intelligence, 2024. find
  91. Shepard, R. N. Multidimensional scaling, tree-fitting, and clustering. Science, 210(4468):390–398, 1980. find
  92. Plato. Republic. c. 375 BC. Shi, Y., De Bortoli, V., Campbell, A., and Doucet, A. Diffusion schrödinger bridge matching. Advances in Neural Information Processing Systems, 36, 2024. find
  93. Putnam, H. Three kinds of scientific realism. The Philosophical Quarterly (1950-), 32(128):195–200, 1982. find
  94. Smola, A. J. and Schölkopf, B. Learning with kernels, volume 4. Citeseer, 1998. find
  95. Radford, A., Jozefowicz, R., and Sutskever, I. Learning to generate reviews and discovering sentiment. arXiv preprint arXiv:1704.01444, 2017.
  96. Solomonoff, R. J. A formal theory of inductive inference. part i. Information and control, 7(1):1–22, 1964. find
  97. Song, L., Smola, A., Gretton, A., Bedo, J., and Borgwardt, K. Feature selection via dependence maximization. Journal of Machine Learning Research, 13(5), 2012. find
  98. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. find
  99. Sorscher, B., Ganguli, S., and Sompolinsky, H. Neural representational geometry underlies few-shot concept learning. Proceedings of the National Academy of Sciences, 119 (43):e2200800119, 2022. find
  100. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748–8763. PMLR, 2021.
  101. Srinivasan, K., Raman, K., Chen, J., Bendersky, M., and Najork, M. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2443–2449, 2021.
  102. Raghu, M., Gilmer, J., Yosinski, J., and Sohl-Dickstein, J. Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017. find
  103. Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022.
  104. Richens, J. and Everitt, T. Robust agents learn causal world models. ICLR, 2024. find
  105. Roeder, G., Metz, L., and Kingma, D. On linear identifiability of learned representations. In International Conference on Machine Learning, pp. 9030–9039. PMLR, 2021. find
  106. Steinberg, E., Jung, K., Fries, J. A., Corbin, C. K., Pfohl, S. R., and Shah, N. H. Language models are an effective representation learning technique for electronic health record data. Journal of biomedical informatics, 113: 103637, 2021. find
  107. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115: 211–252, 2015.
  108. Stoica, G., Bolya, D., Bjorner, J., Hearn, T., and Hoffman, J. Zipit! merging models from different tasks without training. arXiv preprint arXiv:2305.03053, 2023. — [arXiv:2305.03053](https://arxiv.org/abs/2305.03053) find
  109. Sauer, A., Schwarz, K., and Geiger, A. StylegGAN-XL: Scaling StyleGAN to large diverse datasets. In ACM SIG-GRAPH 2022 conference proceedings, pp. 1–10, 2022. find
  110. Sucholutsky, I., Muttenthaler, L., Weller, A., Peng, A., Bobu, A., Kim, B., Love, B. C., Grant, E., Groen, I., Achterberg, J., Tenenbaum, J. B., Collins, K. M., Hermann, K. L., Oktar, K., Greff, K., Hebart, M. N., Jacoby, N., Zhang, Q., Marjieh, R., Geirhos, R., Chen, S., Kornblith, S., Rane, S., Ko… find
  111. Werbos, P. J. Learning how the world works: Specifications for predictive networks in robots and brains. In Proceedings of IEEE International Conference on Systems, Man and Cybernetics, NY, 1987. find
  112. Wightman, R. PyTorch image models. https://github.com/rwightman/pytorch-image-models, 2021. ↗
  113. Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024. find
  114. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019.
  115. Tian, Y., Krishnan, D., and Isola, P. Contrastive multiview coding. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16, pp. 776–794. find
  116. Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machin…
  117. Tian, Y., Wang, Y., Krishnan, D., Tenenbaum, J. B., and Isola, P. Rethinking few-shot image classification: a good embedding is all you need? In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16, pp. 266–282. find
  118. Wu, T.-H., Lian, L., Gonzalez, J. E., Li, B., and Darrell, T. Self-correcting LLM-controlled diffusion models. arXiv preprint arXiv:2311.16090, 2023. find
  119. Tolstoy, L. Anna Karenina. The Russian Messenger, 1877. find
  120. Torralba, A., Fergus, R., and Freeman, W. T. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE transactions on pattern analysis and machine intelligence, 30(11):1958–1970, 2008. find
  121. Xie, S., Ho, Q., and Zhang, K. Unsupervised image-to-image translation with density changing regularization. Advances in Neural Information Processing Systems, 35: 28545–28558, 2022. find
  122. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. LLaMA 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  123. Yamins, D. L., Hong, H., Cadieu, C. F., Solomon, E. A., Seibert, D., and DiCarlo, J. J. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 111(23):8619–8624, 2014. find
  124. Tran, D., Burda, Y., and Sutskever, I. Feature-matching auto-encoders. 2017. find
  125. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can a machine really finish your sentence? In Korhonen, A., Traum, D., and Màrquez, L. (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791– 4800, Florence, Italy, July 201… find
  126. Umeyama, S. Least-squares estimation of transformation parameters between two point patterns. IEEE Transactions on Pattern Analysis & Machine Intelligence, 13(04): 376–380, 1991. find
  127. Urbanek, J., Bordes, F., Astolfi, P., Williamson, M., Sharma, V., and Romero-Soriano, A. A picture is worth more than 77 text tokens: Evaluating CLIP-style models on dense captions, 2023. find
  128. Zhai, X., Puigcerver, J., Kolesnikov, A., Ruyssen, P., Riquelme, C., Lucic, M., Djolonga, J., Pinto, A. S., Neumann, M., Dosovitskiy, A., et al. The visual task adaptation benchmark. 2019. find
  129. Valle-Perez, G., Camargo, C. Q., and Louis, A. A. Deep learning generalizes because the parameter-function map is biased towards simple functions. In International Conference on Learning Representations, 2019. find
  130. Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586–595, 2018. find
  131. Wang, T. and Isola, P. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pp. 9929–9939. PMLR, 2020. find
  132. Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., and Torralba, A. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6):1452–1464, 2017. find
  133. Zhu, J.-Y., Park, T., Isola, P., and Efros, A. A. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Computer Vision (ICCV), 2017 IEEE International Conference on, 2017. find
  134. Zimmermann, R. S., Sharma, Y., Schneider, S., Bethge, M., and Brendel, W. Contrastive learning inverts the data generating process. In International Conference on Machine Learning, pp. 12979–12990. PMLR, 2021. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/huh-2024-platonic-representation-hypothesis/