Elamur Research⌕
☰
home / papers / 2026 / What Do Vision–Language Models Encode for Personalized Image Aesthetics Assessment?

What Do Vision–Language Models Encode for Personalized Image Aesthetics Assessment?

Paper · 2026 · Koki Ryu and Hitomi Yanaka

Summary

The paper asks whether general-purpose vision–language models (VLMs) internally represent the fine-grained aesthetic properties needed for personalized image aesthetics assessment (PIAA), which predicts how a specific user would rate an image. Existing PIAA systems usually depend on multi-stage training over large single-domain GIAA datasets, which is costly and transfers poorly between domains. The authors first apply ridge-regression linear probes to average-pooled hidden states from every layer of VLMs such as Qwen3-VL 2B and Gemma 3 4B. They probe the vision encoder, plus text and vision tokens in the language decoder, recovering the attribute annotations of AADB and PARA. They then fit per-user linear regressors on these frozen representations to predict individual ratings on PARA (photographs) and LAPIS (artworks), with no fine-tuning of the VLM. The probes show that many distinct aesthetic attributes are linearly recoverable and persist into decoder layers. Different architectures store them in different regions. The lightweight regressors beat text-output approaches. This suggests VLM latents are a cheap, domain-robust basis for modeling individual taste.

Key points

  • For probing, each image is paired with the instruction "Assess the aesthetics of this image." and three pooled representation types are extracted per layer: V (vision encoder), LT (decoder text tokens) and LV (decoder vision tokens).
  • AADB, which has 11 continuous attribute dimensions in [−1, 1] plus an overall score, serves as the main probing dataset. PARA's attributes were judged too inter-correlated to serve this purpose well.
  • The linear approach on hidden states (Linear-Hidden) clearly outperforms few-shot prompting and fine-tuning baselines that read scores from generated text. It also holds up better than dedicated PIAA methods under domain shift.
  • Personalization gains are smaller on PARA than on LAPIS, because PARA users agree closely with GIAA scores. On the 50 PARA users who diverge most from GIAA, user-specific regressors still clearly beat GIAA baselines.
  • A "Reduced" variant restricted to the probed attributes yields lower Spearman correlation but higher R² than the full model. Results on artworks point to useful signals, especially in vision tokens, that the photo-based probes do not capture.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of TokyoTokyo · Japan90 works · 9,156 citations🔬RIKENWako · Japan26 works · 526 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksVision & language · 657 works

Cite

Cite as Koki Ryu and Hitomi Yanaka. 2026. What Do Vision–Language Models Encode for Personalized Image Aesthetics Assessment?. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1706

@inproceedings{ryu2026what,
  title = {{What Do Vision–Language Models Encode for Personalized Image Aesthetics Assessment?}},
  author = {Koki Ryu and Hitomi Yanaka},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1706},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1706}
}

References

45 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025. Qwen3-vl technical report. arXiv prepri…
  2. G. Bradski. 2000. The OpenCV Library. Dr. Dobb’s Journal of Software Tools. find
  3. Alexander Buslaev, Vladimir I Iglovikov, Eugene Khvedchenya, Alex Parinov, Mikhail Druzhinin, and Alexandr A Kalinin. 2020. Albumentations: fast and flexible image augmentations. Information, 11(2):125. find
  4. Shivam Chandhok, Wan-Cyuan Fan, Vered Shwartz, Vineeth N. Balasubramanian, and Leonid Sigal. 2025. Response wide shut? surprising observations in basic vision language model capabilities. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper… find
  5. Kuang-Yu Chang, Kung-Hung Lu, and Chu-Song Chen. 2017. Aesthetic critiques generation for photos. In Proceedings of the IEEE international conference on computer vision, pages 3514–3523. find
  6. Alex Clark. 2015. Pillow (pil fork) documentation. find
  7. Mohamed El Banani, Amit Raj, Kevis-Kokitsi Maninis, Abhishek Kar, Yuanzhen Li, Michael Rubinstein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. 2024. Probing the 3d awareness of visual foundation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco… find
  8. Koustav Ghosal, Aakanksha Rana, and Aljosa Smolic. 2019. Aesthetic image captioning from weakly-labelled photographs. In Proceedings of the IEEE/CVF international conference on computer vision workshops, pages 0–0. find
  9. Simon Hentschel, Konstantin Kobs, and Andreas Hotho. 2022. Clip knows image aesthetics. Frontiers in Artificial Intelligence, 5:976235. find
  10. Yipo Huang, Quan Yuan, Xiangfei Sheng, Zhichao Yang, Haoning Wu, Pengfei Chen, Yuzhe Yang, Leida Li, and Weisi Lin. 2024. Aesbench: An expert benchmark for multimodal large language models on image aesthetics perception. arXiv preprint arXiv:2401.08276. — [arXiv:2401.08276](https://arxiv.org/abs/2401.08276) find
  11. Omri Kaduri, Shai Bagon, and Tali Dekel. 2025. What’s in the image? a deep-dive into the vision of vision language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 14549–14558. find
  12. Yannis Kaltampanidis, Alexandros Doumanoglou, and Dimitrios Zarpalas. 2025. Which direction to choose? an analysis on the representation power of self-supervised vits in downstream tasks. In World Conference on Explainable Artificial Intelligence, pages 376–399. Springer. find
  13. Junjie Ke, Keren Ye, Jiahui Yu, Yonghui Wu, Peyman Milanfar, and Feng Yang. 2023. Vila: Learning image aesthetics from user comments with vision-language pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10041–10051. find
  14. Shu Kong, Xiaohui Shen, Zhe Lin, Radomir Mech, and Charless Fowlkes. 2016. Photo aesthetics ranking network with attributes and content adaptation. In ECCV. find
  15. Kun Li, Lai Man Po, Hongzheng Yang, Xuyuan Xu, Kangcheng Liu, and Yuzhi Zhao. 2025. AesBias-Bench: Evaluating bias and alignment in multimodal language models for personalized image aesthetic assessment. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages… find
  16. Lingchen Meng, Jianwei Yang, Rui Tian, Xiyang Dai, Zuxuan Wu, Jianfeng Gao, and Yu-Gang Jiang. 2024. Deepstack: Deeply stacking visual tokens is surprisingly simple and effective for lmms. Advances in Neural Information Processing Systems, 37:23464– 23487. find
  17. Naila Murray, Luca Marchesotti, and Florent Perronnin. 2012. Ava: A large-scale database for aesthetic visual analysis. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 2408–2415. find
  18. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, and 2 others. 2019.… — PyTorch: An Imperative Style, High-Performance Deep Learning Library
  19. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn:… find
  20. Daiqing Qi, Handong Zhao, Jing Shi, Simon Jenni, Yifei Fan, Franck Dernoncourt, Scott Cohen, and Sheng Li. 2025. The photographer’s eye: Teaching multimodal large language models to see, and critique like photographers. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages… find
  21. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8…
  22. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. find
  23. Jian Ren, Xiaohui Shen, Zhe Lin, Radomir Mech, and David J Foran. 2017. Personalized image aesthetics. In Proceedings of the IEEE international conference on computer vision, pages 638–647. find
  24. Anne-Sofie Maerten, Li-Wei Chen, Stefanie De Winter, Christophe Bossens, and Johan Wagemans. 2025. Lapis: A novel dataset for personalized image aesthetic assessment. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 6302–6311. find
  25. Huiying Shi, Jing Guo, Yongzhen Ke, Kai Wang, Shuai Yang, Fan Qin, and Liming Chen. 2024. Personalized image aesthetics assessment based on graph neural network and collaborative filtering. Knowledge-Based Systems, 294:111749. find
  26. Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, Benjamin Bossan, and Marian Tietz. 2022. PEFT: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft. ↗
  27. Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, and 1 others. 2025. Dinov3. arXiv preprint arXiv:2508.10104. 34156 11 — [arXiv:2508.10104](https://arxiv.org/abs/2508.10104) find
  28. Mingxu Tao, Quzhe Huang, Kun Xu, Liwei Chen, Yansong Feng, and Dongyan Zhao. 2024. Probing multimodal large language models for global and local semantic representations. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-C… find
  29. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, and 1 others. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. find
  30. Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. 2024. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568–9578.
  31. Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, and 1 others. 2020. Scipy 1.0: fundamental algorithms for scientific computing in python. Nature methods, 17(3):261–272.
  32. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Transf…
  33. Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, and 1 others. 2023. Q-bench: A benchmark for general-purpose foundation models on low-level vision. arXiv preprint arXiv:2309.14181. — [arXiv:2309.14181](https://arxiv.org/abs/2309.14181) find
  34. Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Kaixin Xu, Chunyi Li, Jingwen Hou, Guangtao Zhai, and 1 others. 2024. Q-instruct: Improving low-level visual abilities for multi-modality foundation models. In Proceedings of the IEEE/CVF conference on computer vision and… find
  35. Yuzhe Yang, Liwu Xu, Leida Li, Nan Qie, Yaqian Li, Peng Zhang, and Yandong Guo. 2022. Personalized image aesthetics assessment with rich attributes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19861– 19869. find
  36. Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. 2022. find
  37. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917.
  38. Jooyeol Yun and Jaegul Choo. 2024. Scaling up personalized image aesthetic assessment via task vector customization. In European Conference on Computer Vision, pages 323–339. Springer. find
  39. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986.
  40. Zhi Zhang, Srishti Yadav, Fengze Han, and Ekaterina Shutova. 2025. Cross-modal information flow in multimodal large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 19781–19791. find
  41. Zhipeng Zhong, Fei Zhou, and Guoping Qiu. 2023. Aesthetically relevant image captioning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 3733–3741. find
  42. Hantao Zhou, Longxiang Tang, Rui Yang, Guanyi Qin, Yan Zhang, Yutao Li, Xiu Li, Runze Hu, and Guangtao Zhai. 2024a. Uniqa: Unified vision-language pre-training for image quality and aesthetic assessment. arXiv preprint arXiv:2406.01069. — [arXiv:2406.01069](https://arxiv.org/abs/2406.01069) find
  43. Zhaokun Zhou, Qiulin Wang, Bin Lin, Yiwei Su, Rui Chen, Xin Tao, Amin Zheng, Li Yuan, Pengfei Wan, and Di Zhang. 2024b. Uniaa: A unified multi-modal image aesthetic assessment baseline and benchmark. arXiv preprint arXiv:2404.09619. — [arXiv:2404.09619](https://arxiv.org/abs/2404.09619) find
  44. Hancheng Zhu, Leida Li, Jinjian Wu, Sicheng Zhao, Guiguang Ding, and Guangming Shi. 2020. Personalized image aesthetics assessment via meta-learning with bilevel gradient optimization. IEEE Transactions on Cybernetics, 52(3):1798–1811. find
  45. Hancheng Zhu, Yong Zhou, Zhiwen Shao, Wenliang Du, Guangcheng Wang, and Qiaoyue Li. 2022. Personalized image aesthetics assessment via multi-attribute interactive reasoning. Mathematics, 10(22):4181. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ryu-2026-what-do-vision-language-models/