Elamur Research⌕
☰
home / papers / 2026 / Neural network embeddings recover value dimensions from psychometric survey items on par with human data

Neural network embeddings recover value dimensions from psychometric survey items on par with human data

Paper · 2026 · Max Pellert, Clemens M Lechner, Indira Sen et al.

Summary

In psychometrics, questionnaires are refined by having many people rate candidate items and then examining how the responses correlate. This process is expensive and suffers from fatigue, response styles and low motivation. Earlier work tried to replace human ratings with embeddings of the item text. It hit a key obstacle: similarities between sentence vectors almost never turn negative, yet opposing constructs and reverse-keyed items should. The authors propose SQuID (Survey and Questionnaire Item Embeddings Differentials), a processing step applied to embeddings from general-purpose models. It produces negative associations between dimensions without fine-tuning on domain data or re-annotating training sets. They embedded the items of the Revised Portrait Value Questionnaire (PVQ-RR) with several models and compared the results to human data on internal consistency, correlations between value dimensions and multidimensional scaling (MDS) configurations. The recovered structure accounts for 55% of the variance in human dimension-to-dimension similarities, and its MDS layout aligns with pooled cross-national data. The findings suggest that embeddings can be a cheap, scalable and flexible complement to human raters in instrument design.

Key points

  • SQuID addresses the lack of negative correlations in item embeddings without domain-specific fine-tuning. Earlier approaches such as the fine-tuned "mpnet-personality" model were included in the comparison.
  • The embedding-based dimension similarities explain 55% of the variance found in human PVQ-RR data.
  • MDS configurations match human data pooled from 49 countries.
  • In generalization tests on three personality inventories (IPIP, BFI-2, HEXACO), SQuID consistently widened the range of obtained correlations.
  • The best-performing model was Linq-Embed-Mistral (7.11B parameters), which also performs well on the Massive Text Embedding Benchmark. The authors acknowledge that PVQ-RR items may appear in training corpora.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of MannheimMannheim · Germany29 works · 8,347 citations🔬Barcelona Supercomputing CenterBarcelona · Spain14 works · 53 citations🔬GESIS - Leibniz Institute for the Social SciencesMannheim · Germany10 works · 5 citations

 Fields, topics and concepts

📚 Shelves 8The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksDecision Sciences · 209 worksSocial Psychology · 306 worksManagement Science and Operations Research · 108 works

Cite

Cite as Max Pellert, Clemens M Lechner, Indira Sen and Markus Strohmaier. 2026. Neural network embeddings recover value dimensions from psychometric survey items on par with human data. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.303

@inproceedings{pellert2026neural,
  title = {{Neural network embeddings recover value dimensions from psychometric survey items on par with human data}},
  author = {Max Pellert, Clemens M Lechner, Indira Sen and Markus Strohmaier},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.303},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.303}
}

References

49 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tommaso Feraco and Enrico Toffalini. 2025. SEM-beddings: How to evaluate model misfit before data collection using large-language models. Frontiers in Psychology, 15:1433339. find
  2. Suhaib Abdurahman, Huy Vu, Wanling Zou, Lyle Ungar, and Sudeep Bhatia. 2024. A deep learning approach to personality assessment: Generalizing across items and expanding the reach of survey-based research. Journal of Personality and Social Psychology, 126(2):312–331. find
  3. Gati Aher, Rosa I. Arriaga, and Adam Tauman Kalai. 2023. Using large language models to simulate multiple humans and replicate human subject studies. In Proceedings of the 40th International Conference on Machine Learning, ICML’23, Honolulu, Hawaii, USA. JMLR.org. find
  4. Georg Ahnert, Max Pellert, David Garcia, and Markus Strohmaier. 2025. Extracting Affect Aggregates from Longitudinal Social Media Data with Temporal Adapters for Large Language Models. Preprint, arXiv:2409.17990. preprocessing, estimating, and analyzing semantic networks. Psychological Methods. — [arXiv:2409.17990](https://arxiv.org/abs/2409.17990) find
  5. Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray. 2023. Can AI language models replace human participants? Trends in Cognitive Sciences. find
  6. Ronald Fischer, Markus Luczak-Roesch, and Johannes A. Karl. 2023. What does ChatGPT return about human values? Exploring value bias in Chat-GPT using a descriptive value theory. arXiv preprint. ArXiv:2304.03612 [cs]. — [arXiv:2304.03612](https://arxiv.org/abs/2304.03612) find
  7. Lewis R. Goldberg. 1992. The development of markers for the Big-Five factor structure. Psychological Assessment, 4(1):26–42. find
  8. Nelson Goodman. 1972. Seven strictures on similarity. In Problems and projects. Bobs-Merril. find
  9. Patrick J.F. Groenen and Ingwer Borg. 2015. Multidimensional Scaling II. In International Encyclopedia of the Social & Behavioral Sciences, pages 40–47. Elsevier. find
  10. Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate. 2023. Out of One, Many: Using Language Models to Simulate Human Samples. Political Analysis, pages 1–15.
  11. Nigel Guenole, E. Damiano D’Urso, Andrew Samo, Tianjun Sun, and Jonas M B Haslbeck. 2025. Enhancing Scale Development: Pseudo Factor Analysis of Language Embedding Similarity Matrices. find
  12. Jan Ketil Arnulf and Kai Rune Larsen. 2021. Semantic and ontological structures of psychological attributes. In Measuring and Modeling Persons and Situations, pages 69–101. Elsevier. find
  13. Friedrich M. Götz, Rakoen Maertens, Sahil Loomba, and Sander van der Linden. 2023. Let the algorithm speak: How to use neural networks for automatic item generation in psychological scale development. Psychological Methods. find
  14. Jan Ketil Arnulf, Kai Rune Larsen, Øyvind Lund Martinsen, and Chih How Bong. 2014. Predicting Survey Responses: How and Why Semantics Shape Survey Statistics on Organizational Behaviour. PLoS ONE, 9(9):e106361. find
  15. Thilo Hagendorff. 2023. Machine Psychology: Investigating Emergent Capabilities and Behavior in Large Language Models Using Psychological Methods. Preprint, arXiv:2303.13988. — [arXiv:2303.13988](https://arxiv.org/abs/2303.13988) find
  16. Marcel Binz and Eric Schulz. 2023. Using cognitive psychology to understand GPT-3. Proceedings of the National Academy of Sciences, 120(6):e2218523120. Publisher: Proceedings of the National Academy of Sciences. find
  17. Chanyeol Choi, Junseong Kim, Seolhwa Lee, Jihoon Kwon, Sangmo Gu, Yejin Kim, Minkyung Cho, and Jy-yong Sohn. 2024. Linq-Embed-Mistral Technical Report. Preprint, arXiv:2412.03223. — [arXiv:2412.03223](https://arxiv.org/abs/2412.03223) find
  18. Alexander P. Christensen and Yoed N. Kenett. 2021. Semantic network analysis (SemNA): A tutorial on Darryl B. Hill and Brian L. B. Willoughby. 2005. The Development and Validation of the Genderism and Transphobia Scale. Sex Roles, 53(7-8):531–544. find
  19. Björn E. Hommel, Franz-Josef M. Wollang, Veronika Kotova, Hannes Zacher, and Stefan C. Schmukle. 2022. Transformer-Based Deep Neural Language Modeling for Construct-Specific Automatic Item Generation. Psychometrika, 87(2):749–772. find
  20. Björn Erik Hommel and Ruben C. Arslan. 2025. Language models accurately infer correlations between psychological items and scales from text alone. find
  21. Xinshuo Hu, Zifei Shan, Xinping Zhao, Zetian Sun, Zhenyu Liu, Dongfang Li, Shaolin Ye, Xinyuan Wei, Qian Chen, Baotian Hu, Haofen Wang, Jun Yu, and Min Zhang. 2025. KaLM-Embedding: Superior Training Data Brings A Stronger Embedding Model. Preprint, arXiv:2501.01028. — [arXiv:2501.01028](https://arxiv.org/abs/2501.01028) find
  22. Oliver P. John, Laura P. Naumann, and Christopher J. Soto. 2008. Paradigm shift to the integrative Big Five trait taxonomy: History, measurement, and conceptual issues. In Handbook of personality: Theory and research, 3rd ed, pages 114–158. The Guilford Press, New York, NY, US. find
  23. Kwanghee Jung and Yoshio Takane. 2015. Multidimensional Scaling I. In International Encyclopedia of the Social & Behavioral Sciences, pages 34–39. Elsevier. find
  24. Paula S. Kerr and Ronald R. Holden. 1996. Development of the Gender Role Beliefs Scale (GRBS). Journal of Social Behavior & Personality, 11:3–16. Place: US Publisher: Select Press. find
  25. Oscar N. E. Kjell, Sverker Sikström, Katarina Kjell, and H. Andrew Schwartz. 2022. Natural language analyzed with AI-based transformers predict traditional subjective well-being measures approaching the theoretical upper limits in accuracy. Scientific Reports, 12(1):3918. find
  26. Clemens M. Lechner, Constanze Beierlein, Eldad Davidov, and Shalom H. Schwartz. 2024. Measuring the Four Higher-Order Values in Schwartz’s Theory: Validation of a 17-Item Inventory. Journal of Personality Assessment, 106(5):651–664. find
  27. Kibeom Lee and Michael C. Ashton. 2018. Psychometric Properties of the HEXACO-100. Assessment, 25(5):543–556. find
  28. Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. MTEB: Massive Text Embedding Benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014–2037, Dubrovnik, Croatia. Association for Computational Linguistic…
  29. August Håkan Nilsson, J. Malte Runge, Adithya V. Ganesan, Carl Viggo N. G. Lövenstierne, Nikita Soni, and Oscar N. E. Kjell. 2025. Automatic implicit motive codings are at least as accurate as humans’ and 99% faster. Journal of Personality and Social Psychology, 128(6):1371–1392. find
  30. Max Pellert, Clemens M. Lechner, Claudia Wagner, Beatrice Rammstedt, and Markus Strohmaier. 2024. AI Psychometrics: Assessing the Psychological Profiles of Large Language Models Through Psychometric Inventories. Perspectives on Psychological Science. find
  31. Lydia Repke, Lukas Birkenmaier, and Clemens Lechner. 2024. Validity in Survey Research - From Research Design to Measurement Instruments (GESIS Survey Guidelines). Technical report, GESIS - Leibniz-Institute for the Social Sciences. find
  32. Hannes Rosenbusch, Florian Wanders, and Ilse L. Pit. 2020. The Semantic Scale Network: An online tool to detect semantic overlap of psychological scales and prevent scale redundancies. Psychological Methods, 25(3):380–392. find
  33. Gerard Saucier and Lewis R. Goldberg. 1996. Evidence for the Big Five in analyses of familiar English personality adjectives. European Journal of Personality, 10(1):61–77. find
  34. Hannes Leitgeb. 2007. A New Analysis of Quasianaly-sis. Journal of Philosophical Logic, 36(2):181–226. Publisher: Springer. find
  35. Zach C. Schudson and Sari M. van Anders. 2022. Gender/sex diversity beliefs: Scale construction, validation, and links to prejudice. Group Processes & Intergroup Relations, 25(4):1011–1036. find
  36. Lea Löhn, Niklas Kiehne, Alexander Ljapunov, and Wolf-Tilo Balke. 2024. Is machine psychology here? On requirements for using human psychological tests on large language models. In Proceedings of the 17th International Natural Language Generation Conference, pages 230–242, Tokyo, Japan. Association… find
  37. Shalom H. Schwartz and Jan Cieciuch. 2022. Measuring the Refined Theory of Individual Values in 49 Cultural Groups: Psychometrics of the Revised Portrait Value Questionnaire. Assessment, 29(5):1005– 1019. find
  38. Urbano Lorenzo-Seva and Jos M. F. Ten Berge. 2006. Tucker’s Congruence Coefficient as a Meaningful Index of Factor Similarity. Methodology, 2(2):57– 64. find
  39. Patrick Mair, Patrick Groenen, and Jan de Leeuw. 2022. Multidimensional scaling using majorization: SMA-COF in R. Journal of Statistical Software, 102(10). find
  40. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. Preprint, arXiv:1301.3781.
  41. Shalom H. Schwartz, Jan Cieciuch, Michele Vecchione, Eldad Davidov, Ronald Fischer, Constanze Beierlein, Alice Ramos, Markku Verkasalo, Jan-Erik Lönnqvist, Kursad Demirutku, Ozlem Dirilen-Gumus, and Mark Konty. 2012. Refining the theory of basic individual values. Journal of Personality and Social P… find
  42. Greg Serapio-García, Mustafa Safdari, Clément Crepy, Luning Sun, Stephen Fitz, Peter Romero, Marwa Abdulhai, Aleksandra Faust, and Maja Matarić. 2025. Personality Traits in Large Language Models. Preprint, arXiv:2307.00184.
  43. Christopher J. Soto and Oliver P. John. 2017. The next Big Five Inventory (BFI-2): Developing and assessing a hierarchical model with 15 facets to enhance bandwidth, fidelity, and predictive power. Journal of Personality and Social Psychology, 113(1):117–143. find
  44. Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang, Markus Krimmel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Nan Wang, and Han Xiao. 2024. Jina-embeddings-v3: Multilingual Embeddings With Task LoRA. Preprint, arXiv:2409.10173. find
  45. Tom Sühr, Florian E. Dorner, Samira Samadi, and Augustin Kelava. 2024. Challenging the Validity of Personality Tests for Large Language Models. Preprint, arXiv:2311.05297. — [arXiv:2311.05297](https://arxiv.org/abs/2311.05297) find
  46. Isaac Waller and Ashton Anderson. 2021. Quantifying social organization and political polarization in online platforms. Nature. find
  47. John Wieting and Douwe Kiela. 2019. No Training Required: Exploring Random Encoders for Sentence Classification. arXiv preprint. ArXiv:1901.10444 [cs]. — [arXiv:1901.10444](https://arxiv.org/abs/1901.10444) find
  48. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: Human language technologies, volume 1 (long pa…
  49. Dirk U. Wulff and Rui Mata. 2025. Semantic embeddings reveal and address taxonomic incommensurability in psychological measurement. Nature Human Behaviour. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pellert-2026-neural-network-embeddings-recover-value/