Elamur Research⌕
☰
home / papers / 2026 / R.U.Psycho? A Framework for Robust Unified Psychometric Testing of Language Models

R.U.Psycho? A Framework for Robust Unified Psychometric Testing of Language Models

Summary

Psychometric questionnaires written for people are increasingly given to generative LLMs to profile model traits, benchmark alignment, or replace human participants through persona simulation. These studies are hard to reproduce. Model outputs fluctuate, and results shift with prompt wording and ordering, generation parameters, and model version, choices that papers often document poorly. The authors respond with R.U.Psycho, an open-source Python package built on LangChain that lets researchers run such experiments with little programming.

A whole experiment is specified in one JSON configuration file, which also serves as its documentation. The file covers:

  • the models and their generation settings,
  • prompt templates with placeholders,
  • persona attributes,
  • the questionnaire, with its instructions and answer options.

The pipeline collects free-text answers, cleans and checks them, maps them onto the allowed options, and exports the results. The authors tested it on four psychometric instruments and one thought experiment, and the outcomes agreed with earlier published findings. The tool's value is that machine-psychology studies become easier to standardize, share, and repeat.

Key points

  • Pipeline: four stages run in sequence.
    1. Experiment definition.
    2. Response generation.
    3. Post-processing, with Cleaner, Validator and Judge components.
    4. Export to CSV, DataFrame or JSONL.
  • Model access: OpenAI-compatible APIs, local Huggingface models (with optional quantization) and the Huggingface Inference API.
    • Both chat-style and single-prompt formats are supported.
    • Temperature, max output tokens and random seeds are fixed in the config.
  • Answer method: answers are generated as free text rather than scored by token probabilities, because the authors cite known problems with probability-based measurement (Wang et al., 2024).
    • Several responses are sampled per question for each model, seed and persona.
  • Personas: stored as attribute dictionaries (e.g., title, name, age, ethnicity) and inserted into prompt templates. The attributes stay available for stratified analysis later.
  • Questionnaire adaptation and resources: human-oriented instructions, such as asking participants to circle an answer, are rewritten into a form suitable for LLMs.
    • The code is available at github.com/julianschelb/rupsycho.
    • An appendix lists 61 strategies for rephrasing answer statements, covering tone, sentence structure, metaphors, and JSON/HTML/XML/YAML formats.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4d7ofew6usug

@inproceedings{schelb2026r,
  title = {{R.U.Psycho? A Framework for Robust Unified Psychometric Testing of Language Models}},
  author = {Schelb, Julian and Borin, Orr and Garcia, David and Spitz, Andreas},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4d7ofew6usug},
  url = {https://doi.org/10.63317/4d7ofew6usug}
}

References

50 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gati V. Aher, Rosa I. Arriaga, and Adam Tauman Kalai. 2023. Using large language models to simulate multiple humans and replicate human subject studies. In International Conference on Machine Learning, ICML. find
  2. L. B. Allal, A. Lozhkov, E. Bakouch, L. von Werra, and T. Wolf. 2024. Smollm — blazingly fast and remarkably powerful. Unpublished manuscript. Retrieved from Hugging Face Blog. find
  3. Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate. 2023. Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3):337–351. find our card
  4. Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Jon Ander Campos, Yi Chern Tan, Kelly Marchisio, Max Bartolo, Sebastian Ruder, Acyr Locatelli, Julia Kreutzer, Nick Frosst, Aidan N. Gomez, Phil Blunsom, Marzieh Fadaee, Ahmet Üstün… find
  5. Aya 23: Open weight releases to further multilingual progress. CoRR, abs/2405.15032. find our card
  6. Christopher A Bail. 2024. Can generative ai improve social science? Proceedings of the National Academy of Sciences, 121(21):e2314021121. find
  7. Simone Balloccu, Patrícia Schmidtová, Mateusz Lango, and Ondrej Dusek. 2024. Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source LLMs. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, EACL. find our card
  8. A. T. Beck, C. H. Ward, M. Mendelson, J. Mock, and J. Erbaugh. 1961. An inventory for measuring depression. Archives of General Psychiatry, 4(6):561–571. find
  9. Aaron T. Beck, Robert A. Steer, and Margery G. Carbin. 1988. Psychometric properties of the beck depression inventory: Twenty-five years of evaluation. Clinical Psychology Review, 8(1):77– 100. find
  10. Marcel Binz and Eric Schulz. 2023. Using cognitive psychology to understand gpt-3. Proceedings of the National Academy of Sciences, 120(6):e2218523120. find
  11. Bojana Bodroža, Bojana M Dinić, and Ljubiša Bojić. 2024. Personality testing of large language models: limited temporal stability, but highlighted prosociality. Royal Society Open Science, 11(10):240180. find
  12. Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray. 2023. Can ai language models replace human participants? Trends in Cognitive Sciences, 27(7):597–600. find
  13. Florian Dorner, Tom Sühr, Samira Samadi, and Augustin Kelava. 2023. Do personality tests generalize to large language models? In Socially Responsible Language Modelling Research, So-LaR Workshop@NeurIPS’23. find
  14. Jinhao Duan, Renming Zhang, James Diffenderfer, Bhavya Kailkhura, Lichao Sun, Elias Stengel-Eskin, Mohit Bansal, Tianlong Chen, and Kaidi Xu. 2024. Gtbench: Uncovering the strategic reasoning limitations of llms via game-theoretic evaluations. CoRR, abs/2402.12348. — [arXiv:2402.12348](https://arxiv.org/abs/2402.12348) find
  15. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, and et al. 2024. The llama 3 herd of models. CoRR, abs/2407.21783. find
  16. Salvatore Giorgi, Tingting Liu, Ankit Aich, Kelsey Jane Isman, Garrick Sherman, Zachary Fried, João Sedoc, Lyle Ungar, and Brenda Curtis. 2024. Modeling human subjectivity in LLMs using explicit and implicit human factors in personas. In Findings of the Association for Computational Linguistics: EMN… find
  17. Shashank Gupta, Vaishnavi Shrivastava, Ameet Deshpande, Ashwin Kalyan, Peter Clark, Ashish Sabharwal, and Tushar Khot. 2024. Bias runs deep: Implicit reasoning biases in persona-assigned llms. In The Twelfth International Conference on Learning Representations, ICLR. find
  18. Thilo Hagendorff. 2023. Machine psychology: Investigating emergent capabilities and behavior in large language models using psychological methods. CoRR, abs/2303.13988. — [arXiv:2303.13988](https://arxiv.org/abs/2303.13988) find
  19. E. Tory Higgins, Ronald S. Friedman, Robert E. Harlow, Lorraine Chen Idson, Ozlem N. Ayduk, and Amy Taylor. 2001. Achievement orientations from subjective histories of success: Promotion pride versus prevention pride. European Journal of Social Psychology, 31(1):3–23. find
  20. John J Horton. 2023. Large language models as simulated economic agents: What can we learn from homo silicus? Working Paper 31122, National Bureau of Economic Research. find
  21. Tiancheng Hu and Nigel Collier. 2024. Quantifying the persona effect in LLM simulations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ACL. find
  22. Tiancheng Hu, Yara Kyrychenko, Steve Rathje, Nigel Collier, Sander van der Linden, and Jon Roozenbeek. 2025. Generative language models exhibit social identity biases. Nature Computational Science, 5(1):65–75. find
  23. Jen-tse Huang, Wenxiang Jiao, Man Ho Lam, Eric John Li, Wenxuan Wang, and Michael Lyu. 2024. On the reliability of psychological scales on large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. find
  24. Silke Husse and Andreas Spitz. 2022. Mind your bias: A critical review of bias detection methods for contextual language models. In Findings of the Association for Computational Linguistics: EMNLP. find
  25. Maurice Jakesch, Jeffrey T Hancock, and Mor Naaman. 2023. Human heuristics for ai-generated language are flawed. Proceedings of the National Academy of Sciences, 120(11):e2208839120. find
  26. Oliver P John, Eileen M Donahue, and Robert L Kentle. 1991. Big Five Inventory (BFI). APA PsycTests. find
  27. Lea Löhn, Niklas Kiehne, Alexander Ljapunov, and Wolf-Tilo Balke. 2024. Is machine psychology here? on requirements for using human psychological tests on large language models. In Proceedings of the 17th International Natural Language Generation Conference. find
  28. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL. find our card
  29. Inbal Magar and Roy Schwartz. 2022. Data contamination: From memorization to exploitation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL. find
  30. Francesco Manigrasso, Stefan F. Schouten, Lia Morra, and Peter Bloem. 2024. Probing llms for logical reasoning. In 18th International Conference on Neural-Symbolic Learning and Reasoning, NeSy, pages 257–278. find
  31. Marilù Miotto, Nicola Rossberg, and Bennett Kleinberg. 2022. Who is GPT-3? an exploration of personality, values and demographics. In Proceedings of the Fifth Workshop on Natural Language Processing and Computational Social Science (NLP+CSS). find
  32. OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08774. find our card
  33. Max Pellert, Clemens M Lechner, Claudia Wagner, Beatrice Rammstedt, and Markus Strohmaier. 2024. AI psychometrics: Assessing the psychological profiles of large language models through psychometric inventories. Perspectives on Psychological Science, 19(5):808–826. find
  34. Nikolay B. Petrov, Gregory Serapio-García, and Jason Rentfrow. 2024. Limited ability of llms to simulate human psychological behaviours: a psychometric analysis. CoRR, abs/2405.07248. — [arXiv:2405.07248](https://arxiv.org/abs/2405.07248) find
  35. Pouya Pezeshkpour and Estevam Hruschka. 2024. Large language models sensitivity to the order of options in multiple-choice questions. In Findings of the Association for Computational Linguistics: NAACL 2024. find our card
  36. ProtectAI.com. 2024. Fine-tuned distilroberta-base for rejection in the output detection. find
  37. Yuanyi Ren, Haoran Ye, Hanjun Fang, Xin Zhang, and Guojie Song. 2024. Valuebench: Towards comprehensively evaluating value orientations and understanding of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ACL. find
  38. Paul Röttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Kirk, Hinrich Schütze, and Dirk Hovy. 2024. Political compass or spinning arrow? towards more meaningful evaluations for values and opinions in large language models. In Proceedings of the 62nd Annual Meeting of the Association… find our card
  39. Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. 2023. Can ai-generated text be reliably detected? CoRR, abs/2303.11156. find our card
  40. Zach C Schudson and Sari M van Anders. 2022. Gender/sex diversity beliefs: Scale construction, validation, and links to prejudice. Group Processes & Intergroup Relations, 25(4):1011– 1036. find
  41. Bangzhao Shu, Lechen Zhang, Minje Choi, Lavinia Dunagan, Lajanugen Logeswaran, Moontae Lee, Dallas Card, and David Jurgens. 2024. You don‘t need a personality test to know these models are unreliable: Assessing the reliability of large language models on psychometric instruments. In Proceedings of t… find
  42. Judith Jarvis Thomson. 1984. The trolley problem. Yale LJ, 94:1395. find
  43. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, and et al. 2023. Zephyr: Direct distillation of LM alignment. CoRR, abs/2310.16944. find
  44. Tomer D. Ullman. 2023. Large language models fail on trivial alterations to theory-of-mind tasks. CoRR, abs/2302.08399. find
  45. Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West. 2023. Artificial artificial artificial intelligence: Crowd workers widely use large language models for text production tasks. CoRR, abs/2306.07899. find
  46. Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul Röttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. 2024. "My Answer is C": First-token probabilities do not match text answers in instruction-tuned language models. In Findings of the Association for Computational Linguistics, ACL. find
  47. Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with chatgpt. CoRR, abs/2302.11382. find our card
  48. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, and et al. 2024. Qwen2.5 technical report. CoRR, abs/2412.15115. find
  49. Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations, ICLR. find
  50. Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2024. Can large language models transform computational social science? Comput. Linguistics, 50(1):237– 291. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schelb-2026-r-u-psycho-framework-robust/