Elamur Research⌕
☰
home / papers / 2026 / Prompt Perturbations Reveal Human-Like Biases in Large Language Model Survey Responses

Prompt Perturbations Reveal Human-Like Biases in Large Language Model Survey Responses

Paper · 2026 · Jens Rupprecht, Georg Ahnert and Markus Strohmaier

Summary

LLMs are increasingly used as stand-ins for human survey participants. It is unclear how stable their answers are, and whether they show the response biases that survey methodology documents in people. The authors built a perturbation framework from 62 value-oriented question–answer pairs, sampled by stratum from the World Values Survey (WVS) Wave 7. They altered either the answer options (e.g., reversed order, an added middle or refusal category) or the question wording (typos, synonym swaps, paraphrases, a priming suffix), and also tested one combined perturbation. Each original and modified item was posed 25 times to 18 instruction-tuned LLMs. They then measured answer consistency with entropy and shifts from the baseline with KL divergence. Larger models proved more stable overall. Still, every model was vulnerable to semantic rewording and to stacked perturbations. Nearly all models also over-selected the final answer option, a recency effect. These results mean synthetic survey data depends heavily on prompt design and should be stress-tested before use. The released dataset supports benchmarking further models.

Key points

  • Scale: 334,800 simulated interviews in total, 18,600 per model. The 62 WVS items cover 10 thematic categories, with scales ranging from 3 to 10 points.
  • Recency bias: almost all models showed it, favouring the last-listed option by up to a factor of 20. The two reasoning models (e.g., Qwen3-30B-A3B-Thinking-2507) showed less of this effect but refused more often.
  • Robustness: Llama-3.3-70B-Instruct and Gemini-1.5-Pro were among the most robust models. Paraphrases and synonym variants, generated with Gemini-1.5-flash, still destabilised even the largest models.
  • Non-responses: Qwen2.5-7B and Llama-3.1-8B often failed to give a valid answer, in more than 30% of cases. For Perception of Elections items, Qwen2.5-7B gave no valid answer in 91.3% of cases, even on unmodified questions.
  • Recommendations: the authors advise using larger non-reasoning models and shorter answer scales. They also suggest thinking carefully before adding middle or refusal categories.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of MannheimMannheim · Germany29 works · 8,347 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSafety Research · 626 worksSociology and Political Science · 626 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Jens Rupprecht, Georg Ahnert and Markus Strohmaier. 2026. Prompt Perturbations Reveal Human-Like Biases in Large Language Model Survey Responses. Proceedings of the Seventh Workshop on Natural Language Processing and Computational Social Science.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.nlpcss-1.1

@inproceedings{rupprecht2026prompt,
  title = {{Prompt Perturbations Reveal Human-Like Biases in Large Language Model Survey Responses}},
  author = {Jens Rupprecht, Georg Ahnert and Markus Strohmaier},
  booktitle = {Proceedings of the Seventh Workshop on Natural Language Processing and Computational Social Science},
  year = {2026},
  doi = {10.18653/v1/2026.nlpcss-1.1},
  url = {https://doi.org/10.18653/v1/2026.nlpcss-1.1}
}

References

42 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. for LLMs: A Unified Robustness Evaluation Framework for Noisy Slot Filling Task. In Natural Language Processing and Chinese Computing, pages 682–694, Cham. Springer Nature Switzerland. find
  2. Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, and Harkirat Behl. 2024. Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. Preprint, arXiv:2404.14219.
  3. Esther Gan, Yiran Zhao, Liying Cheng, Yancan Mao, Anirudh Goyal, Kenji Kawaguchi, Min-Yen Kan, and Michael Shieh. 2024. Reasoning Robustness of LLMs to Adversarial Typographical Errors. Preprint, arXiv:2411.05345. find
  4. Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate. 2023. Out of One, Many: Using Language Models to Simulate Human Samples. Political Analysis, 31(3):337–351.
  5. Mingmeng Geng, Sihong He, and Roberto Trotta. 2024. Are Large Language Models Chameleons? An Attempt to Simulate Social Surveys. Preprint, arXiv:2405.19323. — [arXiv:2405.19323](https://arxiv.org/abs/2405.19323) find
  6. Stacey Aston, James Negen, Marko Nardini, and Ulrik Beierholm. 2021. Central tendency biases must be accounted for to consistently capture Bayesian cue combination in continuous response data. Behavior Research Methods, 54(1):508–521. find
  7. Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, and Soroosh Mariooryad. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. Preprint, arXiv:2403.05530.
  8. John A. Bargh, Mark Chen, and Lara Burrows. 1996. Automaticity of social behavior: Direct effects of trait construct and stereotype activation on action. Journal of Personality and Social Psychology, 71(2):230– 244. find
  9. Matthew Gereti, Alejandro Robinson, Sebastian Williams, Christopher Anderson, and Dominic Walker. 2024. Token-Based Prompt Manipulation for Automated Large Language Model Evaluation. find
  10. James Bisbee, Joshua D. Clinton, Cassy Dorff, Brenton Kenkel, and Jennifer M. Larson. 2024. Synthetic Replacements for Human Survey Data? The Perils of Large Language Models. Political Analysis, 32(4):401–416. find
  11. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, and Alex Vaughan. 2024. The Llama 3 Herd of Models. Preprint, arXiv:2407.21783.
  12. Suhyun Cho, Jaeyun Kim, and Jang Hyun Kim. 2024. LLM-Based Doppelgänger Models: Leveraging Synthetic Data for Human-Like Responses in Survey Simulations. IEEE Access, 12:178917–178927. find
  13. Christian Haerpfer, Ronald Inglehart, Alejandro Moreno, Christian Welzel, Kseniya Kizilova, Jaime Diez-Medrano, Marta Lagos, Pippa Norris, Eduard Ponarin, and Bi Puranen. 2022. World Values Survey Wave 7 (2017-2022) Cross-National Data-Set. find
  14. Lee J. Cronbach. 1946. Response Sets and Test Validity. Educational and Psychological Measurement, 6(4):475–494. find
  15. Paolo Crosetto, Antonio Filippin, Peter Katuščák, and John Smith. 2020. Central tendency bias in belief elicitation. Journal of Economic Psychology, 78:102273. find
  16. Matthias Hagen, Martin Potthast, Marcel Gohsen, Anja Rathgeber, and Benno Stein. 2017. A Large-Scale Query Spelling Correction Corpus. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1261–1264, Shinjuku Tokyo Japan. ACM. find
  17. Vivek Dhakal, Anna Maria Feit, Per Ola Kristensson, and Antti Oulasvirta. 2018. Observations on Typing from 136 Million Keystrokes. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI ’18, pages 1–12, New York, NY, USA. Association for Computing Machinery. find
  18. Edward Tory Higgins. 1996. Knowledge activation: Accessibility, applicability, and salience. In Social Psychology: Handbook of Basic Principles, pages 133–168. The Guilford Press, New York, NY, US. find
  19. H. L. Hollingworth. 1910. The Central Tendency of Judgment. The Journal of Philosophy, Psychology and Scientific Methods, 7(17):461. find
  20. Ricardo Dominguez-Olmedo, Moritz Hardt, and Celestine Mendler-Dünner. 2024. Questioning the survey responses of large language models. In Advances in Neural Information Processing Systems, volume 37, pages 45850–45878. Curran Associates, Inc. find
  21. Jen-tse Huang, Wenxiang Jiao, Man Ho Lam, Eric John Li, Wenxuan Wang, and Michael Lyu. 2024. On the Reliability of Psychological Scales on Large Language Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6152–6173, Miami, Florida, USA. Associati… find
  22. Guanting Dong, Jinxu Zhao, Tingfeng Hui, Daichi Guo, Wenlong Wang, Boqi Feng, Yueyan Qiu, Zhuoma Gongque, Keqing He, Zechen Wang, and Weiran Xu. 2023. Revisit Input Perturbation Problems Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas… find
  23. Yao Qiang, Subhrangshu Nandi, Ninareh Mehrabi, Greg Ver Steeg, Anoop Kumar, Anna Rumshisky, and Aram Galstyan. 2024. Prompt Perturbation Consistency Learning for Robust Language Models. Preprint, arXiv:2402.15833. find
  24. Paul Röttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Rose Kirk, Hinrich Schütze, and Dirk Hovy. 2024. Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models. Preprint, arXiv:2402.16786.
  25. Jarl K. Kampen. 2007. The Impact of Survey Methodology and Context on Central Tendency, Nonresponse and Associations of Subjective Indicators of Government Performance. Quality & Quantity, 41(6):793– 813. find
  26. Jens Rupprecht, Leon Froehling, Claudia Wagner, and Markus Strohmaier. 2026. German General Social Survey Personas: A Survey-Derived Persona Prompt Collection for Population-Aligned LLM Studies. pages 1761–1780, Palma, Mallorca, Spain.
  27. Achim Koch and Michael Blohm. 2016. Nonresponse Bias (GESIS Survey Guidelines)Nonresponse Bias (GESIS Survey Guidelines). Technical report, GESIS - Leibniz Institute for the Social Sciences. find
  28. Howard Schuman and Stanley Presser. 2000. Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context, nachdr. edition. Sage Publ, Thousand Oaks, Calif. find
  29. Jon A. Krosnick. 1991. Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5(3):213– 236. find
  30. Tom Sühr, Florian E. Dorner, Samira Samadi, and Augustin Kelava. 2025. Challenging the Validity of Personality Tests for Large Language Models. In Proceedings of the 5th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, EAAMO ’25, pages 74–81, New York, NY, USA. Associ… find
  31. Jon A. Krosnick and Duane F. Alwin. 1987. An Evaluation of a Cognitive Theory of Response-Order Effects in Survey Measurement. Public Opinion Quarterly, 51(2):201. find
  32. Antonio Laverghetta, Animesh Nighojkar, Jamshidbek Mirzakhalov, and John Licato. 2022. Predicting Human Psychometric Properties Using Computational Language Models. In Quantitative Psychology, pages 151–169, Cham. Springer International Publishing. find
  33. Lindia Tjuatja, Valerie Chen, Sherry Tongshuang Wu, Ameet Talwalkar, and Graham Neubig. 2024. Do LLMs exhibit human-like response biases? A case study in survey design. Preprint, arXiv:2311.04076.
  34. Cheng Li, Jindong Wang, Yixuan Zhang, Kaijie Zhu, Wenxin Hou, Jianxun Lian, Fang Luo, Qiang Yang, and Xing Xie. 2023. Large Language Models Understand and Can be Enhanced by Emotional Stimuli. Preprint, arXiv:2307.11760.
  35. Leah von der Heyde, Anna-Carolina Haensch, and Alexander Wenz. 2025. Vox Populi, Vox AI? Using Language Models to Estimate German Public Opinion. Social Science Computer Review. find
  36. Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul Röttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. 2024. "My Answer is C": First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Models. Preprint, arXiv:2402.14499.
  37. Llama Meta. 2025a. Llama 3.2 Model Card. https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/MODEL_-CARD.md. ↗
  38. Llama Meta. 2025b. Llama 3.3 Model Card. https://github.com/meta-llama/llama-models/blob/main/models/llama3_3/MODEL_-CARD.md. ↗
  39. Evan Weingarten, Qijia Chen, Maxwell McAdams, Jessica Yi, Justin Hepler, and Dolores Albarracín. 2016. From primed concepts to action: A meta-analysis of the behavioral effects of incidentally presented words. Psychological Bulletin, 142(5):472–497. find
  40. Milad Moradi and Matthias Samwald. 2021. Evaluating the Robustness of Neural Language Models to Input Perturbations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1558–1570, Online and Punta Cana, Dominican Republic. Association for Computational Li… find
  41. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jin… find
  42. Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. 2025a. Qwen3 Technical Report. Preprint, arXiv:2505.09388.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rupprecht-2026-prompt-perturbations-reveal-human-like/