Elamur Research⌕
☰
home / papers / 2026 / Who Speaks for Whom? LLM-Generated Survey Data as a Proxy for Public Opinion

Who Speaks for Whom? LLM-Generated Survey Data as a Proxy for Public Opinion

Paper · 2026 · Radhakrishnan Venkatakrishnan, Travis Brodbeck and Michael D. Young

Summary

Survey researchers face rising costs and falling response rates, which makes LLM "personas" that answer questionnaires in place of people an attractive option. The authors test whether such synthetic answers can actually stand in for human public opinion. They take the American Values Survey, run by Siena in 2021, and give each human respondent a persona prompt built from nine demographic attributes. Four open instruction-tuned models (two Llama, two Olmo, at small and large sizes) then answer the same value statements. The evaluation combines ICC, Earth Mover Distance, variance ratios, a lexicon-based hedging score on reasoning traces, and regression on the demographic drivers of the scores.

The models look plausible in aggregate and are stable across repeated runs. However, their answers cluster tightly around the centre and miss the spread found in real human responses. Their reasoning often hedges in ways that do not match the definitive score they give, and the distortions are worse for some demographic groups. The authors conclude that synthetic respondents are not yet a valid replacement for human survey data, although they may help with pre-testing or generating hypotheses.

Key points

  • Data: American Values Survey (N = 6,077), with 34 ten-point Likert items grouped into Liberty, Equality, and Progress subscales. Personas use 9 demographics: age, race, ethnicity, gender, employment, education, party, state, and voter registration.
  • Models and setup: Llama-3.1-8B-Instruct, Llama-3.3-70B-Instruct, Olmo-3-7B-Instruct, and Olmo-3.1-32B-Instruct were each run 5 times per respondent at temperature 0.1.
  • Stereotyped demographic effects:
    • educ_Grade school strongly lowers Progress scores (coefficients −0.43 to −0.50).
    • partyid_Republican lowers Equality (β = −0.40), which flattens the variety of Republican views into one position.
    • Registered-voter status inflates scores across the board.
  • Fixed group treatment: Models treat groups such as unemployed, non-binary, and transgender personas as uniform blocks and ignore the persona's other traits. They give little weight to age and geography. Increased hedging on some identities resembles learned safety behaviour.
  • Limitations and resources: The survey's aggregate results may appear in training data, and base models were not compared with their Instruct versions. Code and data are available on GitHub at rvenka31/llm-proxy-public-opinion-surveys.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Arts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 worksSociology and Political Science · 577 worksGeneral Social Sciences · 227 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Radhakrishnan Venkatakrishnan, Travis Brodbeck and Michael D. Young. 2026. Who Speaks for Whom? LLM-Generated Survey Data as a Proxy for Public Opinion. Proceedings of the Seventh Workshop on Natural Language Processing and Computational Social Science.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.nlpcss-1.9

@inproceedings{venkatakrishnan2026who,
  title = {{Who Speaks for Whom? LLM-Generated Survey Data as a Proxy for Public Opinion}},
  author = {Radhakrishnan Venkatakrishnan, Travis Brodbeck and Michael D. Young},
  booktitle = {Proceedings of the Seventh Workshop on Natural Language Processing and Computational Social Science},
  year = {2026},
  doi = {10.18653/v1/2026.nlpcss-1.9},
  url = {https://doi.org/10.18653/v1/2026.nlpcss-1.9}
}

References

42 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate. 2023. Out of One, Many: Using Language Models to Simulate Human Samples. Political Analysis, 31(3):337–351.
  2. Yash Bhatnagar. 2026. Breaking the turing test: Testing the relevance of the turing test against modern llms. International Journal for Research in Engineering Application Management, pages 1–. find
  3. J Michael Brick, Pat D Brick, Sarah Dipko, Stanley Presser, Clyde Tucker, and Yangyang Yuan. 2007. Cell phone survey feasibility in the us: Sampling and calling cell numbers versus landline numbers. Public Opinion Quarterly, 71(1):23–39. find
  4. Mario Callegaro and Charles DiSogra. 2008. Computing response metrics for online panels. Public opinion quarterly, 72(5):1008–1032. find
  5. Mick P Couper and Peter V Miller. 2008. Web survey methods: Introduction. Public opinion quarterly, 72(5):831–835. find
  6. Rachel E Davis, Timothy P Johnson, Sunghee Lee, and Christopher Werner. 2019. Why do latino survey respondents acquiesce? respondent and interviewer characteristics as determinants of cultural patterns of acquiescence among latino survey respondents. Cross-Cultural Research, 53(1):87–115. find
  7. Jonathan Eggleston. 2024. Frequent survey requests and declining response rates: evidence from the 2020 census and household surveys. Journal of Survey Statistics and Methodology, 12(5):1138–1156. find
  8. Aaron Fanous, Jacob Goldberg, Ank Agarwal, Joanna Lin, Anson Zhou, Sonnet Xu, Vasiliki Bikia, Roxana Daneshjou, and Sanmi Koyejo. 2025. Syceval: Evaluating llm sycophancy. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 8, pages 893–900. find
  9. Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625–630. find
  10. Anna Foka, Gabriele Griffin, Dalia Ortiz Pablo, Paulina Rajkowska, and Sushruth Badri. 2025. Tracing the bias loop: Ai, cultural heritage and bias-mitigating in practice. AI & SOCIETY, 40(8):5835–5847. find
  11. Mansour Ghafourifard. 2024. Survey fatigue in questionnaire based research: The issues and solutions. Journal of caring sciences, 13(4):214–215. find
  12. Chris Gibson and Daniel Lipinski. 2021. Americans, deeply divided, yet share core values of equality, liberty progress. find
  13. Kristian González Barman, Simon Lohse, and Henk W de Regt. 2025. Reinforcement learning from human feedback in llms: Whose culture, whose values, whose perspectives? k. gonzález barman et al. Philosophy & Technology, 38(2):35. find
  14. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  15. Christian Haerpfer, Ronald Inglehart, Alejandro Moreno, Christian Welzel, Kseniya Kizilova, Jaime Diez-Medrano, Marta Lagos, Pippa Norris, Eduard Ponarin, Bjorn Puranen, and 1 others. 2022. World values survey: Round seven-country-pooled datafile version 5.0. serving with pagedattention. In Proceedi… find
  16. Yoichi Hayashi. 2024. Prospects for revolutionary and popular ai technology following the launch of chatgpt in 2023. find
  17. Lujun Li, Lama Sleem, Geoffrey Nichil, Radu State, and 1 others. 2025. Exploring the impact of temperature on large language models: Hot or cold? Procedia Computer Science, 264:242–251. find
  18. Morten Hjortskov, Christian Bøtcher Jacobsen, and Anne Mette Kjeldsen. 2023. Choir of believers? experimental and longitudinal evidence on survey participation, response bias, and public service motivation. International Public Management Journal, 26(2):281–304. find
  19. Jumayel Islam, Lu Xiao, and Robert E Mercer. 2020. A lexicon-based approach for detecting hedges in informal text. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3109–3113.
  20. Liwei Jiang, Taylor Sorensen, Sydney Levine, and Yejin Choi. 2024. Can Language Models Reason about Individualistic Human Values and Preferences? Preprint, arXiv:2410.03868.
  21. Timothy P Johnson, Henning Silber, and Jill E Darling. 2024. Public perceptions of pollsters in the united states: experimental evidence. Social Science Quarterly, 105(1):114–127. find
  22. Indu Joshi, Marcel Grimmer, Christian Rathgeb, Christoph Busch, Francois Bremond, and Antitza Dantcheva. 2024. Synthetic data in human analysis: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(7):4957–4976. find
  23. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2023. Large Language Models are Zero-Shot Reasoners. arXiv preprint. ArXiv:2205.11916 [cs].
  24. Sarah Kreps, R Miles McCain, and Miles Brundage. 2022. All the news that’s fit to fabricate: Ai-generated text as a tool of media misinformation. Journal of experimental political science, 9(1):104– 117. find
  25. Jon A Krosnick. 1991. Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied cognitive psychology, 5(3):213– 236. find
  26. Jon A Krosnick, Allyson L Holbrook, Matthew K Berent, Richard T Carson, W Michael Hanemann, Raymond J Kopp, Robert Cameron Mitchell, Stanley Presser, Paul A Ruud, V Kerry Smith, and 1 others. 2002. The impact of" no opinion" response options on data quality: non-attitude reduction or an invitation t… find
  27. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model Michael W Link, Ali H Mokdad, Dale Kulp, and Ashley Hyon. 2006. Has the national do not call registry helped or hu… find
  28. Michael W Link and Robert W Oldendick. 1999. Call screening: Is it really a problem for survey research? The Public Opinion Quarterly, 63(4):577–589. find
  29. Andy Markus. 2025. AT&T tests new AI digital receptionist. https://about.att.com/blogs/ 2025/ai-digital-receptionist.html. ↗
  30. Robert W Oldendick and Michael W Link. 1994. The answering machine generation: who are they and what problem do they pose for survey research? Public Opinion Quarterly, 58(2):264–273. find
  31. Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, and 1 others. 2025. Olmo 3. arXiv preprint arXiv:2512.13961.
  32. Giada Pistilli, Alina Leidinger, Yacine Jernite, Atoosa Kasirzadeh, Alexandra Sasha Luccioni, and Margaret Mitchell. 2024. Civics: Building a dataset for examining culturally-informed values in large language models. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 7, pag… find
  33. David C Pyrooz, James A Densley, and Jose Antonio Sanchez. 2025. Are online opt-in panels viable data sources on hard-to-reach populations? population and relational inferences on gang membership in the united states. International Criminology, pages 1– 17. find
  34. Patric M Reinbold. 2020. Taking artificial intelligence beyond the turing test. Wis. L. Rev., page 873. find
  35. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? In International conference on machine learning, pages 29971–30004. PMLR.
  36. Patrick E Shrout and Joseph L Fleiss. 1979. Intraclass correlations: uses in assessing rater reliability. Psychological bulletin, 86(2):420. find
  37. Jorre Vannieuwenhuyze, Geert Loosveldt, and Geert Molenberghs. 2010. A method for evaluating mode effects in mixed-mode surveys. Public opinion quarterly, 74(5):1027–1045. find
  38. Michael F Weeks, Richard A Kulka, and Stephanie A Pierson. 1987. Optimal call scheduling for a telephone survey. Public Opinion Quarterly, pages 540– 549. find
  39. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, and 1 others. 2022. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682.
  40. Yu Xie and Yueqi Xie. 2025. Variance reduction in output from generative ai. arXiv preprint arXiv:2503.01033. — [arXiv:2503.01033](https://arxiv.org/abs/2503.01033) find
  41. Wenlong Zhao, Debanjan Mondal, Niket Tandon, Danica Dillion, Kurt Gray, and Yuling Gu. 2024. World-ValuesBench: A Large-Scale Benchmark Dataset for Multi-Cultural Value Awareness of Language Models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Reso… find
  42. Ke Zhou, Marios Constantinides, and Daniele Quercia. 2025. Should llms be weird? exploring weirdness and human rights in large language models. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 8, pages 2808–2820. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/venkatakrishnan-2026-who-speaks-whom-llm-generated/