Elamur Research⌕
☰
home / papers / 2025 / Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations

Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations

Paper · 2025 · Yong Cao, Haijiang Liu, Arnav Arora et al.

Summary

Large cross-national surveys are expensive and slow. The authors ask whether LLMs can be trained, rather than just prompted, to predict how a country's population would divide its answers across a survey question's options. They frame this as group-level distribution prediction and build datasets from the 2017–2022 wave of the World Values Survey (WVS), covering 65 countries, in English and Chinese. Questions and countries are partitioned so that some topics and regions, such as African and medium-GDP countries, stay out of training.

The method fine-tunes models on the probability mass placed on the first answer token, which is constrained to valid option letters. Training minimizes KL divergence from the real human distribution. Across seven LLMs from three families, this specialization clearly outperforms zero-shot prompting and alternative setups on both familiar and held-out countries and questions. It also transfers to the Pew Global Attitudes Survey.

Accuracy nonetheless stays modest, particularly for new questions, and the models vary less between countries than real respondents do. The study therefore shows that training improves simulation while cautioning against relying on it yet.

Key points

  • Data: WVS wave 7 (66 countries, 80,000+ respondents), filtered to 65 countries with more than 1,000 respondents. The first 259 questions are split by topic into Q1 (1–163), Q2 (164–198, religious/ethical values) and Q3 (199–259, political interest and culture). Countries are split into C1 (46), C2 (8 African) and C3 (11 medium-GDP).
  • Three released datasets (two English, one Chinese), using the GlobalOpinionQA prompt template. Missing Chinese questions were translated with GLM-4. Code and data are at github.com/yongcaoplus/SimLLMCultureDist.
  • Evaluation uses 1−JSD, EMD and accuracy. On the unseen Pew survey, fine-tuned Llama3 gains 19.1% and 27.4% in accuracy on the C1′ and C3 country sets.
  • Ablations: KL loss works best, though Wasserstein, JS and cross-entropy losses also beat zero-shot. Shuffling option order lowers scores only slightly, so A-bias is not a major factor.
  • Chinese prompts perform marginally below English ones, and the gap is not significant. No models larger than 32B were fine-tuned.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CopenhagenCopenhagen · Denmark115 works · 15,292 citations🎓University of TübingenTübingen · Germany75 works · 41,081 citations🎓Bocconi UniversityMilan · Italy28 works · 3,077 citations🎓Wuhan University of Science and TechnologyWuhan · China2 works · 54 citations🎓Tübingen AI CenterTübingen · Germany3 works · 30 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Yong Cao, Haijiang Liu, Arnav Arora, Isabelle Augenstein, Paul Röttger and Daniel Hershcovich. 2025. Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.naacl-long.162

@inproceedings{cao2025specializing,
  title = {{Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations}},
  author = {Yong Cao, Haijiang Liu, Arnav Arora, Isabelle Augenstein, Paul Röttger and Daniel Hershcovich},
  booktitle = {Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.162},
  url = {https://doi.org/10.18653/v1/2025.naacl-long.162}
}

References

31 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gati V Aher, Rosa I Arriaga, and Adam Tauman Kalai. 2023. Using large language models to simulate multiple humans and replicate human subject studies. In International Conference on Machine Learning, pages 337–371. PMLR. find
  2. AI@Meta. 2024. Llama 3 model card. find
  3. Badr AlKhamissi, Muhammad ElNokrashy, Mai Alkhamissi, and Mona Diab. 2024. Investigating cultural alignment of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12404–12422. Association for Computational L…
  4. Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate. 2023. Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3):337–351.
  5. Arnav Arora, Lucie-aimée Kaffee, and Isabelle Augenstein. 2023. Probing pre-trained language models for cross-cultural differences in values. In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pages 114–130. Association for Computational Linguistics.
  6. Joris Baan, Wilker Aziz, Barbara Plank, and Raquel Fernandez. 2022. Stop measuring calibration when humans disagree. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1892–1915. Association for Computational Linguistics. find
  7. Joris Baan, Raquel Fernández, Barbara Plank, and Wilker Aziz. 2024. Interpreting predictive probabilities: Model confidence or human label variation? In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pages 268–277… find
  8. Christopher A Bail. 2024. Can generative ai improve social science? Proceedings of the National Academy of Sciences, 121(21):e2314021121. find
  9. Noam Benkler, Drisana Mosaphir, Scott Friedman, Andrew Smart, and Sonja Schmer-Galunder. 2023. Assessing LLMs for moral value pluralism. arXiv preprint arXiv:2312.10075. — [arXiv:2312.10075](https://arxiv.org/abs/2312.10075) find
  10. James Bisbee, Joshua D Clinton, Cassy Dorff, Brenton Kenkel, and Jennifer M Larson. 2023. Synthetic replacements for human survey data? the perils of large language models. Political Analysis, pages 1– 16. find
  11. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  12. Austin Kozlowski and James Evans. 2024. Simulating subjects: The promise and peril of ai stand-ins for social agents and interactions. find
  13. Louis Kwok, Michal Bravansky, and Lewis D Griffin. 2024. Evaluating cultural adaptability of a large language model via simulation of synthetic personas. arXiv preprint arXiv:2408.06929. — [arXiv:2408.06929](https://arxiv.org/abs/2408.06929) find
  14. Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. 2023. Assessing cross-cultural alignment between ChatGPT and human societies: An empirical study. In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pages 53–67. Association for Comput…
  15. Wangyue Li, Liangzhi Li, Tong Xiang, Xiao Liu, Wei Deng, and Noa Garcia. 2024. Can multiple-choice questions really be useful in detecting the abilities of LLMs? In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 202…
  16. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing GPT-4 with 90%* Chat-GPT quality. find
  17. Benjamin S Manning, Kehang Zhu, and John J Horton. 2024. Automated social science: Language models as scientist and subjects. Technical report, National Bureau of Economic Research. find
  18. Ricardo Dominguez-Olmedo, Moritz Hardt, and Celestine Mendler-Dünner. 2023. Questioning the survey responses of large language models. arXiv preprint arXiv:2306.07951. — [arXiv:2306.07951](https://arxiv.org/abs/2306.07951) find
  19. Esin Durmus, Karina Nyugen, Thomas I Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, et al. 2023. Towards measuring the representation of subjective global opinions in language models. arXiv preprint arXiv:2306.16388.
  20. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  21. C. Haerpfer, R. Inglehart, A. Moreno, C. Welzel, K. Kizilova, J. Diez-Medrano, M. Lagos, P. Norris, E. Ponarin, and B. Puranen. 2022. World values survey wave 7 (2017-2022) cross-national data-set. find
  22. Guande He, Peng Cui, Jianfei Chen, Wenbo Hu, and Jun Zhu. 2024. Investigating uncertainty calibration of aligned language models under the multiple-choice setting. find
  23. Luke Hewitt, Ashwini Ashokkumar, Isaias Ghezae, and Robb Willer. 2024. Predicting results of social science experiments using large language models. find
  24. John J Horton. 2023. Large language models as simulated economic agents: What can we learn from homo silicus? Technical report, National Bureau of Economic Research. find
  25. Peter S Park, Philipp Schoenegger, and Chongyang Zhu. 2024. Diminished diversity-of-thought in a standard large language model. Behavior Research Methods, pages 1–17. find
  26. Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. 1998. A metric for distributions with applications to image databases. In Sixth international conference on computer vision (IEEE Cat. No. 98CH36271), pages 59–66. IEEE. find
  27. Seungjong Sun, Eungu Lee, Dongyan Nan, Xiangying Zhao, Wonbyung Lee, Bernard J. Jansen, and Jang Hyun Kim. 2024. Random silicon sampling: Simulating human sub-population opinion using a large language model based on group-level demographic information. Preprint, arXiv:2402.18144. — [arXiv:2402.18144](https://arxiv.org/abs/2402.18144) find
  28. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  29. Dustin Wright, Arnav Arora, Nadav Borenstein, Srishti Yadav, Serge Belongie, and Isabelle Augenstein. 2024. Revealing fine-grained values and opinions in large language models. find
  30. Yahan Yang, Soham Dan, Dan Roth, and Insup Lee. 2024. On the calibration of multilingual question answering llms. Preprint, arXiv:2311.08669. find
  31. Wenlong Zhao, Debanjan Mondal, Niket Tandon, Danica Dillion, Kurt Gray, and Yuling Gu. 2024. Worldvaluesbench: A large-scale benchmark dataset for multi-cultural value awareness of language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resou… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cao-2025-specializing-large-language-models-simulate/