Elamur Research
☰
home / papers / 2022 / Out of One, Many: Using Language Models to Simulate Human Samples

Out of One, Many: Using Language Models to Simulate Human Samples

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2209.06899

Summary

Social scientists usually treat bias in large language models as one flaw to remove. Argyle and colleagues argue that GPT-3 holds many group-specific biases, and that these can be used on purpose. If you condition the model on a detailed demographic profile, it should answer the way people with that background tend to answer. The authors call this property algorithmic fidelity and propose four tests for it: a social-science Turing test, backward continuity, forward continuity and pattern correspondence. To check it, they wrote first-person backstories from real participants in the 2012, 2016 and 2020 American National Election Studies (ANES) and in the "Pigeonholing Partisans" dataset. Each backstory produced one simulated respondent, and together these form a "silicon sample." Three studies compared these samples with the human data: free-text words describing the opposing party, vote choice, and associations among answers to closed-ended survey questions. In each study, the simulated responses reproduced the human patterns closely and in detail. The authors conclude that a well-conditioned model could be a cheap way to test hypotheses in U.S. politics before running costly surveys with real people.

Key points

  • Main contributions: a definition of algorithmic fidelity with four criteria, "silicon sampling" to correct skewed marginal statistics in the training data, and conditioning on first-person demographic backstories.
  • The model is GPT-3 (175 billion parameters), accessed through the paid OpenAI API. The human baselines are ANES 2012/2016/2020 and Rothschild et al.'s "Pigeonholing Partisans" data.
  • Study 3 compared associations between variables (Cramer's V) in the human and GPT-3 data. At temperature 0.7 the mean error was about −0.026 (N = 1,782). At 0.001 the model labelled every respondent as White, and the errors were larger and overstated the associations.
  • The final runs cost little: about $29 for Study 1 (one query per participant), $75 for Study 2 (15,626 queries) and about $1,428 for Study 3 (12 experiments of roughly $119 each).
  • The authors caution that a model cannot reproduce a specific individual. Fidelity shows up only in the distribution of answers for a demographic group, and it must be checked separately for each domain.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

199 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 192 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1017/pan.2023.2 · arXiv 2209.06899

@misc{argyle2022out,
  title = {{Out of One, Many: Using Language Models to Simulate Human Samples}},
  author = {Argyle, Lisa P. and Busby, Ethan C. and Fulda, Nancy and Gubler, Joshua and Rytting, Christopher and Wingate, David},
  year = {2022},
  doi = {10.1017/pan.2023.2},
  eprint = {2209.06899},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1017/pan.2023.2}
}

References

41 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pedro Rodriguez and Arthur Spirling. Word embeddings: What works, what doesn’t, and how to tell the difference for applied research. 2021. find
  2. Kenneth Benoit, Kevin Munger, and Arthur Spirling. Measuring and explaining political sophistication through textual complexity. American Journal of Political Science, 63(2):491–508, 2019. find
  3. Pablo Barberá, Amber E Boydstun, Suzanna Linn, Ryan McMahon, and Jonathan Nagler. Automated text classification of news articles: A practical guide. Political Analysis, 29(1):19–42, 2021. find
  4. Ludovic Rheault and Christopher Cochrane. Word embeddings for the analysis of ideological placement in parliamentary corpora. Political Analysis, 28(1):112–133, 2020. find
  5. Justin Grimmer, Margaret E Roberts, and Brandon M Stewart. Machine learning for social science: An agnostic approach. Annual Review of Political Science, 24:395–419, 2021. find
  6. Kevin T Greene, Baekkwan Park, and Michael Colaresi. Machine learning human rights and wrongs: How the successes and failures of supervised learning algorithms can inform the debate about information effects. Political Analysis, 27(2):223–230, 2019. find
  7. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9, 2019. find
  8. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020. URL http://jmlr. org/papers/v21/2… find
  9. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv:2005.14165, 2020. find
  10. Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977, 2020. find our card
  11. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9, 2019. find
  12. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860, 2019. find
  13. Trishan Panch, Heather Mattie, and Rifat Atun. Artificial intelligence and algorithmic bias: Implications for health systems. Journal of Global Health, 9(2):010318, December 2019. ISSN 2047-2978, 2047-2986. doi: 10.7189/jogh.09.020318. — [doi:10.7189/jogh.09.020318](https://doi.org/10.7189/jogh.09.020318) find
  14. Sandra G. Mayson. Bias in, Bias out. Yale Law Journal, 128:2218, 2018. find
  15. Solon Barocas and Andrew D. Selbst. Big Data’s Disparate Impact. California Law Review, 104:671, 2016. find
  16. See https://electionstudies.org/about-us/. ↗
  17. Jacob E. Rothschild, Adam J. Howat, Richard M. Shafranek, and Ethan C. Busby. Pigeonholing partisans: Stereotypes of party supporters and partisan polarization. Political Behavior, 41(2):423– 443, 2019. find
  18. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623, 2021. find
  19. Gary Marcus. The next decade in ai: four steps towards robust artificial intelligence. arXiv preprint arXiv:2002.06177, 2020. find
  20. Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences, 115(16):E3635– E3644, 2018. find
  21. Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186, 2017. find
  22. Bernard Berelson, Paul F. Lazarsfeld, and William N. McPhee. Voting: a study of opinion formation in a presidential campaign. University of Chicago Press, 1954. find
  23. Angus Campbell, Philip E. Converse, Warren E. Miller, and Donald E. Stokes. The American Voter. University of Chicago Press, 1960. find
  24. Vincent L. Hutchings and Nicholas A. Valentino. The centrality of race in american politics. Annual Review of Political Science, 7(1):383–408, 2004. find
  25. Nancy Burns and Katherine Gallagher. Public opinion on gender issues: The politics of equity and roles. Annual Review of Political Science, 13(1):425–443, 2010. find
  26. James N. Druckman and Arthur Lupia. Preference change in competitive political environments. Annual Review of Political Science, 19(1):13–31, 2016. find
  27. Katherine Cramer. Understanding the role of racism in contemporary us public opinion. Annual Review of Political Science, 23(1):153–169, 2020. find
  28. Edward H. Simpson. The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society, Ser. B, 13:238–241, 1951. find
  29. Shanto Iyengar, Gaurav Sood, and Yphtach Lelkes. Affect, not ideology a social identity perspective on polarization. Public Opinion Quarterly, 76(3):405–431, 2012. find
  30. Lilliana Mason. Uncivil Agreement. University of Chicago Press, 2018. find
  31. Alexander Coppock and Oliver A. McClellan. Validating the demographic, political, psychological, and experimental results obtained from a new source of online survey respondents. Research & Politics, 6(1):1–14, 2019. find
  32. Janet M. Box-Steffensmeier, Suzanna De Boef, and Tse min Lin. The dynamics of the partisan gender gap. American Political Science Review, 98(3):515–528, 2004. find
  33. Katherine Tate. From Protest to Politics: The New Black Voters in American Elections. Harvard University Press, 1994. find
  34. Katherine J. Cramer. The politics of resentment: Rural consciousness in Wisconsin and the rise of Scott Walker. University of Chicago Press, 2016. find
  35. Ashley Jardina. White Identity Politics. Cambridge University Press, 2019. find
  36. Bruce E. Keith, David B. Magleby, Candice J. Nelson, Elizabeth Orr, and Mark C. Westyle. The myth of the independent voter. University of California Press, 1992. find
  37. David B. Magleby, Candice J. Nelson, and Mark C. Westlye. The myth of the independent voter revisited. In Paul M. Sniderman and Benjamin Highton, editors, Facing the challenge of democracy: Explorations in the analysis of public opinion and political participation, pages 238–266. Princeton Universit… find
  38. Samara Klar and Yanna Krupnikov. Independent Politics: How American Disdain for Parties Leads to Political Inaction. Cambridge University Press, 2016. find
  39. Harald Cramér. Mathematical Methods of Statistics. Princeton University Press, 1946. find
  40. Ronald S Ross. Guide for conducting risk assessments (nist sp-800-30rev1). The National Institute of Standards and Technology (NIST), Gaithersburg, 2012. find
  41. Matthew J. Salganik. Bit by Bit: Social Research in the Digital Age. Princeton University Press, Princeton, NJ, open review edition edition, 2017. find