Elamur Research⌕
☰
home / papers / 2026 / Simulating Social Attitudes with LLMs: Accuracy, Demographic Effects, and Refusal Behavior in the Sensitive Domain of Suicide Prevention

Simulating Social Attitudes with LLMs: Accuracy, Demographic Effects, and Refusal Behavior in the Sensitive Domain of Suicide Prevention

Paper · 2026 · Cristina J. Perez, Michael P. Vasquez Jr, Philippe Giabbanelli et al.

Summary

This study tests whether "silicon sampling," where LLMs stand in for survey respondents, holds up for a sensitive policy area: public support for suicide prevention measures such as firearm restrictions, crisis-line funding and mental-health spending. The authors compiled 32 items from seven nationally representative U.S. polls conducted in 2023–2025 and treated their published results as ground truth. They generated simulated answers while varying three factors:

  • Demographic persona: race/ethnicity, gender, age, education, income and party.
  • Prompt framing: direct elicitation, respondent embodiment or specialist embodiment.
  • Model: four LLMs from different providers.

Simulated and real response distributions were compared at the level of single demographic subgroups. On average, the simulated distributions were far from the real ones. Adding demographic details shifted outputs considerably compared with persona-free prompts. Accuracy depended more on the model than on the framing, while refusal rates differed widely across both models and prompt designs. The authors conclude that it is unclear what population LLMs actually reflect on such questions. This casts doubt on using LLM-simulated respondents to estimate policy feasibility on sensitive topics.

Key points

  • Scale: 811,560 prompts in total. Mean absolute error between simulated and human response distributions was about 23 percentage points.
  • Models: the abstract lists GPT-5 Nano, DeepSeek V3.2, Meta Llama 3.1 8B and Mistral Small 24B. The introduction names Qwen3-32B instead of Mistral, so the fourth model should be checked against the full text.
  • Ground truth: sources include AFSP's poll on attitudes toward mental health and suicide prevention, Duke's 988 Awareness Survey, a YouGov Gun Ownership Survey and a help-seeking preferences survey.
  • Refusals: refusal behavior (RQ3) varied sharply by model and framing, which the authors link to guardrails around self-harm topics.
  • Limitation: the surveys report only single-attribute marginals. The LLM aggregation weights all persona combinations equally, whereas the survey marginals reflect the sample's actual composition, and this cannot be corrected without joint distributions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓American UniversityWashington · United States2 works · 6,540 citations

Also on the byline: Old Dominion University.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSafety Research · 626 worksGeneral Social Sciences · 254 works

Cite

Cite as Cristina J. Perez, Michael P. Vasquez Jr, Philippe Giabbanelli and Patrick Y. Wu. 2026. Simulating Social Attitudes with LLMs: Accuracy, Demographic Effects, and Refusal Behavior in the Sensitive Domain of Suicide Prevention. Proceedings of the Seventh Workshop on Natural Language Processing and Computational Social Science.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.nlpcss-1.12

@inproceedings{perez2026simulating,
  title = {{Simulating Social Attitudes with LLMs: Accuracy, Demographic Effects, and Refusal Behavior in the Sensitive Domain of Suicide Prevention}},
  author = {Cristina J. Perez, Michael P. Vasquez Jr, Philippe Giabbanelli and Patrick Y. Wu},
  booktitle = {Proceedings of the Seventh Workshop on Natural Language Processing and Computational Social Science},
  year = {2026},
  doi = {10.18653/v1/2026.nlpcss-1.12},
  url = {https://doi.org/10.18653/v1/2026.nlpcss-1.12}
}

References

38 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Michael D. Anestis, Jennifer Paruk, Jayna Moceri-Brooks, Shelby L. Bandel, Allison E. Bond, and Daniel C. Semenza. 2025. Alignment between self-and perceived peer support for specific firearm policies: Results from a representative survey of adults in nine us states. Preventive Medicine Reports, 54:… find
  2. Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate. 2023. Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3):337–351.
  3. Catherine W. Barber and Matthew J. Miller. 2014. Reducing a suicidal person’s access to lethal means of suicide: a research agenda. American Journal of Preventive Medicine, 47(3):S264–S272. find
  4. Colleen L. Barry and Emma E. McGinty. 2014. Stigma and public support for parity and government spending on mental health: a 2013 national opinion survey. Psychiatric Services, 65(10):1265–1268. find
  5. James Bisbee, Joshua D. Clinton, Cassy Dorff, Brenton Kenkel, and Jennifer M. Larson. 2024. Synthetic replacements for human survey data? the perils of large language models. Political Analysis, 32(4):401–416. find
  6. California Department of Public Health. 2022. California Suicide Prevention Plan, 2020–2025. Suicide Prevention Resource Center / California Department of Public Health. find
  7. Centers for Disease Control and Prevention. 2022. Suicide prevention resource for action: A compilation of the best available evidence. Technical report, National Center for Injury Prevention and Control, Centers for Disease Control and Prevention. find
  8. Centers for Disease Control and Prevention. 2025. Suicide data and statistics. https://www.cdc.gov/ suicide/facts/data.html. ↗
  9. Yujie Chi and Dazhou Lei. 2026. The price of digital compassion: Exposing and managing latent risks in ai survey augmentation. Available at SSRN 6026277. find
  10. Andrew Conner, Deborah Azrael, and Matthew Miller. 2022. Perceptions of firearm accessibility and suicide among us adults living in households with firearms. JAMA network open, 5(10):e2239278. find
  11. Cassandra K. Crifasi, Elizabeth M. Stone, Emma E. McGinty, and Colleen L. Barry. 2021. Differences in public support for gun policies between women and men. American Journal of Preventive Medicine, 60(1):e9–e14. find
  12. Graham Dixon, Kelly Garrett, Mark Susmann, and Brad J. Bushman. 2020. Public opinion perceptions, private support, and public actions of us adults regarding gun safety policy. JAMA Network Open, 3(12):e2029571. find
  13. Lauren N. Forrest, Ariel L. Beccia, Cara Exten, Sarah Gehman, and Emily B. Ansell. 2023. Intersectional prevalence of suicide ideation, plan, and attempt based on gender, sexual orientation, race and ethnicity, and rurality. JAMA Psychiatry, 80(10):1037– 1046. find
  14. Tyler J. Gandee, Sean C. Glaze, and Philippe J. Giabbanelli. 2024. A visual analytics environment for navigating large conceptual models by leveraging generative artificial intelligence. Mathematics, 12(13):1946. find
  15. Philippe J. Giabbanelli. 2026. A guide to large language models in modeling and simulation: From core techniques to critical challenges. arXiv preprint arXiv:2602.05883. — [arXiv:2602.05883](https://arxiv.org/abs/2602.05883) find
  16. Shashank Gupta, Vaishnavi Shrivastava, Ameet Deshpande, Ashwin Kalyan, Peter Clark, Ashish Sabharwal, and Tushar Khot. 2024. Bias runs deep: Implicit reasoning biases in persona-assigned LLMs. In International Conference on Learning Representations, volume 2024, pages 21849–21874.
  17. Nicholas Hemauer and Seth Warner. 2025. Analyzing public support for school-based mental health services. Journal of Health Politics, Policy and Law, 50(5):771–799. find
  18. John J. Horton, Apostolos Filippas, and Benjamin S. Manning. 2023. Large language models as simulated economic agents: What can we learn from homo silicus? Working Paper 31122, National Bureau of Economic Research. find
  19. Julie Jargon. 2026. Gemini said they could only be together if he killed himself. soon, he was dead. The Wall Street Journal. find
  20. Shapeng Jiang, Lijia Wei, and Chen Zhang. 2025. Donald Trumps in the virtual polls: Simulating and predicting public opinions in surveys using large language models. Preprint, arXiv:2411.01582. — [arXiv:2411.01582](https://arxiv.org/abs/2411.01582) find
  21. F. Reed Johnson, Juan Marcos Gonzalez, Jui-Chen Yang, Semra Ozdemir, and Steven Kymes. 2021. Who would pay higher taxes for better mental health? results of a large-sample national choice experiment. The Milbank Quarterly, 99(3):771–793. find
  22. Robert C. Kenter, Martin K. Mayer, and John C. Morris. 2022. Explaining state differences in firearm legislation: A south/non-south analysis. Social Science Quarterly, 103(6):1371–1380. find
  23. Sanguk Lee, Tai-Quan Peng, Matthew H. Goldberg, Seth A. Rosenthal, John E. Kotcher, Edward W. Maibach, and Anthony Leiserowitz. 2024. Can large language models estimate public opinion about global warming? an empirical assessment of algorithmic fidelity and bias. PLOS Climate, 3(8):1–14. find
  24. Andy Liu, Mona Diab, and Daniel Fried. 2024. Evaluating large language model biases in persona-steered generation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 9832–9850. find
  25. Christin L. Munsch, Liberty Barnes, and Zachary D. Kline. 2020. Who’s to blame? partisanship, responsibility, and support for mental health treatment. Socius, 6:2378023120921652. find
  26. Kiwoong Park and Ning Hsieh. 2023. A national study on religiosity and suicide risk by sexual orientation. American Journal of Preventive Medicine, 64(2):235– 243. find
  27. Cora Peterson, Tadesse Haileyesus, and Deborah M. Stone. 2024. Economic cost of US suicide and non-fatal self-harm. American Journal of Preventive Medicine, 67(1):129–133. find
  28. Jonathan Purtle, Amanda I. Mauri, Michael A. Lindsey, and Katherine M. Keyes. 2025. Evidence for public policies to prevent suicide death in the united states. Annual Review of Public Health, 46(1):349–367. find
  29. Jonathan Purtle, Amanda I. Mauri, Anna-Michelle Marie McSorley, Abigail Lin Adera, Matthew L. Goldman, and Michael A. Lindsey. 2024. Demographic variation in preferred sources for suicide prevention and mental health crisis services among us adults. Preventive Medicine Reports, 47:102914. find
  30. Yao Qu and Jue Wang. 2024. Performance and biases of large language models in public opinion simulation. Humanities and Social Sciences Communications, 11(1):1095. find
  31. Hillary Samples, Naomi Cruz, Allison Corr, and Farzana Akkas. 2025. National trends and disparities in suicidal ideation, attempts, and health care utilization among us adults. Psychiatric Services, 76(2):110–119. find
  32. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org.
  33. Morgan C. Shields, Nev Jones, Shyamal Sharma, and Susan H. Busch. 2025. Public attitudes toward mental health treatment policy. JAMA Network Open, 8(9):e2532344. find
  34. Gabriel Simmons and Christopher Hare. 2023. Large language models as subpopulation representative models: A review. Preprint, arXiv:2310.17888. — [arXiv:2310.17888](https://arxiv.org/abs/2310.17888) find
  35. Elizabeth M. Stone and Emma E. McGinty. 2018. Public willingness to pay to improve services for individuals with serious mental illness. Psychiatric Services, 69(8):938–941. find
  36. Bazen Gashaw Teferra, Nabil Johny, Sandra Huang, Alice Rueda, Mohammad Amin Kamaleddin, Katharine Dunlop, Yanbo Zhang, Manish Jha, Divya Sharma, and Venkat Bhat. 2026. Assessing the impact of safety guardrails on large language models using irritability metrics. npj Digital Medicine. find
  37. Lindia Tjuatja, Valerie Chen, Tongshuang Wu, Ameet Talwalkwar, and Graham Neubig. 2024. Do llms exhibit human-like response biases? a case study in survey design. Transactions of the Association for Computational Linguistics, 12:1011–1026.
  38. Stephen Zhong, Nathalie Japkowicz, and Philippe Giabbanelli. 2025b. Do we still need people? comparing human and llm personas in political modeling and simulation. In 2025 ACM/IEEE 28th International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C), pages 512– 521. I… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/perez-2026-simulating-social-attitudes-llms-accuracy/