Elamur Research⌕
☰
home / papers / 2026 / Too Open for Opinion? Embracing Open-Endedness in Large Language Models for Social Simulation

Too Open for Opinion? Embracing Open-Endedness in Large Language Models for Social Simulation

Paper · 2026 · Bolei Ma, Yong Cao, Indira Sen et al.

Summary

Research that uses LLMs to stand in for survey respondents or social agents usually restricts the models to multiple-choice or short-answer formats, because such outputs are easy to score and aggregate. This position paper by Ma et al. (EACL 2026) argues that this restriction wastes the generative nature of LLMs. It can flatten nuanced opinions, push answers into categories the researcher chose in advance, and hide how the model reasons. The authors draw on decades of survey methodology on open-ended questions and connect it to recent NLP work. They map the known advantages of free-text answers onto LLM-based simulation, including better measurement, discovery of unexpected and minority views, less directive bias, more individual expression, and support for pretesting. They also discuss methods for coding and analysing free-form outputs, along with the challenges these raise. The paper presents no experiments or benchmarks. Its contribution is a call for evaluation frameworks that use open-ended diversity instead of suppressing it, which would strengthen the link between NLP and the social sciences.

Key points

  • The authors re-examined the 53 simulation studies catalogued by Anthis et al. (2025). Only 11 (21%) contain any open-text element, and just 4 (8%) rely mainly on free-form outputs for evaluation.
  • The paper defines open-endedness by how the task input is designed, not by how text is generated. Prompts that offer no fixed answer set count as open-ended, unlike outputs that are mapped back to labels through string matching or token probabilities.
  • Benefits of open-ended questions are grouped into eight dimensions taken from survey research, such as measurement, design, unanticipated views, directive bias, expressiveness/individuality, pretesting, and methodological utility.
  • The authors list several risks to address:
    • known LLM biases, such as left-leaning and WEIRD framings, found mostly in closed-ended setups;
    • LLM survey answers that are more positive than human ones;
    • fluent but shallow outputs being mistaken for authentic responses;
    • analytic flexibility that could lead to "silicon-only" findings that cannot be replicated.
  • The paper is purely conceptual and relies on a non-exhaustive literature synthesis. It notes that "coding" in social science means annotating free responses into analytic categories, not programming.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Maryland, College ParkCollege Park · United States141 works · 29,174 citations🎓University of CopenhagenCopenhagen · Denmark116 works · 15,292 citations🎓Ludwig-Maximilians-Universität MünchenMunich · Germany113 works · 56,817 citations🎓University of TübingenTübingen · Germany78 works · 41,280 citations🎓University of MannheimMannheim · Germany29 works · 8,347 citations🎓Munich Center for Machine LearningMunich · Germany62 works · 1,167 citations🎓Tübingen AI CenterTübingen · Germany3 works · 30 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Bolei Ma, Yong Cao, Indira Sen, Anna-Carolina Haensch, Frauke Kreuter, Barbara Plank and Daniel Hershcovich. 2026. Too Open for Opinion? Embracing Open-Endedness in Large Language Models for Social Simulation. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.89

@inproceedings{ma2026too,
  title = {{Too Open for Opinion? Embracing Open-Endedness in Large Language Models for Social Simulation}},
  author = {Bolei Ma, Yong Cao, Indira Sen, Anna-Carolina Haensch, Frauke Kreuter, Barbara Plank and Daniel Hershcovich},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.89},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.89}
}

References

132 references, 33 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Divya Mani Adhikari, Alexander Hartland, Ingmar Weber, and Vikram Kamath Cannanure. 2025. Exploring llms for automated generation and adaptation of questionnaires. In Proceedings of the 7th ACM Conference on Conversational User Interfaces, CUI ’25, New York, NY, USA. Association for Computing Machin… find
  2. Dalal Albudaiwi. 2017. Survey: Open-ended questions. In The SAGE Encyclopedia of Communication Research Methods, volume 4, pages 1716–1717. SAGE Publications, Inc, Thousand Oaks, California. find
  3. Jacy Reese Anthis, Ryan Liu, Sean M Richardson, Austin C. Kozlowski, Bernard Koch, Erik Brynjolfsson, James Evans, and Michael S. Bernstein. 2025. Position: LLM social simulations are a promising research method. In Forty-second International Conference on Machine Learning Position Paper Track. find
  4. Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate. 2023. Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3):337–351.
  5. Joshua Ashkinaze, Emily Fry, Narendra Edara, Eric Gilbert, and Ceren Budak. 2025. Plurals: A system for guiding llms via simulated social ensembles. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI ’25, New York, NY, USA. Association for Computing Machinery. find
  6. Nishant Balepur, Rachel Rudinger, and Jordan Lee Boyd-Graber. 2025. Which of these best describes multiple choice evaluation with LLMs? a) forced B) flawed C) fixable D) all of the above. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper…
  7. Franck Bardol. 2025. Chatgpt reads your tone and responds accordingly – until it does not – emotional framing induces bias in llm outputs. Preprint, arXiv:2507.21083. — [arXiv:2507.21083](https://arxiv.org/abs/2507.21083) find
  8. Jacob Beck, Stephanie Eckman, Christoph Kern, and Frauke Kreuter. 2025. Bias in the loop: How humans evaluate ai-generated suggestions. Preprint, arXiv:2509.08514. — [arXiv:2509.08514](https://arxiv.org/abs/2509.08514) find
  9. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 610–623, New York, NY, USA. Associat… find
  10. Herbert Bless and Norbert Schwarz. 2010. Chapter 6 - mental construal and the emergence of assimilation and contrast effects: The inclusion/exclusion model. In Advances in Experimental Social Psychology, volume 42 of Advances in Experimental Social Psychology, pages 319–373. Academic Press. find
  11. Norman M. Bradburn, Seymour Sudman, and Brian Wansink. 2004. Asking Questions: The Definitive Guide to Questionnaire Design – For Market Research, Political Polls, and Social and Health Questionnaires, 2nd, revised edition. Jossey-Bass, San Francisco, CA. find
  12. James Brand, Ayelet Israeli, and Donald Ngwe. 2023. Using llms for market research. Working Paper 23-062, Harvard Business School Marketing Unit. Last revised: July 8, 2023. find
  13. Yong Cao, Haijiang Liu, Arnav Arora, Isabelle Augenstein, Paul Röttger, and Daniel Hershcovich. 2025. Specializing large language models to simulate survey response distributions for global populations. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association f…
  14. Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. 2023. Assessing cross-cultural alignment between ChatGPT and human societies: An empirical study. In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pages 53–67, Dubrovnik, Croatia. As…
  15. Anthony Chemero. 2023. Llms differ from human cognition because they are not embodied. Nature Human Behaviour, 7(11):1828–1829. find
  16. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. find
  17. Saoirse Connor Desai and Stian Reimers. 2019. Comparing the use of open and closed questions for web-based measures of the continued-influence effect. Behavior Research Methods, 51(3):1426–1440. find
  18. Don A. Dillman, Jolene D. Smyth, and Leah Melani Christian. 2014. Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method, 4th edition. Wiley, Hoboken, NJ. find
  19. Zackary Okun Dunivin. 2025. Scaling hermeneutics: a guide to qualitative coding with llms for reflexive content analysis. EPJ Data Science, 14(1):28. find
  20. Esin Durmus, Karina Nguyen, Thomas Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. 2024. Towards measuring the…
  21. Stephanie Eckman, Barbara Plank, and Frauke Kreuter. 2024. Position: Insights from survey methodology can improve training data. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 12268–12283. PMLR. find
  22. Aparna Elangovan, Ling Liu, Lei Xu, Sravan Babu Bodapati, and Dan Roth. 2024. ConSiDERS-the-human evaluation framework: Rethinking human evaluation for generative large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)… find
  23. Andrew Estornell and Yang Liu. 2024. Multi-llm debate: Framework, principals, and interventions. In Advances in Neural Information Processing Systems, volume 37, pages 28938–28964. Curran Associates, Inc. find
  24. Frederick G. Conrad, Mick P. Couper, and Joseph W. Sakshaug. 2016. Classifying open-ended reports: Factors affecting the reliability of occupation codes. Journal of Official Statistics, 32(1):75–92. find
  25. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889–898, Melbourne, Australia. Association for Computational Linguistics.
  26. Ziyan Cui, Ning Li, and Huaikang Zhou. 2025. A large-scale replication of scenario-based experiments in psychology and management using large language models. Nature Computational Science, 5(8):627– 634. find
  27. Beatrice Ferrario and Stefanie Stantcheva. 2022. Eliciting people’s first-order concerns: Text analysis of open-ended survey questions. AEA Papers and Proceedings, 112:163–169. find
  28. Jamie Cummins. 2025. The threat of analytic flexibility in using large language models to simulate human data: A call to attention. Preprint, arXiv:2509.13397. — [arXiv:2509.13397](https://arxiv.org/abs/2509.13397) find
  29. Angus Deaton and Arthur A. Stone. 2016. Understanding context effects for a measure of life evaluation: how responses matter. Oxford Economic Papers, 68(4):861–870. find
  30. Stefan Feuerriegel, Abdurahman Maarouf, Dominik Bär, Dominique Geissler, Jonas Schweisthal, Nicolas Pröllochs, Claire E. Robertson, Steve Rathje, Jochen Hartmann, Saif M. Mohammad, Oded Netzer, Alexandra A. Siegel, Barbara Plank, and Jay J. Van Bavel. 2025. Using natural language processing to analy… find
  31. James Flamino, Mohammed Shahid Modi, Boleslaw K. Szymanski, Brendan Cross, and Colton Mikolajczyk. 2025. Testing the limits of large language models in debating humans. Scientific Reports, 15(1):13852. find
  32. Zhoushanyue He and Matthias Schonlau. 2021a. Coding text answers to open-ended questions: Human coders and statistical learning algorithms make similar mistakes. methods, data, analyses, 15(1):17. find
  33. Mirta Galesic and Michael Bosnjak. 2009. Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opinion Quarterly, 73(2):349–360. find
  34. Zhoushanyue He and Matthias Schonlau. 2021b. A model-assisted approach for finding coding errors in manual coding of open-ended questions. Journal of Survey Statistics and Methodology, 10(2):365–376. find
  35. John G. Geer. 1988. What do open-ended questions measure? Public Opinion Quarterly, 52(3):365–367. find
  36. Peter Henderson, Xuechen Li, Dan Jurafsky, Tatsunori Hashimoto, Mark A. Lemley, and Percy Liang. 2023. Foundation models and fair use. Journal of Machine Learning Research, 24(400):1–79. find
  37. Julian Iñaki Goñi. 2025. Citizen participation and technology: lessons from the fields of deliberative democracy and science and technology studies. Humanities and Social Sciences Communications, 12(1):287. find
  38. Josh A. Goldstein, Girish Sastry, Micah Musser, Renee DiResta, Matthew Gentzel, and Katerina Sedova. 2023. Generative language models and automated influence operations: Emerging threats and potential mitigations. Preprint, arXiv:2301.04246. — [arXiv:2301.04246](https://arxiv.org/abs/2301.04246) find
  39. Robert M. Groves, Floyd J. Fowler, Jr., Mick P. Couper, James M. Lepkowski, Eleanor Singer, and Roger Tourangeau. 2009. Survey methodology, 2nd edition. John Wiley & Sons, Hoboken, NJ. find
  40. Jairo F Gudiño, Umberto Grandi, and César Hidalgo. 2024. Large language models (llms) as agents for augmented democracy. Philosophical Transactions A, 382(2285):20240100. find
  41. Hyukjun Gweon and Matthias Schonlau. 2023. Automated classification for open-ended questions with bert. Journal of Survey Statistics and Methodology, 12(2):493–504. find
  42. Anna-Carolina Haensch, Bernd Weiß, Patricia Steins, Priscilla Chyrva, and Katja Bitz. 2022. The semi-automatic classification of an open-ended question on panel survey motivation and its application in attrition analysis. Frontiers in Big Data, 5:880554. find
  43. C. Haerpfer, R. Inglehart, A. Moreno, C. Welzel, K. Kizilova, J. Diez-Medrano, M. Lagos, P. Norris, E. Ponarin, and B. Puranen, editors. 2022. World Values Survey: Round Seven - Country-Pooled Datafile Version 5.0. JD Systems Institute & WVSA Secretariat, Madrid, Spain & Vienna, Austria. find
  44. Tyler Hamby and Wyn Taylor. 2016. Survey satisficing inflates reliability and validity measures: An experimental comparison of college and amazon mechanical turk samples. Educational and Psychological Measurement, 76(6):912–932. find
  45. Zhoushanyue He and Matthias Schonlau. 2020. Automatic coding of text answers to open-ended questions: Should you double code the training data? Social Science Computer Review, 38(6):754–765. find
  46. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations.
  47. John J. Horton. 2023. Large language models as simulated economic agents: What can we learn from homo silicus? Preprint, arXiv:2301.07543. — [arXiv:2301.07543](https://arxiv.org/abs/2301.07543) find
  48. Dirk Hovy and Diyi Yang. 2021. The importance of modeling social factors of language: Theory and practice. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 588–602, Online. Association for Computa…
  49. Yukun Huang, Yixin Liu, Raghuveer Thirukovalluru, Arman Cohan, and Bhuwan Dhingra. 2024. Calibrating long-form generations from large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13441–13460, Miami, Florida, USA. Association for Computational Lingu… find
  50. Shivani Kapania, William Agnew, Motahhare Eslami, Hoda Heidari, and Sarah E Fox. 2025. Simulacrum of stories: Examining large language models as qualitative research participants. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI ’25, New York, NY, USA. Association… find
  51. Antonia Karamolegkou, Angana Borah, Eunjung Cho, Sagnik Ray Choudhury, Martina Galletti, Rajarshi Ghosh, Pranav Gupta, Oana Ignat, Priyanka Kargupta, Neema Kotonya, Hemank Lamba, Sun-Joo Lee, Arushi Mangla, Ishani Mondal, Deniz Nazarova, Poli Nemkova, Dina Pisarevskaya, Naquee Rizwan, Nazanin Sabri,… — [arXiv:2505.22327](https://arxiv.org/abs/2505.22327) find
  52. Rabimba Karanjai, Boris Shor, Amanda Austin, Ryan Kennedy, Yang Lu, Lei Xu, and Weidong Shi. 2025. Synthesizing public opinions with llms: Role creation, impacts, and the future to edemorcacy. Preprint, arXiv:2504.00241. — [arXiv:2504.00241](https://arxiv.org/abs/2504.00241) find
  53. Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. 2024. A survey of reinforcement learning from human feedback. Preprint, arXiv:2312.14925.
  54. Christoph Kern, Stephanie Eckman, Jacob Beck, Rob Chew, Bolei Ma, and Frauke Kreuter. 2023. Annotation sensitivity: Training data collection methods affect model performance. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14874–14886, Singapore. Association for Compu… find
  55. Eunsu Kim, Juyoung Suk, Seungone Kim, Niklas Muennighoff, Dongkwan Kim, and Alice Oh. 2025. LLM-as-an-interviewer: Beyond static testing through dynamic LLM evaluation. In Findings of the Association for Computational Linguistics: ACL 2025, pages 26456–26493, Vienna, Austria. Association for Computa… find
  56. Sunwoong Kim, Jongho Jeong, Jin Soo Han, and Donghyuk Shin. 2024. Llm-mirror: A generated-persona approach for survey pre-testing. Preprint, arXiv:2412.03162. — [arXiv:2412.03162](https://arxiv.org/abs/2412.03162) find
  57. Austin C. Kozlowski and James Evans. 2025. Simulating subjects: The promise and peril of artificial intelligence stand-ins for social agents and interactions. Sociological Methods & Research, 54(3):1017–1073. find
  58. Fabienne Kraemer, Henning Silber, Bella Struminskaya, Matthias Sand, Michael Bosnjak, Joanna Koßmann, and Bernd Weiß. 2023. Satisficing response behavior across time: Assessing negative panel conditioning using an experimental design with six repetitions. Survey Research Methods, 17(3). find
  59. Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. Hurdles to progress in long-form question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4940–4957, Online. Association for Computat…
  60. Jon A. Krosnick. 1999. Survey research. Annual Review of Psychology, 50:537–567. find
  61. Jon A. Krosnick and Stanley Presser. 2010. Question and questionnaire design. In Peter V. Marsden and James D. Wright, editors, Handbook of Survey Research, 2 edition, pages 264–313. Emerald, Bingley, UK. find
  62. Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. 2018. Scalable agent alignment via reward modeling: a research direction. Preprint, arXiv:1811.07871.
  63. Timo Lenzner, Cornelia Neuert, and Wanda Otto. 2015. Kognitives Pretesting. GESIS – Leibniz-Institut für Sozialwissenschaften, Mannheim. find
  64. Chance Jiajie Li, Jiayi Wu, Zhenze Mo, Ao Qu, Yuhan Tang, Kaiya Ivy Zhao, Yulu Gan, Jie Fan, Jiangbo Yu, Jinhua Zhao, Paul Liang, Luis Alonso, and Kent Larson. 2025. Position: Simulating society requires simulating thought. Preprint, arXiv:2506.06958. — [arXiv:2506.06958](https://arxiv.org/abs/2506.06958) find
  65. Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. 2024. Encouraging divergent thinking in large language models through multi-agent debate. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17…
  66. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  67. Haijiang Liu, Qiyuan Li, Chao Gao, Yong Cao, Xiangyu Xu, Xun Wu, Daniel Hershcovich, and Jinguang Gu. 2025. Beyond demographics: Enhancing cultural value survey simulation with multi-stage personality-driven cognitive reasoning. In Proceedings of the 2025 Conference on Empirical Methods in Natural L… find
  68. Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, and Andrew M. Dai. 2024. Best practices and lessons learned on synthetic data. In First Conference on Language Modeling.
  69. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Lin…
  70. Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. 2024. On LLMs-driven synthetic data generation, curation, and evaluation: A survey. In Findings of the Association for Computational Linguistics ACL 2024, pages 11065–11082, Bangkok, Thailand and virtual meeting. Ass…
  71. Chenyang Lyu, Minghao Wu, and Alham Aji. 2024. Beyond probabilities: Unveiling the misalignment in evaluating large language models. In Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowLLM 2024), pages 109–131, Bangkok, Thailand. Association for Computational Linguistics… find
  72. Bolei Ma, Xinpeng Wang, Tiancheng Hu, Anna-Carolina Haensch, Michael A. Hedderich, Barbara Plank, and Frauke Kreuter. 2024. The potential and challenges of evaluating attitudes, opinions, and values in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, p… find
  73. Bolei Ma, Berk Yoztyurk, Anna-Carolina Haensch, Xinpeng Wang, Markus Herklotz, Frauke Kreuter, Barbara Plank, and Matthias Aßenmacher. 2025. Algorithmic fidelity of large language models in generating synthetic German public opinions: A case study. In Proceedings of the 63rd Annual Meeting of the As…
  74. Nicole Meister, Carlos Guestrin, and Tatsunori Hashimoto. 2025. Benchmarking distributional alignment of large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long P…
  75. Mark-Alec Mellor and 1 others. 2023. Interview with ChatGPT. The Irenaut, 1(1):152–164. find
  76. Keiichi Namikoshi, Alex Filipowicz, David A. Shamma, Rumen Iliev, Candice L. Hogan, and Nikos Arechiga. 2024. Using llms to model the beliefs and preferences of targeted populations. Preprint, arXiv:2403.20252. — [arXiv:2403.20252](https://arxiv.org/abs/2403.20252) find
  77. Cornelia Neuert, Katharina Meitinger, Dorothée Behr, and Matthias Schonlau. 2021. Editorial: The use of open-ended questions in surveys. Methods, data, analyses : a journal for quantitative methods and survey methodology (mda), 15(1):3–6. find
  78. Shiwen Ni, Guhong Chen, Shuaimin Li, Xuanang Chen, Siyi Li, Bingli Wang, Qiyao Wang, Xingjian Wang, Yifan Zhang, Liyang Fan, Chengming Li, Ruifeng Xu, Le Sun, and Min Yang. 2025. A survey on large language model benchmarks. Preprint, arXiv:2508.15361. — [arXiv:2508.15361](https://arxiv.org/abs/2508.15361) find
  79. Jacopo Nudo, Mario Edoardo Pandolfo, Edoardo Loru, Mattia Samory, Matteo Cinelli, and Walter Quattrociocchi. 2026. Generative exaggeration in llm social agents: Consistency, bias, and toxicity. Online Social Networks and Media, 51:100344. find
  80. Yfke P. Ongena and Wil Dijkstra. 2006. Methods of behavior coding of survey interviews. Journal of Official Statistics, 22(3):1–34. find
  81. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Train…
  82. Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST ’23, New York, NY, USA. Ass… find
  83. Joon Sung Park, Lindsay Popowski, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2022. Social simulacra: Creating populated prototypes for social computing systems. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, UIST ’22, New Yo…
  84. Joon Sung Park, Carolyn Q. Zou, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Robb Willer, Percy Liang, and Michael S. Bernstein. 2024. Generative agent simulations of 1,000 people. Preprint, arXiv:2411.10109.
  85. Pouya Pezeshkpour and Estevam Hruschka. 2024. Large language models sensitivity to the order of options in multiple-choice questions. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2006–2017, Mexico City, Mexico. Association for Computational Linguistics.
  86. Chau Minh Pham, Alexander Hoyle, Simeng Sun, Philip Resnik, and Mohit Iyyer. 2024. TopicGPT: A prompt-based topic modeling framework. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pape…
  87. Jinghua Piao, Yuwei Yan, Jun Zhang, Nian Li, Junbo Yan, Xiaochong Lan, Zhihong Lu, Zhiheng Zheng, Jing Yi Wang, Di Zhou, Chen Gao, Fengli Xu, Fang Zhang, Ke Rong, Jun Su, and Yong Li. 2025. Agentsociety: Large-scale simulation of llm-driven generative agents advances understanding of human behaviors… — [arXiv:2502.08691](https://arxiv.org/abs/2502.08691) find
  88. Natalia Pinzón, Vikram Koundinya, Ryan E. Galt, William O’R. Dowling, Marcela Baukloh, Namah C. Taku-Forchu, Tracy Schohr, Leslie M. Roche, Samuel Ikendi, Mark Cooper, Lauren E. Parker, and Tapan B. Pathak. 2024. Ai-powered fraud and the erosion of online survey integrity: an analysis of 31 fraud de… find
  89. Rolf Porst. 2014. Fragebogen, 4 edition. Studien-skripten zur Soziologie. Springer VS, Wiesbaden. find
  90. Paul Röttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Kirk, Hinrich Schuetze, and Dirk Hovy. 2024. Political compass or spinning arrow? towards more meaningful evaluations for values and opinions in large language models. In Proceedings of the 62nd Annual Meeting of the Associatio…
  91. Jessie Rouder, Olivia Saucier, Rachel Kinder, and Matt Jans. 2021. What to Do With All Those Open-Ended Responses? Data Visualization Techniques for Survey Researchers. Survey Practice. find
  92. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org.
  93. Matthias Schonlau and Mick P. Couper. 2016. Semi-automated categorization of open-ended questions. Survey Research Methods, 10(2):143–152. find
  94. Matthias Schonlau, Julia Weiß, and Jan Marquardt. 2023. Multi-label classification of open-ended questions with bert. In 2023 Big Data Meets Survey Science (BigSurv), pages 1–8. find
  95. Jonathon P. Schuldt and Sungjong Roh. 2014. Media frames and cognitive accessibility: What do “global warming” and “climate change” evoke in partisan minds? Environmental Communication, 8(4):529– 548. find
  96. Sander Schulhoff, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chenglei Si, Yinheng Li, Aayush Gupta, HyoJung Han, Sevien Schulhoff, Pranav Sandeep Dulepet, Saurav Vidyadhara, Dayeon Ki, Sweta Agrawal, Chau Pham, Gerson Kroiz, Feileen Li, Hudson Tao, Ashay Srivastava, and 12 other… — The Prompt Report: A Systematic Survey of Prompt Engineering Techniques
  97. Howard Schuman and Stanley Presser. 1979. The open and closed question. American Sociological Review, 44(5):692–712. find
  98. Howard Schuman and Stanley Presser. 1996. Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context. SAGE Publications, Inc, Thousand Oaks, CA. find
  99. Indira Sen, Marlene Lutz, Elisa Rogers, David Garcia, and Markus Strohmaier. 2025a. Missing the margins: A systematic literature review on the demographic representativeness of LLMs. In Findings of the Association for Computational Linguistics: ACL 2025, pages 24263–24289, Vienna, Austria. Associati… find
  100. Indira Sen, Bolei Ma, Georg Ahnert, Anna-Carolina Haensch, Tobias Holtdirk, Frauke Kreuter, and Markus Strohmaier. 2025b. Connecting natural language processing and survey methodology: Potentials, challenges, and open questions. find
  101. Burr Settles. 2009. Active learning literature survey. Technical report, University of Wisconsin-Madison Department of Computer Sciences.
  102. Joongi Shin, Michael A. Hedderich, Bartłomiej Jakub Rey, Andrés Lucero, and Antti Oulasvirta. 2024. Understanding human-ai workflows for generating personas. In Proceedings of the 2024 ACM Designing Interactive Systems Conference, DIS ’24, page 757–781, New York, NY, USA. Association for Computing M… find
  103. Bangzhao Shu, Lechen Zhang, Minje Choi, Lavinia Dunagan, Lajanugen Logeswaran, Moontae Lee, Dallas Card, and David Jurgens. 2024. You don’t need a personality test to know these models are unreliable: Assessing the reliability of large language models on psychometric instruments. In Proceedings of t… find
  104. Eleanor Singer and Mick P. Couper. 2017. Some methodological uses of responses to open questions and other verbatim comments in quantitative surveys. methods, data, analyses, 11(2):115–134. find
  105. Oliver Slumbers, Joel Z. Leibo, and Marco A. Janssen. 2025. Using large language models to simulate human behavioural experiments: Port of mars. Preprint, arXiv:2506.05555. — [arXiv:2506.05555](https://arxiv.org/abs/2506.05555) find
  106. Kenneth O. Stanley and Joel Lehman. 2015. Why Greatness Cannot Be Planned, 1 edition. Springer Cham, Cham, Switzerland. find
  107. Amir Taubenfeld, Yaniv Dover, Roi Reichart, and Ariel Goldstein. 2024. Systematic biases in LLM simulations of debates. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 251–267, Miami, Florida, USA. Association for Computational Linguistics.
  108. Kea Tijdens. 2014. Dropout rates and response times of an occupation search tree in a web survey. Journal of Official Statistics, 30(1):23–43. find
  109. Petter Törnberg, Diliara Valeeva, Justus Uitermark, and Christopher Bail. 2023. Simulating social media using large language models to evaluate alternative news feed algorithms. arXiv preprint arXiv:2310.05984. — [arXiv:2310.05984](https://arxiv.org/abs/2310.05984) find
  110. Roger Tourangeau, Lance J. Rips, and Kenneth Rasin-ski. 2000. The Psychology of Survey Response. Cambridge University Press, Cambridge, UK. find
  111. Amos Tversky and Daniel Kahneman. 1981. The framing of decisions and the psychology of choice. Science, 211(4481):453–458. find
  112. Petter Törnberg. 2025. Large language models outperform expert coders and supervised classifiers at annotating political social media messages. Social Science Computer Review, 43(6):1181–1195. find
  113. Leah von der Heyde, Anna-Carolina Haensch, Bernd Weiß, and Jessica Daikeler. 2025a. Using large language models for coding german open-ended survey responses on survey motivation. Survey Research Methods, 19(4):355–370. find
  114. Leah von der Heyde, Anna-Carolina Haensch, and Alexander Wenz. 2025b. Vox populi, vox ai? using large language models to estimate german vote choice. Social Science Computer Review, 0(0):08944393251337014. find
  115. Qian Wang, Jiaying Wu, Zhenheng Tang, Bingqiao Luo, Nuo Chen, Wei Chen, and Bingsheng He. 2025a. What limits llm-based human simulation: Llms or our design? Preprint, arXiv:2501.08579. — [arXiv:2501.08579](https://arxiv.org/abs/2501.08579) find
  116. Xinpeng Wang, Chengzhi Hu, Bolei Ma, Paul Rottger, and Barbara Plank. 2024a. Look at the text: Instruction-tuned language models are more robust multiple choice selectors than you think. In First Conference on Language Modeling. find
  117. Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul Röttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. 2024b. “my answer is C”: First-token probabilities do not match text answers in instruction-tuned language models. In Findings of the Association for Computational Linguistics ACL 2024…
  118. Xunzhi Wang, Zhuowei Zhang, Gaonan Chen, Qiongyu Li, Bitong Luo, Zhixin Han, Haotian Wang, Zhiyu Li, Hang Gao, and Mengting Hu. 2025b. UBench: Benchmarking uncertainty in large language models with multiple choice questions. In Findings of the Association for Computational Linguistics: ACL 2025, pag… find
  119. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Associates, Inc.
  120. Sheng-Lun Wei, Cheng-Kuang Wu, Hen-Hsen Huang, and Hsin-Hsi Chen. 2024. Unveiling selection biases: Exploring order and token sensitivity in large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5598– 5621, Bangkok, Thailand. Association for Computation… find
  121. Joseph Wilson, Benjamin Pollard, John M. Aiken, Marcos D. Caballero, and H. J. Lewandowski. 2022. Classification of open-ended responses to a research-based assessment using natural language processing. Phys. Rev. Phys. Educ. Res., 18:010141. find
  122. Alexander Wuttke, Matthias Aßenmacher, Christopher Klamm, Max M. Lang, Quirin Würschinger, and Frauke Kreuter. 2025. AI conversational interviewing: Transforming surveys with LLMs as adaptive interviewers. In Proceedings of the 9th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heri… find
  123. Weijia Xu, Nebojsa Jojic, Sudha Rao, Chris Brockett, and Bill Dolan. 2025. Echoes in ai: Quantifying lack of plot diversity in llm outputs. Proceedings of the National Academy of Sciences, 122(35):e2504966122. find
  124. Diyi Yang, Dirk Hovy, David Jurgens, and Barbara Plank. 2025. Socially aware language technologies: Perspectives and practices. Computational Linguistics, 51:689–703.
  125. Simone Zhang, Janet Xu, and AJ Alvero. 2025. Generative ai meets open-ended survey responses: Research participant use of ai and homogenization. Sociological Methods & Research, 54(3):1197–1242. find
  126. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020a. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. find
  127. Yiqun Zhang, Xiaocui Yang, Shi Feng, Daling Wang, Yifei Zhang, and Kaisong Song. 2024. Can llms beat humans in debating? a dynamic multi-agent framework for competitive debate. Preprint, arXiv:2408.04472. — [arXiv:2408.04472](https://arxiv.org/abs/2408.04472) find
  128. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020b. DIALOGPT : Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin…
  129. Zhisong Zhang, Emma Strubell, and Eduard Hovy. 2022. A survey of active learning for natural language processing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6166–6190, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  130. Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations. find
  131. Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2024. Can large language models transform computational social science? Computational Linguistics, 50(1):237–291.
  132. Cornelia Züll. 2016. Open-Ended Questions. GESIS – Leibniz Institute for the Social Sciences, Mannheim, Germany. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ma-2026-too-open-opinion-embracing-open/