Elamur Research⌕
☰
home / papers / 2025 / Learning from Convenience Samples: A Case Study on Fine-Tuning LLMs for Survey Non-response in the German Longitudinal Election Study

Learning from Convenience Samples: A Case Study on Fine-Tuning LLMs for Survey Non-response in the German Longitudinal Election Study

Paper · 2025 · Tobias Holtdirk, Dennis Assenmacher, Arnim Bleier et al.

Summary

Surveys are becoming expensive, and many people do not answer. Researchers therefore often rely on convenience samples, which can distort conclusions. The authors ask whether LLMs can fill in a missing answer when part of the survey is already known. The case study is self-reported vote choice in the German Longitudinal Election Study (GLES), specifically the 2017 post-election cross-section.

They compare three approaches:

  • instruction-tuned LLMs fine-tuned on available respondents, with each respondent's predictor variables written out as a text description;
  • zero-shot LLMs, including GPT-4o;
  • tabular classifiers such as CatBoost.

Missingness is simulated in two ways: purely at random, and systematically, by training only on students, only on residents of Thuringia, or only on unemployed people. Results are scored on two levels: per-person accuracy (macro-F1) and the overall vote-share distribution (TVD).

When data are missing at random, fine-tuned models tie with tabular methods and beat zero-shot. When the training data are skewed, small fine-tuned Llama-3 models usually recover both individuals and aggregates better than the other approaches. This matters for designs that sample only easy-to-reach groups.

Key points

  • Data: GLES Cross-Section Cumulation 2009–2017 (face-to-face pre- and post-election interviews). The target is the 2017 post-election vote choice, chosen to allow comparison with earlier zero-shot GPT-3.5 work.
  • Setup:
    • Stratified 5-fold cross-validation; for the biased scenarios, the training split is filtered.
    • Experiments: E1 = missing completely at random, E2 = students only, E3 = Thuringia only, E4 = unemployed only.
    • Metrics: macro-F1 and total variation distance, reported with 95% confidence intervals.
  • Models and robustness: Llama-3 (3B, 8B) and Qwen-2.5 were fine-tuned, with results holding across sizes from 0.5B to 8B. Fine-tuned models were less hurt than zero-shot ones when party identification, the strongest predictor, was removed.
  • Where fine-tuning fails: in E4, the training set was only N=43. Fine-tuning lost its edge, over-predicted non-voters, and a zero-shot proprietary model did better.
  • Practical cost: for at-random gaps in closed-ended items, CatBoost-style classifiers stay just as good and are much cheaper. LLMs pay off mainly for biased samples or open-ended answers.

Interesting ideas

  • I1. Score every model on two levels, per-item correctness and distance between predicted and true label distributions, because a model can look fine per item while skewing the aggregate.
  • I2. Build deliberately biased training sets by filtering one subgroup out of a full dataset, then test on the whole population to measure how far a model generalises.
  • I3. Remove the single strongest predictor and see how much each method collapses, as a robustness check on shortcut reliance.
  • I4. Fine-tuning on a fixed answer format sidesteps the zero-shot mismatch between first-token probabilities and the text actually generated.
  • I5. Gains from fine-tuning vanished with very few examples (N=43), so the data threshold matters and a zero-shot fallback can win below it.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Alongside LAS, report TVD between predicted and gold deprel/UPOS distributions on silver data, to catch teacher parsers that systematically over-produce certain relations.
  • I2 (tree): Train the perceptron parser or gparse on a single treebank genre and evaluate on all genres, quantifying domain transfer before trusting silver data from a narrow source.
  • I3 (learn): Use this as a negative control, e.g. hide word forms or lemmas from the tagger or parser input, to verify that decisions rest on structure and not one lexical cue.
  • I4 (learn): For LLM teacher tiers, train or constrain the teacher to a rigid output schema and check that its stated label matches its scored label before accepting silver annotations.
  • I5 (frames): For rare VerbNet classes or frames, set a minimum-example threshold below which role assignment defers to a general teacher or rule rather than a weakly trained classifier.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Arts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksMathematics · 177 worksSociology and Political Science · 577 worksGeneral Social Sciences · 227 worksStatistics and Probability · 16 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Tobias Holtdirk, Dennis Assenmacher, Arnim Bleier and Claudia Wagner. 2025. Learning from Convenience Samples: A Case Study on Fine-Tuning LLMs for Survey Non-response in the German Longitudinal Election Study. arXiv:2509.25063.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2509.25063 · arXiv 2509.25063

@misc{holtdirk2025learning,
  title = {{Learning from Convenience Samples: A Case Study on Fine-Tuning LLMs for Survey Non-response in the German Longitudinal Election Study}},
  author = {Tobias Holtdirk, Dennis Assenmacher, Arnim Bleier and Claudia Wagner},
  year = {2025},
  doi = {10.48550/arxiv.2509.25063},
  eprint = {2509.25063},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2509.25063}
}

References

36 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alberto Acerbi and Joseph M. Stubbersfield. 2023. Large Language Models Show Human-like Content Biases in Transmission Chain Experiments. Proceedings of the National Academy of Sciences, 120(44): e2313790120. ↗
  2. Georg Ahnert, Max Pellert, David Garcia, and Markus Strohmaier. 2025. Extracting Affect Aggregates from Longitudinal Social Media Data with Temporal Adapters for Large Language Models. In Proceedings of the International AAAI Conference on Web and Social Media, volume 19, 15–36. find
  3. Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua Gubler, Christopher Rytting, and David Wingate. 2023. Out of One, Many: Using Language Models to Simulate Human Samples. Political Analysis, 31(3): 337–351. ↗
  4. Adam J. Berinsky. 2017. Measuring Public Opinion with Surveys. Annual Review of Political Science, 20(1): 309– 329. ↗
  5. James Bisbee, Joshua D. Clinton, Cassy Dorff, Brenton Kenkel, and Jennifer M. Larson. 2024. Synthetic Replacements for Human Survey Data? The Perils of Large Language Models. Political Analysis, 1–16. find
  6. Yong Cao, Haijiang Liu, Arnav Arora, Isabelle Augenstein, Paul Röttger, and Daniel Hershcovich. 2025. Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 3141–3154. Albuquerque, New Mexico: Association for Computational Linguistics. ↗
  7. Eric Chu, Jacob Andreas, Stephen Ansolabehere, and Deb Roy. 2023. Language Models Trained on Media Diets Can Predict Public Opinion. arXiv:2303.16779. ↗
  8. Joshua D Clinton, John S Lapinski, and Marc J Trussler. 2022. Reluctant Republicans, Eager Democrats?: Partisan Nonresponse and the Accuracy of 2020 Presidential Pre-election Telephone Polls. Public Opinion Quarterly, 86(2): 247–269. find
  9. Mick P. Couper. 2017. New Developments in Survey Data Collection. Annual Review of Sociology, 43(1): 121–145. ↗
  10. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314. ↗
  11. Die Bundeswahlleiterin. 2025. Bundestagswahl 2025: Ergebnisse Thüringen. https://www.bundeswahlleiterin.de/en/bundestagswahlen/2025/ergebnisse/bund-99/land-16.html. Accessed: 2025-09-13. ↗
  12. Ricardo Dominguez-Olmedo, Moritz Hardt, and Celestine Mendler-Dünner. 2023. Questioning the Survey Responses of Large Language Models. arXiv:2306.07951. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, and 516 others. 2024. The Llama 3 Herd of Models. arXiv:2407.21783. ↗
  13. Sarah Feldman. 2022. Who Are The People Who Don’t Respond To Polls? Clémentine Fourrier, Nathan Habib, Thomas Wolf, and Lewis Tunstall. 2023. Lighteval. find
  14. GLES. 2020. GLES Cross-Section 2009-2017, Cumulation-GLES Querschnitt 2009-2017, Kumulation. find
  15. Hang Jiang, Doug Beeferman, Brandon Roy, and Deb Roy. 2022. CommunityLM: Probing Partisan Worldviews from Language Models. In Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, and Seung-Hoon Na, eds., Proceedings of the 29th International Conference on Computational Linguistics, 6818– 6826. Gyeongju, Republic of Korea: International Committee on Computational Linguistics. ↗
  16. Damjan Kalajdzievski. 2023. A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA. arXiv:2312.03732. ↗
  17. Junsol Kim and Byungkyu Lee. 2023. AI-Augmented Surveys: Leveraging Large Language Models and Surveys for Opinion Prediction. arXiv:2305.09620. ↗
  18. Markus Klein. 2014. Gesellschaftliche Wertorientierungen, Wertewandel und Wählerverhalten. In Jürgen W. Falter and Harald Schoen, eds., Handbuch Wahlforschung, 563–590. Wiesbaden: Springer Fachmedien. ISBN 978-3-658-05164-8. ↗
  19. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, and 31 others. 2023. Holistic Evaluation of Language Models. arXiv:2211.09110. ↗
  20. Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A. Raffel. 2022. Few-Shot Parameter-Efficient Fine-Tuning Is Better and Cheaper than In-Context Learning. Advances in Neural Information Processing Systems, 35: 1950–1965.
  21. Bolei Ma, Berk Yoztyurk, Anna-Carolina Haensch, Xinpeng Wang, Markus Herklotz, Frauke Kreuter, Barbara Plank, and Matthias Aßenmacher. 2025. Algorithmic Fidelity of Large Language Models in Generating Synthetic German Public Opinions: A Case Study. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, eds., Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1785–1809. Vienna, Austria: Association for Computational Linguistics. ISBN 979-8-89176-251-0. ↗
  22. Yinan Mei, Shaoxu Song, Chenguang Fang, Haifeng Yang, Jingyun Fang, and Jiang Long. 2021. Capturing Semantics for Imputation with Pre-trained Language Models. In 2021 IEEE 37th International Conference on Data Engineering (ICDE), 61–72. find
  23. Kevin J. Mullinix, Thomas J. Leeper, James N. Druckman, and Jeremy Freese. 2015. The Generalizability of Survey Experiments. Journal of Experimental Political Science, 2(2): 109–138. ↗
  24. Avanika Narayan, Ines Chami, Laurel Orr, Simran Arora, and Christopher Ré. 2022. Can Foundation Models Wrangle Your Data? arXiv:2205.09911. ↗
  25. Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative Agents: Interactive Simulacra of Human Behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST ’23, 1–22. New York, NY, USA: Association for Computing Machinery. ISBN 9798400701320. find
  26. Yao Qu and Jue Wang. 2024. Performance and Biases of Large Language Models in Public Opinion Simulation. Humanities and Social Sciences Communications, 11(1): 1–13. Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, and 24 others. 2025. Qwen2.5 Technical Report. arXiv:2412.15115. ↗
  27. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose Opinions Do Language Models Reflect? arXiv:2303.17548. ↗
  28. Rüdiger Schmitt-Beck, Sigrid Roßteutscher, Harald Schoen, Bernhard Weßels, and Christof Wolf. 2022. A New Era of Electoral Instability. In Rüdiger Schmitt-Beck, Sigrid Roßteutscher, Harald Schoen, Bernhard Weßels, Christof Wolf, Rüdiger Schmitt-Beck, Sigrid Roßteutscher, Harald Schoen, Bernhard Weßels, and Christof Wolf, eds., The Changing German Voter, 3–24. Oxford University Press. ISBN 978-0-19-884751-9. ↗
  29. Harald Schoen, Sigrid Roßteutscher, Rüdiger Schmitt-Beck, Bernhard Weßels, Christof Wolf, Harald Schoen, Sigrid Roßteutscher, Rüdiger Schmitt-Beck, Bernhard Weßels, and Christof Wolf, eds. 2017. Voters and Voting in Context: Multiple Contexts and the Heterogeneous German Electorate. Oxford, New York: Oxford University Press. ISBN 978-0-19-879213-0. find
  30. Murray Shanahan, Kyle McDonell, and Laria Reynolds. 2023. Role Play with Large Language Models. Nature, 623(7987): 493–498. ↗
  31. Eleanor Singer and Richard A Kulka. 2002. Paying respondents for survey participation. Studies of welfare populations: Data collection and research issues, 4: 105–128. find
  32. Joseph Suh, Erfan Jahanparast, Suhong Moon, Minwoo Kang, and Serina Chang. 2025. Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinions. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 21147–21170. Vienna, Austria: Association for Computational Linguistics.
  33. Stef Van Buuren. 2018. Flexible imputation of missing data. CRC press. find
  34. Leah von der Heyde, Anna-Carolina Haensch, and Alexander Wenz. 2024. Vox Populi, Vox AI? Using Language Models to Estimate German Public Opinion. find
  35. Boshi Wang, Xiang Yue, and Huan Sun. 2023. Can Chat-GPT Defend Its Belief in Truth? Evaluating LLM Reasoning via Debate. In Houda Bouamor, Juan Pino, and Kalika Bali, eds., Findings of the Association for Computational Linguistics: EMNLP 2023, 11865–11881. Singapore: Association for Computational Linguistics. find
  36. Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul Röttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. 2024. ”My Answer Is C”: First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Models. arXiv:2402.14499. *Figure 6: The original prompt design without translated survey items.* A.2 GLES Questionnaire Variable GLES Code GLES Questionnaire (German) GLES Questionnaire (English transl.) Year year Erhebungsjahr Survey year* Age d2 Sagen Sie mir bitte, wie alt Sie sind. Please tell me how old you are.* Gender d1 [Interviewer] Ist die männlich oder weiblich? Zielperson [Interviewer] Is the respondent male or female? Education d5 Welchen höchsten allgemeinbildenden Schulabschluss haben Sie? What’s your highest level of general education? Income d43 Wie hoch ist das monatliche Netto-Einkommen IHRES HAUSHALTES INSGESAMT? Ich meine dabei die Summe, die nach Abzug von Steuern und Sozialversicherungsbeiträgen übrig bleibt. Bitte ordnen Sie Ihr Haushaltseinkommen in die Kategorien der Liste ein und nennen mir den Buchstaben. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/holtdirk-2025-learning-convenience-samples-case-study/