Elamur Research⌕
☰
home / papers / 2024 / Beyond Prompt Brittleness: Evaluating the Reliability and Consistency of Political Worldviews in LLMs

Beyond Prompt Brittleness: Evaluating the Reliability and Consistency of Political Worldviews in LLMs

Paper · 2024 · Tanise Ceron, Neele Falk, Ana Barić et al.

Summary

Earlier studies found that LLMs answering political questionnaires tend to side with left-liberal positions. It remained unclear whether those answers survive changes in prompt wording, and whether the leaning holds together across policy areas. The authors separate political bias, meaning a stable preference on individual statements, from a political worldview, which also requires coherent stances across issues. They build ProbVAA, a dataset of voting-advice questionnaire statements from seven EU countries. It includes party answers, policy-issue labels, and paraphrased, negated and meaning-inverted variants of each statement. Models from 7B to 70B parameters give binary agree/disagree answers under several instruction variants, with repeated sampling. Only statements that pass every robustness check are used for stance analysis. Reliability rises with model size. Larger models match left-leaning parties most closely, but their positions are not internally consistent. They support environmental protection, the welfare state and liberal social policy, yet also law and order, and take no clear side on migration or foreign policy. The authors therefore warn against simply calling LLMs left-wing, and they offer a reusable evaluation recipe that accounts for prompt sensitivity.

Key points

  • ProbVAA dataset: voting-advice application statements from seven EU countries, with political parties' answers and policy-issue annotations. Native speakers checked the translations and wrote negated and opposite versions. Models are evaluated only on English translations.
  • Reliability tests:
    • stability across 30 sampled responses per prompt, with a significance check on the majority answer;
    • robustness to paraphrase, negation and semantic inversion of the statement;
    • robustness to personal vs. impersonal instructions, synonymous answer labels, and swapped label order.
  • Size and reliability: reliability increases with parameter count (7B–70B). Base models were too unreliable to analyse, so the effect of RLHF could not be studied.
  • Stance findings: overall alignment is closest to left-leaning parties, but it mixes left-wing positions (environment, welfare, liberal society) with right-wing ones (law and order). Migration and foreign policy show no consistent preference.
  • Answer simplification: collapsing party answers to agree/neutral/disagree barely changes inter-party distances (average r = 0.96). Funding came from DFG (project MARDY 2, priority program RATIO).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of StuttgartStuttgart · Germany91 works · 46,376 citations🎓University of ManchesterManchester · United Kingdom87 works · 44,938 citations🎓University of ZagrebZagreb · Croatia10 works · 1,179 citations🎓Faculty of Electrical Engineering and Computing in ZagrebZagreb · Croatia2 works · 287 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Tanise Ceron, Neele Falk, Ana Barić, Dmitry Nikolaev and Sebastian Padó. 2024. Beyond Prompt Brittleness: Evaluating the Reliability and Consistency of Political Worldviews in LLMs. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00710

@article{ceron2024beyond,
  title = {{Beyond Prompt Brittleness: Evaluating the Reliability and Consistency of Political Worldviews in LLMs}},
  author = {Tanise Ceron, Neele Falk, Ana Barić, Dmitry Nikolaev and Sebastian Padó},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.1162/tacl\_a\_00710},
  url = {https://doi.org/10.1162/tacl_a_00710}
}

References

53 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. American Educational Research Association, American Psychological Association, and National Council on Measurement in Education, editors. 1999. Standards for Educational and Psychological Testing. American Educational Research Association. find
  2. Arnav Arora, Lucie-aimée Kaffee, and Isabelle Augenstein. 2023. Probing pre-trained language models for cross-cultural differences in values. In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pages 114–130, Dubrovnik, Croatia. Association for Computational Lingui…
  3. Jack M. Balkin. 2017. Digital speech and democratic culture: A theory of freedom of expression for the information society. In Law and Society Approaches to Cyberspace, pages 325–382, Routledge. https://doi.org/10.4324/9781351154161-9 — [doi:10.4324/9781351154161-9](https://doi.org/10.4324/9781351154161-9) ↗
  4. Victoria Basmov, Yoav Goldberg, and Reut Tsarfaty. 2024. Simple linguistic inferences of large language models (LLMs): Blind spots and blinds. ArXiv, abs/2305.14785. — [arXiv:2305.14785](https://arxiv.org/abs/2305.14785) find
  5. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Margaret Mitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623. https://doi.org/10 .1145/3442188.3445922 find
  6. Kenneth Benoit and Michael Laver. 2006. Party Policy in Modern Democracies. Routledge. https://doi.org/10.4324/9780203028179 ↗
  7. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of ‘‘bias’’ in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454–5476, Online. Association for Computational Linguistic…
  8. Ian Budge. 2013. The standard right-left scale. Technical report, Comparative Manifesto Project. find
  9. Ian Budge, Hans-Dieter Klingemann, Andrea Volkens, Judith Bara, and Eric Tanenbaum, editors. 2001. Mapping Policy Preferences: Estimates for Parties, Electors, and Governments 1945–1998. OUP, Oxford, New York. https://doi.org/10.1093/oso/9780199244003.001.0001 ↗
  10. Tanise Ceron, Dmitry Nikolaev, and Sebastian Padó. 2023. Additive manifesto decomposition: A policy domain aware method for understanding party positioning. In Findings of the Association for Computational Linguistics: ACL 2023, pages 7874–7890, Toronto, Canada. Association for Computational Lingui… find
  11. Leshem Choshen, Ariel Gera, Yotam Perlitz, Michal Shmueli-Scheuer, and Gabriel Stanovsky. 2024. Navigating the modern evaluation landscape: Considerations in benchmarks and frameworks for large language models (LLMs). In Proceedings of the 2024 Joint International Conference on Computational Linguis… find
  12. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53. the survey responses of large language models. ArX… find
  13. Esin Durmus, Karina Nyugen, Thomas Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. 2023. Towards measuring the…
  14. Ullrich K. H. Ecker, Brandon K. N. Sze, and Matthew Andreotta. 2021. Corrections of political misinformation: No evidence for an effect of partisan worldview in a US convenience sample. Philosophical Transactions of the Royal Society B: Biological Sciences, 376(1822):20200145. https://doi .org/10.10… — [doi:10.1098/rstb.2020.0145](https://doi.org/10.1098/rstb.2020.0145) find
  15. David Esiobu, Xiaoqing Tan, Saghar Hosseini, Megan Ung, Yuchen Zhang, Jude Fernandes, Jane Dwivedi-Yu, Eleonora Presani, Adina Williams, and Eric Smith. 2023. ROBBIE: Robust bias evaluation of large generative language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Lan… find
  16. Geoffrey Evans, Anthony Heath, and Mansur Lalljee. 1996. Measuring left-right and libertarian-authoritarian values in the British electorate. The British Journal of Sociology, 47(1):93–112. https://doi.org/10 .2307/591118 find
  17. Erenay Dayanik, Thang Vu, and Sebastian Padó. 2022. Bias identification and attribution in NLP models with regression and effect sizes. Northern European Journal of Language Technology, 8(1). https://doi.org/10 .3384/nejlt.2000-1533.2022.3505 find
  18. Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. 2023. From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair NLP models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume…
  19. Ricardo Dominguez-Olmedo, Moritz Hardt, and Celestine Mendler-Dunner. 2023. find
  20. Questioning Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The pile: An 800GB dataset of diverse text for language modeling. ArXiv, abs/2101.00027. find
  21. Goran Glavaš, Federico Nanni, and Simone Paolo Ponzetto. 2017. Unsupervised cross-lingual scaling of political texts. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 688–693, Valencia, Spain. Association f… find
  22. Rishav Hada, Agrima Seth, Harshita Diddee, and Kalika Bali. 2023. ‘‘Fifty shades of bias’’: Normative ratings of gender bias in GPT generated English text. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1862–1876, Singapore. Association for Computati…
  23. Jochen Hartmann, Jasper Schwenzow, and Maximilian Witte. 2023. The political ideology of conversational AI: Converging evidence on ChatGPT’s pro-environmental, left-libertarian orientation. SSRN Electronic Journal. https://doi.org/10.2139/ssrn .4316084 ↗
  24. Michael Hermann and Heinrich Leuthold. 2001. Weltanschauung und ihre soziale Basis im Spiegel eidgenössischer Volksabstim-mungen. Swiss Political Science Review, 7(4):39–63. https://doi.org/10.1002/j.1662-6370.2001.tb00327.x Michael Hermann and Heinrich Leuthold. 2003. Atlas der politischen Landsc… ↗
  25. Andrew Heywood. 2021. Political Ideologies: An Introduction. Bloomsbury Publishing. find
  26. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In Proceedings of ICLR. Virtual.
  27. Dirk Hovy and Shrimai Prabhumoye. 2021. Five sources of bias in natural language processing. Language and Linguistics Compass, 15(8):e12432. https://doi.org/10 .1111/lnc3.12432 find
  28. Torben Iversen. 1994. Political leadership and representation in West European democracies: A test of three models of voting. American Journal of Political Science, 38(1):45–74. https://doi.org/10.2307/2111335 — [doi:10.2307/2111335](https://doi.org/10.2307/2111335) ↗
  29. Detlef Jahn. 2023. The changing relevance and meaning of left and right in 34 party systems from 1945 to 2020. Comparative European Politics, 21:308–332. https://doi.org/10 .1057/s41295-022-00305-5 find
  30. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroi… find
  31. Seth Jolly, Ryan Bakker, Liesbet Hooghe, Gary Marks, Jonathan Polk, Jan Rovny, Marco Steenbergen, and Milada Anna Vachudova. 2022. Chapel Hill expert survey trend file, 1999–2019. Electoral Studies, 75:102420. https://doi.org/10.1016/j.electstud.2021.102420 ↗
  32. Daniel Khashabi, Xinxi Lyu, Sewon Min, Lianhui Qin, Kyle Richardson, Sean Welleck, Hannaneh Hajishirzi, Tushar Khot, Ashish Sabharwal, Sameer Singh, and Yejin Choi. 2022. Prompt waywardness: The curious case of discretized interpretation of continuous prompts. In Proceedings of the 2022 Conference o… find
  33. Herbert Kitschelt. 1994. The Transformation of European Social Democracy. Cambridge University Press. https://doi.org/10 .1017/CBO9780511622014 find
  34. Hadas Kotek, Rikker Dockum, and David Sun. 2023. Gender bias and stereotypes in large language models. In Proceedings of The ACM Collective Intelligence Conference. ACM. https://doi.org/10.1145/3582269.3615599 ↗
  35. Measuring ChatGPT political bias. Public Choice, 198:3–23. https://doi.org/10 .1007/s11127-023-01097-2 Dilek Küçük and Fazli Can. 2020. Stance detection: A survey. ACM Computing Surveys, 53(1). https://doi.org/10.1145/3369026 find
  36. Jesutofunmi A. Omiye, Jenna C. Lester, Simon Spichak, Veronica Rotemberg, and Roxana Daneshjou. 2023. Large language models propagate race-based medicine. npj Digital Medicine, 6(1). https://doi.org/10 .1038/s41746-023-00939-z Michael Laver, Kenneth Benoit, and John Garry. 2003. Extracting policy po… — [doi:10.1017/s0003055403000698](https://doi.org/10.1017/s0003055403000698) find
  37. Yang Liu, Jiahuan Cao, Chongyu Liu, Kai Ding, and Lianwen Jin. 2024. Datasets for large language models: A comprehensive survey.ArXiv, abs/2402.18041. https://doi.org/10 .21203/rs.3.rs-3996137/v1 find
  38. Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2021. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR), 54(6):1–35. https://doi.org/10.1145/3457607 ↗
  39. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048–11064, Ab…
  40. Thomas R. Palfrey and Keith T. Poole. 1987. The relationship between information, ideology, and voting behavior. American Journal of Political Science, 31(3):511–530. https://doi.org/10.2307/2111281 — [doi:10.2307/2111281](https://doi.org/10.2307/2111281) ↗
  41. Jérôme Rutinowski, Sven Franke, Jan Endendyk, Ina Dormuth, Moritz Roidl, and Markus Pauly. 2024. The self-perception and political biases of chatgpt. Human Behavior and Emerging Technologies, 2024(1):7115633. https://doi.org/10.1155/2024/7115633 — [doi:10.1155/2024/7115633](https://doi.org/10.1155/2024/7115633) ↗
  42. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? In Proceedings of the 40th International Conference on Machine Learning, Honolulu, Hawaii, USA.
  43. Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. 2024. State of what art? A call for multi-prompt llm evaluation.Transactions of the Association for Computational Linguistics, 12:933–949. https://doi.org/10 .1162/tacl_a_00681 find
  44. Bangzhao Shu, Lechen Zhang, Minje Choi, Lavinia Dunagan, Lajanugen Logeswaran, Moontae Lee, Dallas Card, and David Jurgens. 2024. You don’t need a personality test to know these models are unreliable: Assessing the reliability of large language models on psychometric instruments. In Proceedings of t… find
  45. Adam N. Moskowitz and J. Craig Jenkins. 2004. Structuring political opinions: Attitude consistency and democratic competence among the u.s. mass public. The Sociological Quarterly, 45(3):395–419. https://doi.org/10.1111/j.1533-8525.2004.tb02296.x Jonathan B. Slapin and Sven-Oliver Proksch. 2008. A… — [doi:10.1145/3465416.3483305](https://doi.org/10.1145/3465416.3483305) ↗
  46. Margit Tavits. 2007. Principle vs. pragmatism: Policy shifts and political competition. American Journal of Political Science, 51(1):151–165. https://doi.org/10.1111/j.1540-5907.2007.00243.x Lindia Tjuatja, Valerie Chen, Sherry Tongshuang Wu, Ameet Talwalkar, and Graham Neubig. 2023. Do llms exhibi… ↗
  47. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  48. Cunxiang Wang, Sirui Cheng, Qipeng Guo, Yuanhao Yue, Bowen Ding, Zhikun Xu, Yidong Wang, Xiangkun Hu, Zheng Zhang, and Yue Zhang. 2023a. Evaluating Open-QA Evaluation. In Advances in Neural Information Processing Systems, volume 36, pages 77013–77042. Curran Associates, Inc. find
  49. Wenxuan Wang, Wenxiang Jiao, Jingyuan Huang, Ruyi Dai, Jen-tse Huang, Zhaopeng Tu, and Michael R. Lyu. 2023b. Not all countries celebrate Thanksgiving: On the cultural dominance in large language models. ArXiv, abs/2310.12481.
  50. Albert Webson and Ellie Pavlick. 2022. Do prompt-based models really understand the meaning of their prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2300–2344, Seattle, United States. As…
  51. Vinzenz Wolf and Christian Maier. 2024. Chatgpt usage in everyday life: A motivation-theoretic mixed-methods study. International Journal of Information Management, 79:102821. https://doi.org/10 .1016/j.ijinfomgt.2024.102821 find
  52. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12697–12706.
  53. Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2023. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ceron-2024-beyond-prompt-brittleness-evaluating-reliability/