Elamur Research⌕
☰
home / papers / 2025 / Biased LLMs can Influence Political Decision-Making

Biased LLMs can Influence Political Decision-Making

Paper · 2025 · Jillian Fisher, Shangbin Feng, Robert Aron et al.

Summary

The paper asks whether partisan bias in chat-based LLMs can change people's political views and choices during short, free-form conversations. The authors recruited 299 self-identified Democrats and Republicans on Prolific. Each participant was secretly paired with one of three models: a liberal-leaning, a conservative-leaning, or a neutral control. The leanings were created by instructing the model to answer as a liberal, conservative, or neutral citizen.

Participants completed two tasks, each with a survey before and after chatting with the model:

  • Topic Opinion Task: forming opinions on little-known policy topics.
  • Budget Allocation Task: acting as a mayor and dividing city funds across four sectors.

Opinions and allocations moved toward the stance of the assigned model. This shift appeared even when the model's leaning contradicted the participant's own party. Noticing the bias did not protect participants, but self-reported AI knowledge was weakly linked to less influence.

The findings suggest that biased LLMs could shape public discourse and policy preferences. They also point to AI literacy as a mitigation strategy alongside technical fixes.

Key points

  • Design: a 3×2 setup crossing model leaning (liberal, conservative, control) with participant party (Republican n=150, Democrat n=149).
    • Sample size came from a simulation power analysis based on a 30-person pilot.
    • Opinion change was analysed with ordinal logistic regression.
  • Opinion topics: the liberal-supported issues were the Lacey Act of 1900 along with multifamily housing, while the conservative-supported ones were covenant marriage and international unilateralism.
  • Budget task: funds were split among Public Safety, Education, Veteran Services and Welfare.
    • Biased models used different framing: security and veterans for the conservative model, equity and welfare for the liberal one.
    • Their persuasive techniques did not differ.
  • Interactions: each participant had 3–20 turns with the model, averaging about five. Only about 54% of participants in biased conditions correctly recognised the bias.
  • Limitations: the study used a single OpenAI model (reported as GPT-3 Turbo) and only US politics. It measured immediate effects only and relied on non-representative online recruitment.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States434 works · 314,419 citations🎓University of WashingtonSeattle · United States353 works · 200,043 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Arts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 worksSociology and Political Science · 577 worksGeneral Social Sciences · 227 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Jillian Fisher, Shangbin Feng, Robert Aron, Thomas Richardson, Yejin Choi, Daniel W Fisher, Jennifer Pan, Yulia Tsvetkov and Katharina Reinecke. 2025. Biased LLMs can Influence Political Decision-Making. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.328

@inproceedings{fisher2025biased,
  title = {{Biased LLMs can Influence Political Decision-Making}},
  author = {Jillian Fisher, Shangbin Feng, Robert Aron, Thomas Richardson, Yejin Choi, Daniel W Fisher, Jennifer Pan, Yulia Tsvetkov and Katharina Reinecke},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.328},
  url = {https://doi.org/10.18653/v1/2025.acl-long.328}
}

References

84 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. American national election studies. electionstudies.org. Political compass test. politicalcompass.org. https://https://www. find
  2. Alberto Acerbi and Joseph M. Stubbersfield. 2023. Large language models show human-like content biases in transmission chain experiments. Proceedings of the National Academy of Sciences, 120(44). find
  3. Swati Aggarwal, Tushar Sinha, Yash Kukreti, and Siddarth Shikhar. 2020. Media bias detection and bias short term impact assessment. Array, 6:100025. find
  4. Rohini Ahluwalia. 2000. Examination of psychological processes underlying resistance to persuasion. Journal of Consumer Research, 27(2):217–232. find
  5. Seth D Baum. 2024. Manipulating aggregate societal values to bias AI social choice ethics. AI and ethics (Online). find
  6. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 610–623, New York, NY, USA. Associat… find
  7. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of “bias” in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454– 5476, Online. Association for Computational Linguistics.
  8. Fatima Boatman, Robert Reeves, Mikitaka Masuyama, Deru Schelhaas, and Patricia Gomes Rego de Almeida. 2020. Artificial intelligence: Innovation in parliaments. Inter-Parliamentary Union: Innovation tracker, 4. find
  9. Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, page 4356–4364, Red Hook, NY…
  10. David E. Broockman and Joshua L. Kalla. 2024. Consuming cross-cutting media causes learning and moderates attitudes: A field experiment with Fox news viewers. The Journal of Politics. find
  11. Anna Brown. 2017. Republicans more likely than democrats to have confidence in police. Technical report, Pew Research Center, Washington, D.C. find
  12. John G Bullock, Alan S Gerber, Seth J Hill, and Gregory A Huber. 2015. Partisan bias in factual beliefs about politics. Journal of Political Science, 10. find
  13. Riccardo Cantini, Giada Cosenza, Alessio Orsino, and Domenico Talia. 2024. Are large language models really bias-free? jailbreak prompts for assessing adversarial robustness to bias elicitation. ArXiv. find
  14. Dallas Card, Amber E. Boydstun, Justin H. Gross, Philip Resnik, and Noah A. Smith. 2015. The media frames corpus: Annotations of frames across issues. In Annual Meeting of the Association for Computational Linguistics.
  15. Pew Research Center. 2016. Political values: Government regulation, environment, immigration, race, views of Islam. Technical report, Pew Research Center. find
  16. Pew Research Center. 2017. Partisans differ widely in views of police officers, college professors. Technical report, Pew Research Center, Washington, D.C. find
  17. Pew Research Center. 2019. In a politically polarized era, sharp divides in both partisan coalitions. Technical report, Pew Research Center, Washington, D.C. find
  18. Pew Research Center. 2024. From businesses and banks to colleges and churches: Americans’ views of U.S. institutions. Technical report, Pew Research Center, Washington, D.C. find
  19. Chun-Fang Chiang and Brian Knight. 2011. Media bias and influence: Evidence from newspaper endorsements. The Review of Economic Studies, 78(3):795– 820. find
  20. Brian Czech and Rena Borkhataria. 2001. The relationship of political party affiliation to wildlife conservation attitudes. Politics Life Science. find
  21. Jenn Hatfield. 2023. Partisan divides over K-12 education in 8 charts. Technical report, Pew Research Center, Washington, D.C. find
  22. Justin de Benedictis-Kessner, Daniel Jones, and Chris Warshaw. 2022. How partisanship in cities influences housing policy. RWP21, 35. find
  23. Alan J. Hawkins, Steven L. Nock, Julia C. Wilson, Laura Sanchez, and James D. Wright. 2002. Attitudes about covenant marriage and divorce: Policy implications from a three-state comparison. Family Relations, 51(2):166–75. find
  24. Stefano DellaVigna and Ethan Kaplan. 2007. The Fox News Effect: Media Bias and Voting. The Quarterly Journal of Economics, 122(3):1187–1234. find
  25. Stefano DellaVigna and Ethan Kaplan. 2008. The political impact of media bias. Information and Public Choice, pages 79–106. find
  26. James N. Druckman and Michael Parkin. 2005. The impact of media bias: How editorial slant affects voters. The Journal of Politics, 67(4):1030–1049. find
  27. Fatma Elsafoury, Steven R. Wilson, Stamos Katsigiannis, and Naeem Ramzan. 2022. SOS: Systematic offensive stereotyping bias in word embeddings. In International Conference on Computational Linguistics. find
  28. Robert M. Entman. 2004. Projections of Power: Framing News, Public Opinion, and U.S. Foreign Policy. University of Chicago Press, Chicago. find
  29. Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. 2023. From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair NLP models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume…
  30. Sean Freeder, Gabriel S. Lenz, and Shad Turney. 2019. The importance of knowing “what goes with what”: Reinterpreting the evidence on policy attitude stability. The Journal of Politics, 81(1):274–290. find
  31. Helen Ai He, Naomi Yamashita, Chat Wacharaman-otham, Andrea B. Horn, Jenny Schmid, and Elaine M. Huang. 2017. Two sides to every story: Mitigating intercultural conflict through automated feedback and shared self-reflections in global virtual teams. Proc. ACM Hum.-Comput. Interact., 1(CSCW). find
  32. Valentin Hofmann, Pratyusha Ria Kalluri, Dan Jurafsky, and Sharese King. 2024. AI generates covertly racist decisions about people based on their dialect. Nature, 633,8028:147–154. find
  33. Dirk Hovy and Prabhumoye Shrimai. 2021. Five sources of bias in natural language processing. Language and Linguistics Compass, vol. 15.8. find
  34. Beizhe Hu, Qiang Sheng, Juan Cao, Yuhui Shi, Yang Li, Danding Wang, and Peng Qi. 2023. Bad actor, good advisor: Exploring the role of large language models in fake news detection. In AAAI Conference on Artificial Intelligence. find
  35. Shanto Iyengar, Yphtach Lelkes, Matthew Levendusky, Neil Malhotra, and Sean J Westwood. 2019. The origins and consequences of affective polarization in the united states. Annual Review of Political Science, 22(1):129–146. find
  36. Maurice Jakesch, Advait Bhat, Daniel Buschek, Lior Zalmanson, and Mor Naaman. 2023. Co-writing with opinionated language models affects users’ views. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. find
  37. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30):e2305016120.
  38. Jakob Jensen, Courtney Scherr, Natasha Brown, Christina Jones, Katheryn Christy, and Ryan Hurley. 2014. Public estimates of cancer frequency: Cancer incidence perceptions mirror distorted media depictions. Journal of Health Communication, 19. find
  39. Nicole Gillespie, Steven Lockey, Caitlin Curtis, Javad Pool, and Ali Akbari. 2023. Trust in artificial intelligence: A global study. The University of Queensland and KPMG Australia. find
  40. Nisha Jain John Halpin, Karl Agne. 2021. Americans want the federal government to help people in need. www.americanprogress.org. find
  41. Kobi Hackenburg and Helen Margetts. 2024. Evaluating the persuasive influence of political microtargeting with large language models. Proceedings of the National Academy of Sciences, 121(24):e2403116121. find
  42. Rong Han, Jianxing Xu, and Ding Pan. 2022. How media exposure, media trust, and media bias perception influence public evaluation of covid-19 pandemic in international metropolises. International Journal of Environmental Research and Public Health, 19(7):3942. find
  43. Hannah Rose Kirk, Yennie Jun, Filippo Volpin, Haider Iqbal, Elias Benussi, Frederic Dreyer, Aleksandar Shtedritski, and Yuki Asano. 2021. Bias out-of-the-box: An empirical analysis of intersectional occupational biases in popular generative language models. In Advances in Neural Information Processi… find
  44. Ken Knapton. 2023. Council post: Navigating the biases in LLM generative AI: A guide to responsible implementation. forbes. Forbes. find
  45. Anne C Kroon, Toni G L A van der Meer, and Thomas Pronk. 2022. Does information about bias attenuate selective exposure? the effects of implicit bias feedback on the selection of outgroup-rich news. Human Communication Research, 48(2):346–373. find
  46. Jacob M. Montgomery, Brendan Nyhan, and Michelle Torres. 2018. How conditioning on posttreatment variables can ruin your experiment and what to do about it. American Journal of Political Science, 62(3):760–775. find
  47. Sachin Kumar, Vidhisha Balachandran, Lucille Njoo, Antonios Anastasopoulos, and Yulia Tsvetkov. 2023. Language generation models can cause harm: So what can we do about it? an actionable survey. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Lingui…
  48. Malvina Nissim, Rik van Noord, and Rob van der Goot. 2020. Fair is better than sensational: Man is to doctor as woman is to doctor. Computational Linguistics, 46(2):487–497.
  49. Shachi H Kumar, Saurav Sahay, Sahisnu Mazumder, Eda Okur, Ramesh Manuvinakurike, Nicole Beckage, Hsuan Su, Hung yi Lee, and Lama Nachman. 2024. Decoding biases: Automated methods and llm judges for gender bias detection in language models. Preprint, arXiv:2408.03907. — [arXiv:2408.03907](https://arxiv.org/abs/2408.03907) find
  50. OpenAI. 2023. gpt-3.5-turbo-1106. //platform.openai.com/docs/models/ gpt-3-5-turbo. Accessed: 2023-09-02. find
  51. Gabriel S. Lenz. 2012. Follow the Leader? How Voters Respond to Politicians’ Policies and Performance. University of Chicago Press, Chicago, IL. find
  52. Sinead O’Connor and Helen Liu. 2023. Gender bias perpetuation and mitigation in AI technologies: challenges and opportunities. AI & SOCIETY, pages 1–13. find
  53. Zihao (Michael) Li. 2023. The dark side of ChatGPT: Legal and ethical challenges from stochastic parrots and hallucination. ArXiv. find
  54. Ruixi Lin and Hwee Tou Ng. 2023. Mind the biases: Quantifying cognitive biases in language model prompting. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5269–5281, Toronto, Canada. Association for Computational Linguistics. find
  55. Winston Lin. 2013. Agnostic notes on regression adjustments to experimental data: Reexamining Freedman’s critique. The Annals of Applied Statistics, 7(1). find
  56. Ruibo Liu, Chenyan Jia, Jason Wei, Guangxuan Xu, and Soroush Vosoughi. 2022. Quantifying and alleviating political bias in language models. Artificial Intelligence, 304:103654. find
  57. Charles Lord, Lee Ross, and Mark Lepper. 1979. Biased assimilation and attitude polarization: The effects of prior theories on subsequently considered evidence. Journal of Personality and Social Psychology, 37:2098–2109. find
  58. Brian E. Mennecke, Joseph S. Valacich, and Bradley C. Wheeler. 2000. The effects of media and task on user performance: A test of the task-media fit hypothesis. Group Decision and Negotiation, 9(6):507–529. find
  59. Brendan Nyhan and Jason Reifler. 2010. When corrections fail: The persistence of political misperceptions. Political Behavior. https: find
  60. OpenAI. 2024. Gpt-4-turbo. https://www.openai.com/research/gpt-4-Turbo. Accessed: 2024-08-11. ↗
  61. Pew Research Center. 2023. News platform fact sheet. Technical report, Washington, D.C. find
  62. Pew Research Center. 2024. Americans’ use of Chat-GPT is ticking up, but few trust its election information. Technical report, Washington, D.C. find
  63. Jakub Piskorski, Nicolas Stefanovitch, Giovanni Da San Martino, and Preslav Nakov. 2023a. SemEval-2023 task 3: Detecting the category, the framing, and the persuasion techniques in online news in a multi-lingual setup. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval…
  64. Jakub Piskorski, Nicolas Stefanovitch, Giovanni Da San Martino, and Preslav Nakov. 2023b. Semeval-2023 task 3: Detecting the category, the framing, and the persuasion techniques in online news in a multilingual setup. In International Workshop on Semantic Evaluation. find
  65. Prolific. Prolific. https://www.prolific.com. ↗
  66. Paul Röttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Rose Kirk, Hinrich Schutze, and Dirk Hovy. 2024. Political compass or spinning arrow? towards more meaningful evaluations for values and opinions in large language models. In Annual Meeting of the Association for Computational…
  67. Cade Metz. 2023. What makes A.I. Chatbots go wrong? New York Times. find
  68. Lydia Saad. 2023. Public firm in view government doing too much, too powerful. GALLUP. find
  69. Mirja Mittermaier, Marium M. Raza, and Joseph C. Kvedar. 2023. Bias in AI-based models for medical applications: challenges and mitigation strategies. NPJ Digital Medicine, 6. find
  70. Dina Smeltz, Ivo Daalder, Craig Kafura, and Brendan Helm. 2020. Divided we stand. Chicago Council Survey of American Public Opinion and US Foreign Policy. find
  71. Sanjari Srivastava, Piotr Mardziel, Zhikhun Zhang, Archana Ahlawat, Anupam Datta, and John C Mitchell. 2024. De-amplifying bias from differential privacy in language model fine-tuning. Preprint, arXiv:2402.04489. — [arXiv:2402.04489](https://arxiv.org/abs/2402.04489) find
  72. Christopher Starke and Marco Lünich. 2020. Artificial intelligence for political decision-making in the european union: Effects on citizens’ perceptions of input, throughput, and output legitimacy. Data & Policy, 2. find
  73. Valerie Strauss. 2023. What House Republicans want to do to public education funding. Washington Post. summarization. Transactions of the Association for Computational Linguistics, 12:39–57. find
  74. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. 2019. Gender bias in contextualized word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1…
  75. Charles S. Taber and Milton Lodge. 2006. Motivated skepticism in the evaluation of political beliefs. Journal of Political Science, 50(3):755–769. find
  76. Gallup-Bently University. 2024. 2024 bentley-gallup business in society report. Technical report. find
  77. Lucía Vicente and Matute Helena. 2023. Humans inherit artificial intelligence biases. Scientific reports. find
  78. Catherine Vitro, Angus D. Clark, Carter Sherman, Mary M. Heitzeg, and Brian M. Hicks. 2022. Attitudes about police and race in the united states 2020-2021: Mean-level trends and associations with political attitudes, psychiatric problems, and covid-19 outcomes. PLOS ONE. find
  79. Thiemo Wambsganss, Xiaotian Su, Vinitra Swamy, Seyed Parsa Neshaei, Roman Rietsche, and Tanja Kaser. 2023. Unraveling downstream gender bias from large language models: A study on AI educational writing assistance. In Conference on Empirical Methods in Natural Language Processing. find
  80. Yuxuan Wan, Wenxuan Wang, Pinjia He, Jiazhen Gu, Haonan Bai, and Michael R. Lyu. 2023. Biasasker: Measuring the bias in conversational AI system. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2023, pag… find
  81. Christopher Winship and Robert D Mare. 1984. Regression models with ordinal variables. American sociological review, pages 512–525. find
  82. Fang Xiao, Che Shangkun, Mao Minjia, Zhang Hongzhe, Zhao Ming, and Zhao Xiaohang. 2023. Bias of AI-generated content: an examination of news produced by large language models. Scientific Reports, 14. find
  83. Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. 2024. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge AI safety by humanizing LLMs. Preprint, arXiv:2401.06373.
  84. Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. 2024. Benchmarking large language models for news Part find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/fisher-2025-biased-llms-can-influence-political/