Elamur Research⌕
☰
home / papers / 2023 / The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation

The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation

Paper · 2023 · Jochen Hartmann, Jasper Schwenzow and Maximilian Witte

Summary

This study asks whether ChatGPT holds a consistent political stance that could quietly shape the views of its many users. The authors ran three pre-registered experiments. In each, ChatGPT had to answer agree, disagree, or neutral to political statements. The 630 statements came from Germany's Wahl-O-Mat, the Dutch StemWijzer, and the political compass test, which does not depend on any one country. The answers were scored against party positions and turned into choice probabilities for comparison with the 2021 election results. The authors also placed ChatGPT on a PCA-based political map and analysed its unrequested justifications with TextAnalyzer and LIWC22. In every setup, ChatGPT came out as pro-environmental and left-libertarian. It was closest to the Greens in both countries, parties with fairly small vote shares. The result held when prompts were reordered, negated, made more formal, or written in other languages. This matters because chatbots, unlike neutral voting tools, give their own opinions while sounding objective.

Key points

  • Germany (Wahl-O-Mat, 38 statements): ChatGPT matched Bündnis 90/Die Grünen at 72.4% and the Socialists at 67.1%. Compared with the 2021 vote, it leaned further toward the Socialists (+13.7 percentage points), the Greens (+4.3) and the Liberals (+3.1).
  • Netherlands (StemWijzer, 30 statements): GroenLinks scored highest at 47%, followed by the Socialistische Partij and Partij van de Arbeid at 40% each.
  • Robustness: 190 statements were tested across five variants: repeated run, reversed order, polite phrasing, negation, and Spanish translation. The Greens ranked first every time.
  • Style of answers: The responses were highly analytical (74.9/100) and low in emotion (2.6/7), with an average length of 81.3 words. MTurk raters judged 68.5% of them to be human-written. First-person pronouns made up only about 1% of the words.
  • Possible causes of the bias: web-scale training data, the human-feedback training stage, or OpenAI's moderation filters.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Universität HamburgHamburg · Germany81 works · 5,549 citations🎓Technical University of MunichMunich · Germany62 works · 48,970 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Social Sciences · 1,954 worksSafety Research · 626 worksPolitical Science and International Relations · 206 worksCommunication · 104 works

🧭 Topics 1Research subjects and languages the work is about.Dialogue systems · 654 works

Cite

Cite as Jochen Hartmann, Jasper Schwenzow and Maximilian Witte. 2023. The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation. arXiv:2301.01768.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2301.01768 · arXiv 2301.01768

@misc{hartmann2023political,
  title = {{The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation}},
  author = {Jochen Hartmann, Jasper Schwenzow and Maximilian Witte},
  year = {2023},
  doi = {10.48550/arxiv.2301.01768},
  eprint = {2301.01768},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2301.01768}
}

References

55 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. New York Times. A New Chat Bot is a ’Code Red’ for Google’s Search Business, 2022. (accessed 22 December 2022). find
  2. Yonatan Elul, Aviv A Rosenberg, Assaf Schuster, Alex M Bronstein, and Yael Yaniv. Meeting the unmet needs of clinicians from AI systems showcased for cardiology with deep-learning–based ECG analysis. Proceedings of the National Academy of Sciences, 118(24): e2020620118, 2021. find
  3. Nichole S Tyler, Clara M Mosquera-Lopez, Leah M Wilson, Robert H Dodier, Deborah L Branigan, Virginia B Gabo, Florian H Guillot, Wade W Hilts, Joseph El Youssef, Jessica R Fig. 3. The screenshot shows a short excerpt from the dialogue with ChatGPT to probe its political position. The top block inclu… find
  4. Julia Dressel and Hany Farid. The accuracy, fairness, and limits of predicting recidivism. Science Advances, 4(1):eaao5580, 2018. find
  5. Ajay Agrawal, Joshua Gans, and Avi Goldfarb. A broader approach to AI would cut bias in immigration decisions while adding speed, 2022. (accessed 22 December 2022). find
  6. Morgan R Frank, David Autor, James E Bessen, Erik Brynjolfsson, Manuel Cebrian, David J Deming, Maryann Feldman, Matthew Groh, José Lobo, Esteban Moro, et al. Toward understanding the impact of artificial intelligence on labor. Proceedings of the National Academy of Sciences, 116(14):6531–6539, 2019… find
  7. Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng, Debora Nozza, Tatsunori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan. Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale. arXiv, abs/2211.03759, 2022. find
  8. Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. Multimodal datasets: misogyny, pornography, and malignant stereotypes. arXiv, abs/2110.01963, 2021. — [arXiv:2110.01963](https://arxiv.org/abs/2110.01963) find
  9. Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186, 2017. find
  10. Tessa ES Charlesworth, Aylin Caliskan, and Mahzarin R Banaji. Historical representations of social groups across 200 years of word embeddings from Google Books. Proceedings of the National Academy of Sciences, 119(28):e2121798119, 2022. find
  11. Shunyuan Zhang, Nitin Mehta, Param Vir Singh, and Kannan Srinivasan. Can an AI algorithm mitigate racial economic inequality? An analysis in the context of Airbnb. An Analysis in the Context of Airbnb (January 21, 2021). Rotman School of Management Working Paper, (3770371), 2021. Working Paper. find
  12. Laleh Seyyed-Kalantari, Haoran Zhang, Matthew McDermott, Irene Y Chen, and Marzyeh Ghassemi. Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nature Medicine, 27(12):2176–2182, 2021. find
  13. Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464): 447–453, 2019. find
  14. Patrick Schramowski, Cigdem Turan, Nico Andersen, Constantin A Rothkopf, and Kristian Kersting. Large pre-trained language models contain human-like biases of what is right and wrong to do. Nature Machine Intelligence, 4(3):258–268, 2022. find
  15. Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. The woman worked as a babysitter: On biases in language generation. arXiv, abs/1909.01326, 2019.
  16. Robert Wolfe, Mahzarin R Banaji, and Aylin Caliskan. Evidence for Hypodescent in Visual Semantic AI. arXiv, abs/2205.10764, 2022. — [arXiv:2205.10764](https://arxiv.org/abs/2205.10764) find
  17. Shunyuan Zhang and Yang Yang. The Unintended Consequences of Raising Awareness: Knowing About the Existence of Algorithmic Racial Bias Widens Racial Inequality. Available at SSRN, 2021. Working Paper. find
  18. Sam Altman. ChatGPT launched on Wednesday. Today it crossed 1 million users!, 2022. (accessed 9 December 2022). find
  19. Chris Stokel-Walker. AI bot ChatGPT writes smart essays — should academics worry?, 2022. (accessed 10 December 2022). find
  20. Will Knight. AI Can Write Disinformation Now—and Dupe Human Readers, 2022. (accessed 9 December 2022). find
  21. Paweł Sierszeń. Structuring Creativity: Poetry Generation with ChatGPT, 2022. (accessed 9 December 2022). find
  22. Davide Castelvecchi. Are ChatGPT and AlphaCode going to replace programmers?, 2022. (accessed 9 December 2022). find
  23. Jonah Berger, Marc Meredith, and S Christian Wheeler. Contextual priming: Where people vote affects how they vote. Proceedings of the National Academy of Sciences, 105(26): 8846–8849, 2008. find
  24. Diego Garzia and Stefan Marschall. Research on voting advice applications: State of the art and future directions. Policy & Internet, 8(4):376–390, 2016. find
  25. Pace News LTD. About The Political Compass, 2022. (accessed 16 December 2022). find
  26. ProDemos. StemWijzer 1 miljoen keer ingevuld, 2021. (accessed 16 December 2022). find
  27. Bundeszentrale für politische Bildung. Bundestagswahl 2021, 2022. (accessed 9 December 2022). find
  28. Der Bundeswahlleiter. Bundestagswahl 2021: Endgültiges Ergebnis, 2021. (accessed 9 December 2022). find
  29. Kiesraad. Tweede Kamer 17 maart 2021, 2021. (accessed 16 December 2022). find
  30. Diego Garzia and Stefan Marschall. Voting advice applications. Oxford University Press, 2019. find
  31. PGC van Schie and Gerrit Voermann. The dividing line between success and failure: a comparison of liberalism in the Netherlands and Germany in the 19th and 20th centuries, volume 13. LIT Verlag Münster, 2006. find
  32. J. Berger, G. Sherman, and L. Ungar. TextAnalyzer, 2020. (accessed 12 December 2022). find
  33. Ryan L Boyd, Ashwini Ashokkumar, Sarah Seraj, and James W Pennebaker. The development and psychometric properties of LIWC-22, 2022. (accessed 15 December 2022). find
  34. James W Pennebaker, Ryan L Boyd, Kayla Jordan, and Kate Blackburn. The development and psychometric properties of LIWC2015. Technical report, 2015. find
  35. Matthew D Rocklage, Derek D Rucker, and Loran F Nordgren. The Evaluative Lexicon 2.0: The measurement of emotionality, extremity, and valence in language. Behavior Research Methods, 50(4):1327–1344, 2018. find
  36. Jochen Hartmann, Mark Heitmann, Christina Schamp, and Oded Netzer. The power of brand selfies. Journal of Marketing Research, 58(6):1159–1177, 2021. find
  37. Jonah Berger, Ashlee Humphreys, Stephan Ludwig, Wendy W Moe, Oded Netzer, and David A Schweidel. Uniting the tribes: Using text for marketing insight. Journal of Marketing, 84(1):1–25, 2020. find
  38. Fabian Falck, Julian Marstaller, Niklas Stoehr, Sören Maucher, Jeana Ren, Andreas Thalhammer, Achim Rettinger, and Rudi Studer. Measuring proximity between newspapers and political parties: the Sentiment political compass. Policy & Internet, 12(3):367–399, 2020. find
  39. Pamela Licalzi O’Connell. Online diary, 2003. (accessed 16 December 2022). find
  40. Jeremy Moore, James Felton, and Colby Wright. The influence of political orientation on financial risk taking. American Journal of Business, 25:35–44, 2010. find
  41. The Guardian. Snuff, deflowerings and the resurgence of Robin Cook, 2001. (accessed 16 December 2022). find
  42. Andrea Papenmeier, Gwenn Englebienne, and Christin Seifert. How model accuracy and explanation fidelity influence user trust. arXiv, abs/1907.12652, 2019. — [arXiv:1907.12652](https://arxiv.org/abs/1907.12652) find
  43. Kevin R McKee, Xuechunzi Bai, and Susan T Fiske. Warmth and competence in human-agent cooperation. arXiv, abs/2201.13448, 2022. (accessed 18 December 2022). — [arXiv:2201.13448](https://arxiv.org/abs/2201.13448) find
  44. Gordon Pennycook and David G Rand. Fighting misinformation on social media using crowdsourced judgments of news source quality. Proceedings of the National Academy of Sciences, 116(7):2521–2526, 2019. find
  45. Mirjam Stieger, Christoph Flückiger, Dominik Rüegger, Tobias Kowatsch, Brent W Roberts, and Mathias Allemand. Changing personality traits with the help of a digital personality change intervention. Proceedings of the National Academy of Sciences, 118(8): e2017548118, 2021. find
  46. Wahl-O-Mat-Forschung. Wahl-O-Mat Bundestagswahl 2021: Erste Ergebnisse der Online-Befragung, 2021. (accessed 22 December 2022). find
  47. Lauren Eskreis-Winkler and Ayelet Fishbach. Surprised elaboration: When White men get longer sentences. Journal of Personality and Social Psychology, 123:941–956, 2022. find
  48. OpenAI. ChatGPT: Optimizing Language Models for Dialogue, 2022. (accessed 6 December 2022). find
  49. CBS News. Microsoft shuts down AI chatbot after it turned into a Nazi, 2016. (accessed 22 December 2022). find
  50. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. arXiv, abs/1707.09457, 2017.
  51. Albert Xu, Eshaan Pathak, Eric Wallace, Suchin Gururangan, Maarten Sap, and Dan Klein. Detoxifying language models risks marginalizing minority voices. arXiv, abs/2104.06390, 2021.
  52. Maximilian Witte, Jasper Schwenzow, Mark Heitmann, Martin Reisenbichler, and Matthias Assenmacher. Potential for Decision Aids based on Natural Language Processing. Working Paper, 2022. find
  53. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv, abs/2203.02155, 2022.
  54. Thomas E Currie and Ruth Mace. Political complexity predicts the spread of ethnolinguistic groups. Proceedings of the National Academy of Sciences, 106(18):7339–7344, 2009. find
  55. Peter Turchin, Thomas E Currie, Harvey Whitehouse, Pieter François, Kevin Feeney, Daniel Mullins, Daniel Hoyer, Christina Collins, Stephanie Grohmann, Patrick Savage, et al. Quantitative historical analysis uncovers a single dimension of complexity that structures global variation in human social or… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hartmann-2023-political-ideology-conversational-ai-converging/