Elamur Research⌕
☰
home / papers / 2025 / Large Means Left: Political Bias in Large Language Models Increases with Their Number of Parameters

Large Means Left: Political Bias in Large Language Models Increases with Their Number of Parameters

Paper · 2025 · David Exler, Mark Schutera, Markus Reischl et al.

Summary

The paper asks whether popular open-weight LLMs lean toward particular political positions, and whether model size, release date, origin or prompt language shape that lean. The authors use the Wahl-O-Mat, the German voting-advice tool from the Bundeszentrale für politische Bildung, for the 2025 Bundestag election. Each model received all 38 theses, once in the original German and once in an English translation by native speakers. It had to answer Yes, Neutral or No inside answer tags. The Wahl-O-Mat formula turned these answers into agreement scores with twelve parties. To get a single number, the authors converted each model's agreement with the five parliamentary parties into a hypothetical seat distribution. They then averaged those seats along the left-to-right seating order of the chamber, giving a position score θ. Every model tested sat left of centre, and the lean grew with parameter count. Language and release date also moved the score, but the country where a model was built did not. The authors argue this matters because hidden bias, unlike a factual error, cannot be caught by checking facts.

Key points

  • Models tested: Llama 2 7B and 70B, Llama 3 8B and 70B, Mistral 7B, DeepSeek R1 32B and Simplescaling s1 32B, all from Hugging Face. DeepSeek R1 is the only model of Chinese origin.
  • Answer coding: Yes = 0, Neutral = 1, No = 2. For each thesis, agreement with a party is 1 minus half the absolute difference, averaged over the 38 theses. Answers in the wrong format were mapped to a label by hand.
  • Seat positions for θ: Die Linke −1, SPD −0.5, GRÜNE 0, CDU/CSU 0.5, AfD 1. Parties outside parliament are dropped, so θ is relative to the current Bundestag.
  • Language effect: Mean θ was 13.5% for German prompts and 15.0% for English ones. English tended to produce firmer answers and a stronger lean, although DeepSeek R1 and Mistral did not follow this pattern.
  • Neutral answers: s1 answered Neutral 21 times in German and never in English. DeepSeek R1 showed the opposite pattern, agreeing more in German and choosing Neutral more in English.

Interesting ideas

  • I1. Reasoning models write long text that cannot be cut to one word, so the prompt asks for the final label inside explicit tags and the label is extracted from there; this makes free-form output machine-readable without truncating it.
  • I2. Answers on an ordered three-point scale are compared by distance, so a Neutral against a firm Yes earns half credit instead of zero; this gives a graded measure of disagreement rather than a strict match.
  • I3. Many per-category scores are collapsed into one interpretable number by turning them into a proportional allocation and averaging over a fixed ordinal axis; this makes systems directly comparable along a single dimension.
  • I4. The same items are run in two languages to expose shifts caused only by wording, with human translation keeping the content fixed; this works as a built-in control for surface-form effects.
  • I5. The finding that bias grows with model size is a reminder that a stronger teacher is not automatically a neutral one, so its outputs need auditing on value-laden material.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): LLM teacher tiers could be required to wrap their verdict on a parse or frame in fixed tags, so silver-data harvesting stays robust even when the teacher explains itself at length.
  • I2 (answer): school-test and fable-QA graders could score "cannot tell" answers by distance from the gold label, rewarding cautious readers over confidently wrong ones.
  • I3 (learn): agreement profiles of witness parsers across label categories could be reduced to one positioning score, so a witness that drifts systematically toward some labels stands out.
  • I4 (back): translation through the meta-world could be checked by feeding both versions through the pipeline and confirming that the frames and the world model come out identical.
  • I5 (learn): when larger LLM teachers label frames or speech acts on opinionated texts, negative controls with balanced statements could detect a systematic skew before it enters training data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Karlsruhe Institute of TechnologyKarlsruhe · Germany12 works · 834 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSafety Research · 543 worksSociology and Political Science · 577 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksBias, fairness & safety · 517 works

Cite

Cite as David Exler, Mark Schutera, Markus Reischl and Luca Rettenberger. 2025. Large Means Left: Political Bias in Large Language Models Increases with Their Number of Parameters. arXiv:2505.04393.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2505.04393 · arXiv 2505.04393

@misc{exler2025large,
  title = {{Large Means Left: Political Bias in Large Language Models Increases with Their Number of Parameters}},
  author = {David Exler, Mark Schutera, Markus Reischl and Luca Rettenberger},
  year = {2025},
  doi = {10.48550/arxiv.2505.04393},
  eprint = {2505.04393},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2505.04393}
}

References

26 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Gray, “Chatgpt ”contamination”: estimating the prevalence of llms in the scholarly literature,” arXiv preprint arXiv:2403.16887, 2024. ↗
  2. T. Brown, B. Mann, N. Ryder et al., “Language models are few-shot learners,” vol. 33, pp. 1877–1901, 2020. ↗
  3. F. Duarte. (2025) Chatgpt users: Key stats you should know. Accessed: 2025-03-02. [Online]. Available: https://explodingtopics.com/blog/chatgpt-users ↗
  4. Z. A. Nazi and W. Peng, “Large language models in healthcare and medical domain: A review,” Informatics, vol. 11, no. 3, 2024. ↗
  5. Q. Zhang, K. Ding, T. Lv et al., “Scientific large language models: A survey on biological & chemical domains,” ACM Comput. Surv., vol. 57, no. 6, 2025. ↗
  6. E. M. Bender, T. Gebru, A. McMillan-Major et al., “On the dangers of stochastic parrots: Can language models be too big?” in Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, ser. FAccT ’21, 2021, p. 610–623. find
  7. A. Gilson, C. W. Safranek, T. Huang et al., “How does chatgpt perform on the united states medical licensing examination? the implications of large language models for medical education and knowledge assessment,” JMIR Med Educ, vol. 9, Feb 2023. find
  8. L. Li, J. Li, C. Chen et al., “Political-llm: Large language models in political science,” arXiv preprint arXiv:2412.06864, 2024. ↗
  9. tagesschau.de. (2025, Feb.) Nutzerrekord bei Wahl-O-Mat. [Online]. Available: https://www.tagesschau.de/inland/bundestagswahl/ wahl-o-mat-rekord-100.html ↗
  10. H. Touvron, L. Martin, K. Stone et al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023. ↗
  11. A. Q. Jiang, A. Sablayrolles, A. Mensch et al., “Mistral 7b,” arXiv preprint arXiv:2310.0 6825, 2023. find
  12. L. Rettenberger, M. Reischl, and M. Schutera, “Assessing political bias in large language models,” Journal of Computational Social Science, vol. 8, no. 2, pp. 1–17, 2025. ↗
  13. D. Guo, D. Yang, H. Zhang et al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” arXiv preprint arXiv:2501.12948, 2025. ↗
  14. N. Muennighoff, Z. Yang, W. Shi et al., “s1: Simple test-time scaling,” arXiv preprint arXiv:2501.19393, 2025. ↗
  15. Deutscher Bundestag, “Sitzverteilung im 20. deutschen bundestag,” 2025, accessed: 2025-03-02. [Online]. Available: https://www.bundestag.de/sitzverteilung ↗
  16. L. Osman and D. Sewell. (2022) Report - hugging face. [Online]. Available: https://research.contrary.com/reports/hugging-face ↗
  17. H. Naveed, A. U. Khan, S. Qiu et al., “A comprehensive overview of large language models,” arXiv preprint arXiv:2307.06435, 2023. ↗
  18. D. Hendrycks, C. Burns, S. Basart et al., “Measuring massive multitask language understanding,” arXiv preprint arXiv:2009.03300, 2020. ↗
  19. A. Mirza, N. Alampara, S. Kunchapu et al., “Are large language models superhuman chemists?” arXiv preprint arXiv:2404.01475, 2024. ↗
  20. J. Wei, X. Wang, D. Schuurmans et al., “Chain-of-thought prompting elicits reasoning in large language models,” in Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 24 824–24 837. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/2022/file/ 9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf ↗
  21. Bundeszentrale für politische Bildung. (2025) Wahl-o-mat. Accessed: 2025-03-02. [Online]. Available: https://www.bpb.de/ themen/wahl-o-mat/ ↗
  22. O. Schmidtke, “Migration as a building bloc of middle-class nation-building? the growing rift between germany’s centre-right and right-wing parties,” Journal of Ethnic and Migration Studies, vol. 50, no. 7, pp. 1677–1695, 2024. find
  23. K. Jäger, “When do party supporters abandon the party leader? the intraparty conflict of the alternative for germany,” Party Politics, vol. 27, no. 3, pp. 478–488, 2021. find
  24. E. Wax. (2023) European parliament set to grow by 15 meps in 2024. [Online]. Available: https://www.politico.eu/article/ eu-parliament-mep-election-2024/ ↗
  25. M. H. Susumu Shikano and P. W. Thurner, “Strategic voting under proportional representation: Threshold insurance in german elections,” West European Politics, vol. 32, no. 3, pp. 634–656, 2009. ↗
  26. M. Herrmann and F. U. Pappi, “Strategic voting in german constituencies,” Electoral Studies, vol. 27, no. 2, pp. 228–244, 2008. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/exler-2025-large-means-left-political-bias/