Elamur Research⌕
☰
home / papers / 2026 / News Credibility Assessment by LLMs and Humans: Implications for Political Bias

News Credibility Assessment by LLMs and Humans: Implications for Political Bias

Paper · 2026 · Pia Wenzel Neves, Charlott Jakob and Vera Schmitt

Summary

The paper asks whose viewpoint LLMs reflect when they judge news outlets. Earlier work compared model ratings only with expert ratings. Here the authors also compare them with public opinion split by political camp. They built a dataset of 47 outlets, 27 from the U.S. and 20 from Germany. Each outlet has expert labels from Media Bias/Fact Check (mbfc) and survey ratings by camp: AllSides and YouGov for the U.S., Medienkompass for Germany. Llama 3.3 70B, Mixtral 8x7B and GPT-OSS 120B rated each outlet's political bias and credibility. The prompts varied systematically in language and prompting style. Spearman correlations with Bonferroni correction measured agreement. All three models agreed closely with experts but less and less consistently with the public. For U.S. outlets, every model was closest to center-left views. German results were more mixed. The authors warn that leaning toward left camps may come from matching experts rather than from the models' own bias. Memorised training data may also inflate the agreement with experts.

Key points

  • Scale and setup: 22,560 prompts examined how the choice of model, the prompt language (English or German) and the way prompts were phrased influence how stable and correct the ratings are.
  • Survey camps: the U.S. surveys group respondents as Democrats, Independents and Republicans. Medienkompass uses seven camps, combining ideological orientation with attitudes toward mainstream media.
  • German credibility results: correlations with mbfc experts were Llama 0.925, GPT 0.917 and Mixtral 0.860. Correlations with individual political camps were lower and often not significant.
  • Possible contamination: GPT named Media Bias/Fact Check in 1.8% of its reasonings, and Llama in 0.04%. Mixtral showed no explicit trace of the source.
  • Language effect: German prompts succeeded less often than English ones (89.0% vs. 93.6%) and produced more refusals, especially with GPT.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬German Research Centre for Artificial IntelligenceKaiserslautern · Germany77 works · 29,479 citations🎓Technische Universität BerlinBerlin · Germany22 works · 957 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Social Sciences · 1,772 worksSafety Research · 543 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Bias, fairness & safety · 517 works

Cite

Cite as Pia Wenzel Neves, Charlott Jakob and Vera Schmitt. 2026. News Credibility Assessment by LLMs and Humans: Implications for Political Bias. The Proceedings for the 15th Workshop on Computational Approaches to Subjectivity, Sentiment Social Media Analysis (WASSA 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.wassa-1.15

@inproceedings{neves2026news,
  title = {{News Credibility Assessment by LLMs and Humans: Implications for Political Bias}},
  author = {Pia Wenzel Neves, Charlott Jakob and Vera Schmitt},
  booktitle = {The Proceedings for the 15th Workshop on Computational Approaches to Subjectivity, Sentiment Social Media Analysis (WASSA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.wassa-1.15},
  url = {https://doi.org/10.18653/v1/2026.wassa-1.15}
}

References

79 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dmitrii Aksenov, Peter Bourgonje, Karolina Zaczynska, Malte Ostendorff, Julian Moreno-Schneider, and Georg Rehm. 2021. Fine-grained classification of political bias in German news: A data set and initial experiments. In Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021), pages 121… find
  2. Dolores Albarracin, Julia Albarracin, Man-pui Sally Chan, and Kathleen Hall Jamieson. 2021. Creating Conspiracy Beliefs: How Our Thoughts Are Shaped. Cambridge University Press. S2CID 244413957. find
  3. Nouar Aldahoul, Hazem Ibrahim, Matteo Varvello, Aaron Kaufman, Talal Rahwan, and Yasir Zaki. 2025. Large language models are often politically extreme, usually ideologically inconsistent, and persuasive even in informational contexts. Preprint, arXiv:2505.04171. — [arXiv:2505.04171](https://arxiv.org/abs/2505.04171) find
  4. Bassel AlKhamissi, Mohamed ElNokrashy, Marwa AlKhamissi, and Mona Diab. 2024. Investigating cultural alignment of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), Volume 1: Long Papers, pages 12404–12422.
  5. AllSides Staff. 2024. Rating the bias of abc news: August-september 2024. Accessed: 2025-10-17. find
  6. AllSides Staff. 2025a. About allsides. Accessed: 2025-10-17. find
  7. AllSides Staff. 2025b. Media bias rating methods. Accessed: 2025-10-17. find
  8. R. A. Armstrong. 2014. When to use the bonferroni correction. Ophthalmic and Physiological Optics, 34(5):502–508. find
  9. Isabelle Augenstein, Timothy Baldwin, Meeyoung Cha, Tanmoy Chakraborty, Giovanni Luca Ciampaglia, David Corney, Renee DiResta, Emilio Ferrara, Scott Hale, Alon Halevy, Eduard Hovy, Heng Ji, Filippo Menczer, Ruben Miguez, Preslav Nakov, Dietram Scheufele, Shivam Sharma, and Giovanni Zagni. 2024. Fact… — Factuality Challenges in the Era of Large Language Models
  10. Yejin Bang, Danni Chen, Nayeon Lee, and Pascale Fung. 2024. Measuring political bias in large language models: What is said and how it is said. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11142–11159, Bangkok, Thailand. As…
  11. Donald A. Barclay. 2018. Fake News, Propaganda, and Plain Old Lies: How to Find Trustworthy Information in the Digital Age. Rowman & Littlefield, Lanham, MD. find
  12. Jan Batzner, Volker Stocker, Stefan Schmid, and Gjergji Kasneci. 2024. Germanpartiesqa: Benchmarking commercial large language models for political bias and sycophancy. Preprint, arXiv:2407.18008. — [arXiv:2407.18008](https://arxiv.org/abs/2407.18008) find
  13. Federico Errica, Giuseppe Siracusano, Davide Sanvito, and Roberto Bifulco. 2024. What did i do wrong? quantifying llms’ sensitivity and consistency to prompt engineering. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP). ArXiv:2406.12334v4.
  14. C. E. Bonferroni. 1936. Teoria statistica delle classi e calcolo delle probabilità, volume 8. find
  15. Lia Bozarth, Aparajita Saraf, and Ceren Budak. 2020. Higher ground? how groundtruth labeling impacts our understanding of fake news about the 2016 u.s. presidential nominees. In Proceedings of the International AAAI Conference on Web and Social Media, volume 14, pages 48–59. Despite the varied label… find
  16. David A. Broniatowski, Daniel Kerchner, Fouzia Farooq, Xiaolei Huang, Amelia M. Jamison, Mark Dredze, Sandra Crouse Quinn, and John W. Ayers. 2022. Twitter and facebook posts about covid-19 are less likely to spread misinformation compared to other health topics. PLOS ONE, 17(1):e0261768. find
  17. Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Amrit Singh Bedi, and Mengdi Wang. 2024. Maxmin-rlhf: Alignment with diverse human preferences. Preprint, arXiv:2402.08925. — [arXiv:2402.08925](https://arxiv.org/abs/2402.08925) find
  18. Davide Chicco, Alessandro Sichenze, and Giovanni Jurman. 2025. A simple guide to the use of student’s t-test, mann-whitney u test, chi-squared test, and kruskal-wallis test in biostatistics. BioData Mining, 18:56. find
  19. Jan Chołoniewski, Julian Sienkiewicz, Naum Dretnik, Gregor Leban, Mike Thelwall, and Janusz A. Hołyst. 2020. A calibrated measure to compare fluctuations of different entities across timescales. Scientific Reports, 10(1):20673. find
  20. Edward E. Cureton. 1956. Rank-biserial correlation. Psychometrika, 21:287–290. find
  21. Valentin Danchev. 2022. Reproducible data science with python: An open learning resource. Journal of Open Source Education, 5(56):156. Accessed: 2025-10-17. find
  22. Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker’s guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383–1392, Melbourne, A… find
  23. Rotem Dror, Lotem Peled-Cohen, Segev Shlomov, and Roi Reichart. 2020. Deep Significance, pages 35–50. Springer International Publishing, Cham. find
  24. Olive Jean Dunn. 1961. Multiple comparisons among means. Journal of the American Statistical Association, 56(293):52–64. find
  25. Mats Faulborn, Indira Sen, Max Pellert, Andreas Spitz, and David Garcia. 2025. Only a little to the left: A theory-grounded measure of political bias in large language models. Preprint, arXiv:2503.16148. find
  26. Daniel Funke and Alexios Mantzarlis. 2018. Here’s what to expect from fact-checking in 2019. Accessed: 2025-10-17. find
  27. John Gable, Julie Mastrine, and Rick Wytmar. 2020. Blind bias survey – allsides – august 2020. Accessed: 2025-10-17. find
  28. Gene V. Glass. 1965. A ranking variable analogue of biserial correlation: Implications for short-cut item analysis. Journal of Educational Measurement, 2(1):91– 95. find
  29. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  30. D. Hadar-Shoval, K. Asraf, Y. Mizrachi, Y. Haber, and Z. Elyoseph. 2024. Assessing the alignment of large language models with human values for mental health integration: Cross-sectional study using schwartz’s theory of basic values. JMIR Mental Health, 11:e55988. find
  31. Jutta Haider and Olof Sundin. 2022. Information literacy challenges in digital culture: conflicting engagements of trust and doubt. Information, Communication & Society, 25(8):1176–1191. find
  32. Kevin A. Hallgren. 2012. Computing inter-rater reliability for observational data: An overview and tutorial. Tutorials in Quantitative Methods for Psychology, 8(1):23–34. find
  33. Inter-Parliamentary Union. 2025. Germany - german bundestag election results (2025-02-23). Accessed: 2025-10-17. find
  34. Cloud IONOS. 2025. Ai model hub. https://cloud.ionos.com/managed/ai-model-hub. Accessed: 2025-10-18. ↗
  35. Charlott Jakob, David Harbecke, Patrick Parschan, Pia Wenzel Neves, and Vera Schmitt. 2025. Polbix: Detecting llms’ political bias in fact-checking through x-phemisms. arXiv preprint arXiv:2509.15335. find
  36. Charlott Jakob, Pia Wenzel, Salar Mohtaj, and Vera Schmitt. 2024. Augmented political leaning detection: Leveraging parliamentary speeches for classifying news articles. In Proceedings of the 4th Workshop on Computational Linguistics for the Political and Social Sciences: Long and short papers, page… find
  37. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… find
  38. Mahammed Kamruzzaman and Gene Louis Kim. 2025. Prompting techniques for reducing social bias in llms through system 1 and system 2 cognitive processes. Preprint, arXiv:2404.17218. find
  39. Meng Zhen Larsen, Michael R. Haupt, Tiana McMann, Raphael E. Cuomo, and Tim K. Mackey. 2023. The influence of news consumption habits and dispositional traits on trust in medical scientists. International Journal of Environmental Research and Public Health, 20(10):5842. find
  40. Noah Lee, Jiwoo Hong, and James Thorne. 2024. Evaluating the consistency of llm evaluators. Preprint, arXiv:2412.00543.
  41. João Alves Leite, Olga Razuvayevskaya, Kalina Bontcheva, and Carolina Scarton. 2025. Weakly supervised veracity classification with llm-predicted credibility signals. EPJ Data Science, 14(1):16. find
  42. Edoardo Loru, Jacopo Nudo, Niccolò Di Marco, Alessandro Santirocchi, Roberto Atzeni, Matteo Cinelli, Vincenzo Cestari, Clelia Rossi-Arnaud, and Walter Quattrociocchi. 2025. The simulation of judgment in llms. Proceedings of the National Academy of Sciences, 122(42). find
  43. Insa Maab, Edison Marrese-Taylor, Sebastian Padó, and Yutaka Matsuo. 2024. Media bias detection across families of language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2024), pages 4083–4098, Mexico City, Mexico.… find
  44. Media Bias/Fact Check. 2025a. About. https://mediabiasfactcheck.com/about/. Accessed: 2025-10-17. ↗
  45. Media Bias/Fact Check. 2025b. Methodology. Accessed: 2025-10-17. find
  46. Medienkompass.org. 2025a. Deutsche medienlandschaft. Accessed: 17. find
  47. Medienkompass.org. 2025b. Medienkompass – eine persönliche umfrage zur aktuellen medienlandschaft. Accessed: 2025-10-17. find
  48. Medienkompass.org. 2025c. Über medienkompass.org. Accessed: 17. find
  49. Fabio Motoki, Victor Pinho Neto, and Victor Rodrigues. 2024. More human than human: Measuring chatgpt political bias. Public Choice, 198(1-2):3–23. find
  50. L. Myers and M. J. Sirois. 2004. Spearman correlation coefficients, differences between. In Encyclopedia of Statistical Sciences, volume 12. Wiley. find
  51. OpenAI, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, and 107 others. 2025. gpt-oss-120b gpt-oss-20b…
  52. Taylor Orth and Carl Bialik. 2025. Trust in media 2025: Which news sources americans use and trust. Accessed: 2025-10-17. find
  53. Tai-Quan Peng, Kaiqi Yang, Sanguk Lee, Hang Li, Yucheng Chu, Yuping Lin, and Hui Liu. 2025. Beyond partisan leaning: A comparative analysis of political bias in large language models. Preprint, arXiv:2412.16746. — [arXiv:2412.16746](https://arxiv.org/abs/2412.16746) find
  54. Dorian Quelle and Alexandre Bovet. 2024. The perils and promises of fact-checking with large language models. Frontiers in Artificial Intelligence, Volume 7 - 2024. find
  55. Luca Rettenberger, Markus Reischl, and Mark Schutera. 2024. Assessing political bias in large language models. Preprint, arXiv:2405.13041. find
  56. R. Rogers. 2021. Marginalizing the mainstream: How social media privilege political information. Frontiers in Big Data, 4:689036. find
  57. George-Cristinel Rotaru, Sorin Anagnoste, and Marian Oancea. 2024. How artificial intelligence can influence elections: Analyzing the large language models (llms) political bias. Proceedings of the International Conference on Business Excellence, 18:1882–1891. find
  58. David Rozado. 2024. The political preferences of llms. PLOS ONE. find
  59. Paul Röttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Rose Kirk, Hinrich Schütze, and Dirk Hovy. 2024. Political compass or spinning arrow? towards more meaningful evaluations for values and opinions in large language models. Preprint, arXiv:2402.16786.
  60. Geir Kjetil Sandve, Anton Nekrutenko, James Taylor, and Eivind Hovig. 2013. Ten simple rules for reproducible computational research. PLoS Computational Biology, 9(10):e1003285. Accessed: 2025-10-17. find
  61. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Celestine Lee, Percy Liang, and Tatsunori B. Hashimoto. 2023. Whose opinions do language models reflect? In Proceedings of the 40th International Conference on Machine Learning (ICML), volume 202, pages 29971–30004. PMLR.
  62. Nikhil Sharma, Kenton Murray, and Ziang Xiao. 2025. Faux polyglot: A study on information disparity in multilingual large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volum… find
  63. C. Spearman. 1904. The proof and measurement of association between two things. The American Journal of Psychology, 15(1):72–101. find
  64. S. S. Stevens. 1946. On the theory of scales of measurement. Science, 103(2684):677–680. find
  65. Rickard Stureborg, Dimitris Alikaniotis, and Yoshi Suhara. 2024. Large language models are inconsistent and biased evaluators. Preprint, arXiv:2405.01724.
  66. Yilin Tao, Oskar Viberg, Ryan S. Baker, and René F. Kizilcec. 2024. Cultural bias and cultural alignment of large language models. PNAS Nexus, 3(9):pgae346. find
  67. Kai-Cheng Yang and Filippo Menczer. 2025. Accuracy and political bias of news source credibility ratings by large language models. In Proceedings of the 17th ACM Web Science Conference 2025. Association for Computing Machinery. find
  68. YouGov. 2025a. About yougov. Accessed: 2025-10-17. find
  69. YouGov. 2025b. Panel methodology. Accessed: 2025-10-17. find
  70. YouGov. 2025c. Trust in media poll results. Accessed: 2025-10-17. find
  71. J. H. Zar. 1972. Significance testing of the spearman rank correlation coefficient. Journal of the American Statistical Association, 67(339):578–580. find
  72. Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Chen Xu, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. 2025. Siren’s song in the ai ocean: A survey on hallucination in large language models. Preprint, arXiv:2309.012…
  73. Jiaqi Zhuo, Shuai Zhang, Xinyu Fang, Haodong Duan, Dahua Lin, and Kai Chen. 2024. Prosa: Assessing and understanding the prompt sensitivity of llms. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1950–1976. Association for Computational Linguistics. find
  74. Maciej Tomczak and Ewa Tomczak-Łukaszewska. 2014. The need to report effect size estimates revisited. an overview of some recommended measures of effect size. TRENDS in Sport Sciences, 21(1):19–25. find
  75. Hong T Vu, Liefu Jiang, Lourdes M Cueva Chacón, Martin J Riedl, Duc V Tran, and Piotr S Bobkowski. 2019. What influences media effects on public perception? a cross-national study of comparative agenda setting. International Communication Gazette, 81(6-8):580– 601. find
  76. Li Wang, Xi Chen, XiangWen Deng, Hao Wen, MingKe You, WeiZhi Liu, Qi Li, and Jian Li. 2024. Prompt engineering in consistency and reliability with the evidence-based guideline for large language models. npj Digital Medicine, 7:41. Accessed: 2025-10-17. find
  77. Qihan Wang, Shidong Pan, Tal Linzen, and Emily Black. 2025. Multilingual prompting for improving llm generation diversity. Preprint, arXiv:2505.15229. find
  78. Galen Weld, Maria Glenski, and Tim Althoff. 2021. Political bias and factualness in news sharing across more than 100,000 online communities. In Proceedings of the International AAAI Conference on Web and Social Media, volume 15, pages 796–807. S2CID 231942492, retrieved 8 June 2023. find
  79. Tamar Wilner. 2018. We can probably measure media bias. but do we want to? Accessed: 2025-10-17. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/neves-2026-news-credibility-assessment-llms-humans/