Elamur Research⌕
☰
home / papers / 2025 / Investigating Bias in LLM-Based Bias Detection: Disparities between LLMs and Human Perception

Investigating Bias in LLM-Based Bias Detection: Disparities between LLMs and Human Perception

Paper · 2025 · Luyang Lin, Lingzhi Wang, Jinsong Guo et al.

Summary

Large language models are increasingly used to label the political slant of news, but they may carry their own political leanings into that task. This paper examines the detector itself rather than the media content. It compares LLM judgments with human-annotated ideology labels on two political bias benchmarks, FlipBias and ABP. The analysis asks four questions: whether LLMs are politically biased, whether that bias is consistent across topics, how it can be reduced, and whether different models lean the same way. To probe bias, the authors use two tasks, prediction and text continuation, along with topic-level analysis built on a bias tendency index (BTI). Vanilla GPT-3.5 shows a left-leaning tilt: it often labels left-leaning articles as centrist, yet classifies right-leaning articles comparatively well. Prompt-based debiasing, especially debiasing statements, pulls BTI toward neutral but costs some accuracy. Fine-tuning improves classification scores but can deepen the skew. Since LLMs are becoming standard tools for media analysis, the work shows that their outputs need auditing against human perception before they are trusted.

Key points

  • Data: FlipBias contains 3,066 articles (1,022 left/center/right triples about the same event), and ABP contains 37,554 articles.
  • Baseline performance: Vanilla gpt-3.5-turbo-0613 reaches only 26.2 F1 on FlipBias. It predicts center for 60.5% of truly left articles, a much higher rate than for right articles.
  • Fine-tuning setups: gpt-3.5-turbo was tuned on 300 examples in three label mixes (L-FT, LC-FT, LCR-FT), and BERT served as a comparison baseline. LCR-FT raised BiF1 to about 77.8 but shifted the BTI distribution.
  • Prompt-based debiasing: Techniques such as bias label explanation, few-shot prompting and Debiasing Statements (DS) were tested. DS was the simplest and kept topic-level BTI close to zero.
  • Cross-model comparison: Llama-2-7B-Chat, Vicuna-7B-v1.5, Mistral-7B-v0.1 and gpt-4-0125-preview all show bias, and Mistral is the only one leaning right. LLaMa2 and Mistral refused or gave off-task output on roughly 20% of test items, and the authors found that accuracy did not track the degree of bias.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Harbin Institute of TechnologyHarbin · China158 works · 28,841 citations🎓University College LondonLondon · United Kingdom88 works · 19,432 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSafety Research · 543 worksSociology and Political Science · 577 works

Cite

Cite as Luyang Lin, Lingzhi Wang, Jinsong Guo and Kam-Fai Wong. 2025. Investigating Bias in LLM-Based Bias Detection: Disparities between LLMs and Human Perception. Proceedings of the 31st International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{lin2025investigating,
  title = {{Investigating Bias in LLM-Based Bias Detection: Disparities between LLMs and Human Perception}},
  author = {Luyang Lin, Lingzhi Wang, Jinsong Guo and Kam-Fai Wong},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.709/}
}

References

38 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ramy Baly, Giovanni Da San Martino, James Glass, and Preslav Nakov. 2020. We can detect your bias: Predicting the political ideology of news articles. arXiv preprint arXiv:2010.05338.
  2. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623. find
  3. Camiel J Beukeboom and Christian Burgers. 2019. How stereotypes are shared through language: a review and introduction of the aocial categories and stereotypes communication (scsc) framework. Review of Communication Research, 7:1–37. find
  4. Su Lin Blodgett, Lisa Green, and Brendan O’Connor. 2016. Demographic dialectal variation in social media: A case study of african-american english. arXiv preprint arXiv:1608.08868.
  5. Wei-Fan Chen, Henning Wachsmuth, Khalid Al-Khatib, and Benno Stein. 2018. Learning to flip the bias of news headlines. In Proceedings of the 11th International Conference on Natural Language Generation, pages 79–88, Tilburg University, The Netherlands. Association for Computational Linguistics. find
  6. Lina Conti and Guillaume Wisniewski. 2023. Using artificial French data to understand the emergence of gender bias in transformer language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10362– 10371, Singapore. Association for Computational L… find
  7. This work is subject to limitations in two main aspects: (1) Limited Focus on LLM Bias in Media Bias Prediction: The scope of bias analysis is constrained by the availability of three-way (left-, center-, and right-leaning) labeled data. Our study David Esiobu, Xiaoqing Tan, Saghar Hosseini, Megan U… find
  8. Xiao Fang, Shangkun Che, Minjia Mao, Hongzhe Zhang, Ming Zhao, and Xiaohang Zhao. 2023. Bias of ai-generated content: an examination of news produced by large language models. arXiv preprint arXiv:2309.09825. — [arXiv:2309.09825](https://arxiv.org/abs/2309.09825) find
  9. Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. 2023. Bias and fairness in large language models: A survey. arXiv preprint arXiv:2309.00770.
  10. Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas Liao, Kamilė Lukošiūtė, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, et al. 2023. The capacity for moral self-correction in large language models. arXiv preprint arXiv:2302.07459.
  11. Aparna Garimella, Rada Mihalcea, and Akhash Amarnath. 2022. Demographic-aware language model fine-tuning as a bias mitigation technique. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Na… find
  12. Michael Gira, Ruisu Zhang, and Kangwook Lee. 2022. Debiasing pre-trained language models via efficient fine-tuning. In Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion, pages 59–69. find
  13. Luyang Lin, Lingzhi Wang, Jinsong Guo, Jing Li, and Kam-Fai Wong. Inditag: An online media bias analysis and annotation system using fine-grained bias indicators. find
  14. Luyang Lin, Lingzhi Wang, Xiaoyan Zhao, Jing Li, and Kam-Fai Wong. 2024. Indivec: An exploration of leveraging large language models for media bias detection with fine-grained bias indicators. arXiv preprint arXiv:2402.00345. find
  15. Hui Liu, Wenya Wang, Haoru Li, and Haoliang Li. 2024. Teller: A trustworthy framework for explainable, generalizable and controllable fake news detection. arXiv preprint arXiv:2402.07776. find
  16. Ruibo Liu, Chenyan Jia, Jason Wei, Guangxuan Xu, Lili Wang, and Soroush Vosoughi. 2021. Mitigating political bias in language models through reinforced calibration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14857– 14866. find
  17. Yujian Liu, Xinliang Frederick Zhang, David Wegsman, Nick Beauchamp, and Lu Wang. 2022. Politics: pre-training with same-story article comparison for ideology prediction and stance detection. arXiv preprint arXiv:2205.00619.
  18. Fabio Motoki, Valdemar Pinho Neto, and Victor Rodrigues. 2024. More human than human: Measuring chatgpt political bias. Public Choice, 198(1):3–23. find
  19. Gustavo Gonçalves and Emma Strubell. 2023. Understanding the effect of model compression on social bias in large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2663–2675. find
  20. Ruoling Peng, Kang Liu, Po Yang, Zhipeng Yuan, and Shunbao Li. 2023. Embedding-based retrieval with llm for effective agriculture information extracting from unstructured data. arXiv preprint arXiv:2308.03107. — [arXiv:2308.03107](https://arxiv.org/abs/2308.03107) find
  21. Rishav Hada, Agrima Seth, Harshita Diddee, and Kalika Bali. 2023. “fifty shades of bias”: Normative ratings of gender bias in gpt generated english text. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1862–1876.
  22. Rebecca Qian, Candace Ross, Jude Fernandes, Eric Smith, Douwe Kiela, and Adina Williams. 2022. Perturbation augmentation for fairer nlp. arXiv preprint arXiv:2205.12586. find
  23. Mohit Iyyer, Peter Enns, Jordan Boyd-Graber, and Philip Resnik. 2014. Political ideology detection using recursive neural networks. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1113–1122.
  24. Przemyslaw Joniak and Akiko Aizawa. 2022. Gender biases and where to find them: Exploring gender bias in pre-trained transformer-based language models using movement pruning. arXiv preprint arXiv:2207.02463. find
  25. Masahiro Kaneko, Danushka Bollegala, Naoaki Okazaki, and Timothy Baldwin. 2024. Evaluating gender bias in large language models via chain-of-thought prompting. arXiv preprint arXiv:2401.15585. — [arXiv:2401.15585](https://arxiv.org/abs/2401.15585) find
  26. David Rozado. 2023. The political biases of chatgpt. Social Sciences, 12(3):148. find
  27. David Rozado. 2024. The political preferences of llms. arXiv preprint arXiv:2402.01789. — [arXiv:2402.01789](https://arxiv.org/abs/2402.01789) find
  28. Eric Michael Smith, Melissa Hall, Melanie Kambadur, Eleonora Presani, and Adina Williams. 2022. “i’m sorry to hear that”: Finding new biases in language models with a holistic descriptor dataset. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9180–92…
  29. Amir Taubenfeld, Yaniv Dover, Roi Reichart, and Ariel Goldstein. 2024. Systematic biases in llm simulations of debates. arXiv preprint arXiv:2402.04049.
  30. Ewoenam Kwaku Tokpo and Toon Calders. 2022. Text style transfer for bias mitigation using masked language modeling. arXiv preprint arXiv:2201.08643. find
  31. Aleksandra Urman and Mykola Makhortykh. 2023. The silence of the llms: Cross-lingual analysis of political bias and false information prevalence in chatgpt, google bard, and bing chat. find
  32. Pranav Narayanan Venkit, Sanjana Gautam, Ruchi Panchanadikar, Ting-Hao’Kenneth’ Huang, and Shomir Wilson. 2023. Nationality bias in text generation. arXiv preprint arXiv:2302.02463. find
  33. Thiemo Wambsganss, Xiaotian Su, Vinitra Swamy, Seyed Neshaei, Roman Rietsche, and Tanja Käser. 2023a. Unraveling downstream gender bias from large language models: A study on AI educational writing assistance. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10275–1028… find
  34. Thiemo Wambsganss, Xiaotian Su, Vinitra Swamy, Seyed Parsa Neshaei, Roman Rietsche, and Tanja Käser. 2023b. Unraveling downstream gender bias from large language models: A study on ai educational writing assistance. arXiv preprint arXiv:2311.03311. find
  35. Fei Wang, Wenjie Mo, Yiwei Wang, Wenxuan Zhou, and Muhao Chen. 2023. A causal view of entity bias in (large) language models. arXiv preprint arXiv:2305.14695. find
  36. Tae Yano, Philip Resnik, and Noah A Smith. 2010. Shedding (a thousand points of) light on biased language. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, pages 152–158.
  37. Bei Yu, Stefan Kaufmann, and Daniel Diermeier. 2008. Classifying party affiliation from political speech. Journal of Information Technology & Politics, 5(1):33–48. find
  38. Abdelrahman Zayed, Prasanna Parthasarathi, Gonçalo Mordido, Hamid Palangi, Samira Shabanian, and Sarath Chandar. 2023. Deep learning on a healthy data diet: Finding important examples for fairness. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 14593–14601. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lin-2025-investigating-bias-llm-based-bias/