Elamur Research⌕
☰
home / papers / 2026 / Uncovering Ideological Bias in RAG with Lexical Multidimensional Analysis: A Case Study on COVID-19

Uncovering Ideological Bias in RAG with Lexical Multidimensional Analysis: A Case Study on COVID-19

Summary

Retrieval-Augmented Generation (RAG) is usually promoted as a way to reduce hallucinations, but the authors ask whether ideologically slanted retrieved documents can also pull an LLM's answers toward a particular framing. They build a knowledge base of 1,117 academic articles on COVID-19 treatments. The articles are split between controversial therapies (e.g., hydroxychloroquine, azithromycin) and treatments endorsed by health authorities. They then apply Lexical Multidimensional Analysis (LMDA), a corpus-linguistics method that runs factor analysis over lexical co-occurrence, to find three discourse dimensions with opposing poles. Four LLMs answer questions derived from these dimensions under three conditions: no context, a regular RAG prompt with framed texts, and an "enhanced" prompt that also includes LMDA descriptions. Alignment with the reference texts is measured by cosine similarity over semantic and lexical representations. Retrieved framed texts consistently moved answers toward their framing, and enhanced prompts strengthened this shift. The work shows that RAG can spread bias by accident and can also be used for deliberate discourse steering, which is a particular concern in healthcare.

Key points

  • Corpus sources:
    • Controversial texts came from Cureus, MedicosPelaVidaCovid19, HCQ for COVID-19 and Retraction Watch.
    • Endorsed texts came from the LitCovid Database (NIH).
  • Models tested: GPT-4o-mini, GPT-3.5, Gemini 2.0 and Qwen. Each answered with and without retrieval, under both regular and enhanced prompts.
  • Effect size with enhanced prompts: average semantic similarity rose from about 0.62–0.78 (no retrieval) to about 0.74–0.89 (RAG), depending on the dimension pole. Qwen was the exception, showing a slight drop with enhanced prompts.
  • Enhanced prompts alone are not enough: without retrieved texts, they had little effect and sometimes lowered similarity.
  • Statistical tests (ANOVA, LLM vs. RAG):
    • Semantic similarity differed significantly under both prompt types (F = 37.34 enhanced, 15.87 regular).
    • Lexical similarity was significant only with enhanced prompts (F = 21.10).
  • Limitations noted by the authors: similarity scores may partly reflect topical overlap or instruction-following rather than ideological alignment. The findings are also limited to the COVID-19 domain.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.starsem-conference.7

@inproceedings{salari2026uncovering,
  title = {{Uncovering Ideological Bias in RAG with Lexical Multidimensional Analysis: A Case Study on COVID-19}},
  author = {Salari, Elmira and Delfino, Maria Claudia Nunes and Amamou, Hazem and Souza, José Victor de and Kshirsagar, Shruti and Davoust, Alan and Avila, Anderson},
  booktitle = {Proceedings of the 15th Joint Conference on Lexical and Computational Semantics (*SEM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.starsem-conference.7},
  url = {https://doi.org/10.18653/v1/2026.starsem-conference.7}
}

References

31 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623. find
  2. T Berber Sardinha. 2014. On being american and brazilian in google books: A multi-dimensional perspective. In Looking at cultural shifts in English over time: A Multi-Dimensional perspective. American Association for Corpus Linguistics Conference, Flagstaff, AZ. find
  3. Tony Berber Sardinha. 2019. Using multidimensional analysis to detect representations of national identity, pages 231–258. Routledge, London. find
  4. Tony Berber Sardinha. 2020. A historical characterization of American and Brazilian cultures based on lexical representations. Corpora, 15(2):183–212. find
  5. Tony Berber Sardinha and Shannon Fitzsimmons-Doolan. 2025. Lexical Multidimensional Analysis. Cambridge University Press, Cambridge. find
  6. Maarten Buyl, Alexander Rogiers, Sander Noels, Guillaume Bied, Iris Dominguez-Catena, Edith Heiter, Iman Johary, Alexandru-Cristian Mara, Raphaël Romero, Jefrey Lijffijt, and 1 others. 2026. Large language models reflect the ideology of their creators. npj Artificial Intelligence, 2(1):7. find
  7. Banghao Chen, Zhaofeng Zhang, Nicolas Langrené, and Shengxin Zhu. 2025. Unleashing the potential of prompt engineering for large language models. Patterns. find
  8. Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2024a. Benchmarking large language models in retrieval-augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence. find our card
  9. Kai Chen, Zihao He, Jun Yan, Taiwei Shi, and Kristina Lerman. 2024b. How susceptible are large language models to ideological manipulation? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17140–17161. find
  10. Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. 2024. The power of noise: Redefining retrieval for rag systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Developm… find
  11. Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625–630. find
  12. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, and Sebastian Riedel. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing System… find
  13. Aleksandre Maskharashvili, Symon Stevens-Guille, Xintong Li, and Michael White. 2021. Neural methodius revisited: Do discourse relations help with pre-trained models too? In Proceedings of the 14th International Conference on Natural Language Generation, pages 12–23. find
  14. Anna Neumann, Elisabeth Kirsten, Muhammad Bilal Zafar, and Jatinder Singh. 2025. Position is power: System prompts as a mechanism of bias in large language models (llms). In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, pages 573–598. find
  15. Shannon Fitzsimmons-Doolan. 2014. Using lexical variables to identify language ideologies in a policy corpus. Corpora, 9(1):57–82. find
  16. Google. 2024. Gemini models documentation. https://ai.google.dev/gemini-api/docs/ models/gemini. Accessed: 2025-01-20. ↗
  17. Rem Hida, Masahiro Kaneko, and Naoaki Okazaki. 2024. Social bias evaluation for large language models requires prompt variations. arXiv preprint arXiv:2407.03129. find
  18. Manari Hirose and Masato Uchida. 2025. Decoding the mind of large language models: A quantitative evaluation of ideology and biases. arXiv preprint arXiv:2505.12183. — [arXiv:2505.12183](https://arxiv.org/abs/2505.12183) find
  19. Giwon Hong, Jeonghwan Kim, Junmo Kang, Sung-Hyon Myaeng, and Joyce Jiyoung Whang. 2024. Why so gullible? enhancing the robustness of retrieval-augmented models against counterfactual noise. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2474–2495. find
  20. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and 1 others. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems,… find our card
  21. Junsol Kim, James Evans, and Aaron Schein. 2025a. Linear representations of political perspective emerge in large language models. arXiv preprint arXiv:2503.02080. — [arXiv:2503.02080](https://arxiv.org/abs/2503.02080) find
  22. Taeyoun Kim, Jacob Mitchell Springer, Aditi Raghunathan, and Maarten Sap. 2025b. Mitigating bias in rag: Controlling the embedder. In Findings of the Association for Computational Linguistics: ACL 2025, pages 18999–19024. find
  23. Wei-Jen Ko and Junyi Jessy Li. 2020. Assessing discourse relations in language generation from gpt-2. arXiv preprint arXiv:2004.12506. find our card
  24. OpenAI. 2024. Openai api models documentation. https://platform.openai.com/docs/ models. Accessed: 2025-01-20. ↗
  25. Qwen Team. 2024. Qwen2.5: Large language models. https://github.com/QwenLM/Qwen2.5. Accessed: 2025-01-20. ↗
  26. John Sinclair. 1991. Corpus, Concordance, Collocation. Oxford University Press, Oxford. find
  27. Jonas Wallat, Maria Heuss, Maarten de Rijke, and Avishek Anand. 2025. Correctness is not faithfulness in retrieval augmented generation attributions. In Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR), pages 22–32. find
  28. Xuyang Wu, Shuowei Li, Hsin-Tai Wu, Zhiqiang Tao, and Yi Fang. 2025. Does rag introduce unfairness in llms? evaluating fairness in retrieval-augmented generation systems. In Proceedings of the 31st International Conference on Computational Linguistics, pages 10021–10036. find
  29. Shicheng Xu, Danyang Hou, Liang Pang, Jingcheng Deng, Jun Xu, Huawei Shen, and Xueqi Cheng. 2024. Invisible relevance bias: Text-image retrieval models prefer ai-generated images. In Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, page… find
  30. Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Gui, Ziran Jiang, Ziyu Jiang, and 1 others. 2025. Crag-comprehensive rag benchmark. Advances in Neural Information Processing Systems, 37:10470–10490. find
  31. Yingqi Zhao, Vasilis Efthymiou, Jyrki Nummenmaa, and Kostas Stefanidis. 2025. Refarag: Re-ranking for bias mitigation in retrieval-augmented generation. Communications in Computer and Information Science, 2676:516–530. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/salari-2026-uncovering-ideological-bias-rag-lexical/