Elamur Research⌕
☰
home / papers / 2026 / Framing Political Bias in Multilingual LLMs Across Pakistani Languages

Framing Political Bias in Multilingual LLMs Across Pakistani Languages

Paper · 2026 · Afrozah Nadeem, Mark Dras and Usman Naseem

Summary

Most audits of political leaning in LLMs use English or other well-resourced Western languages. This study looks at Pakistan instead, where language is closely bound up with ethnic, religious and regional politics. The authors built a culturally adapted version of the Political Compass Test (PCT). Its 62 statements were translated with human verification into Urdu, Punjabi, Sindhi, Pashto and Balochi, and linked to 11 locally contested themes such as blasphemy laws, the death penalty and language policy. They then evaluated 13 open and closed LLMs.

Each response is placed on economic and social axes. A framing layer then examines how the answer is phrased, using Boydstun's frame taxonomy, named entity recognition and lexical polarity.

Overall, the models lean left and liberal, which fits their largely Western training data. In the regional languages, however, they shift toward more authoritarian positions, so the language of the prompt changes the ideology the model expresses. This matters because speakers of low-resource languages may receive systematically different political framing, which argues for culturally grounded bias audits.

Key points

  • Coverage: 13 LLMs, five Pakistani languages and 11 socio-political topics. Together these languages are spoken by over 80% of Pakistan's population.
  • Stance scoring: PCT answers on a four-level Likert scale are mapped to coordinates in [−10, 10], with S eco giving the position on the economic dimension and S soc the position on the social one.
  • Language-dependent shifts: GPT models land in the libertarian-left quadrant for English but move to authoritarian-left for Pakistani languages. DeepSeek stays left-libertarian.
  • Frame classification: GPT-3.5-turbo labelled the frames, and a human check of a 20% sample gave κ > 0.7.
  • Code and data: released at github.com/Afx-Msh/PoliticalBiasEvaluation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksMultilingual & cross-lingual NLP · 828 worksBias, fairness & safety · 517 works

Cite

Cite as Afrozah Nadeem, Mark Dras and Usman Naseem. 2026. Framing Political Bias in Multilingual LLMs Across Pakistani Languages. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1689

@inproceedings{nadeem2026framing,
  title = {{Framing Political Bias in Multilingual LLMs Across Pakistani Languages}},
  author = {Afrozah Nadeem, Mark Dras and Usman Naseem},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1689},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1689}
}

References

51 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Fozia Abbas and Saadiyah Javed Bidin. 2022. Language policy and planning in pakistan: A critical analysis of the language planning and policy (lpp) in pakistan and its impact on indigenous languages of pakistan. Eurasian Journal of Applied Linguistics, pages 85–96. find
  2. Suhaib Abdurahman, Mohammad Atari, Farzan Karimi-Malekabadi, Mona J. Xue, Jackson Trager, Peter S. Park, Preni Golazizian, Ali Omrani, and Morteza Dehghani. 2024. Perils and opportunities in using large language models in psychological research. PNAS Nexus, 3(7):pgae245. Advance access publication 1… find
  3. Samee Arif, Sualeha Farid, Awais Athar, and Agha Ali Raza. 2024. UQA: Corpus for Urdu Question Answering. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 17237–17244, Torino, Italia. ELRA and ICCL. find
  4. Ramy Baly, Giovanni Da San Martino, James Glass, and Preslav Nakov. 2020. We Can Detect Your Bias: Predicting the Political Ideology of News Articles. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4982–4991, Online. Association for Computati…
  5. Yejin Bang, Delong Chen, Nayeon Lee, and Pascale Fung. 2024. Measuring Political Bias in Large Language Models: What Is Said and How It Is Said. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11142–11159, Bangkok, Thailand. A…
  6. Yejin Bang, Nayeon Lee, Etsuko Ishii, Andrea Madotto, and Pascale Fung. 2021. Assessing Political Prudence of Open-domain Chatbots. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 548–555, Singapore and Online. Association for Computational Li… find
  7. Ehsan Barkhordar, Surendrabikram Thapa, Ashwarya Maratha, and Usman Naseem. 2024. Why the Unexpected? Dissecting the Political and Economic Bias in Persian Small and Large Language Models. ELRA Language Resource Association, pages 410–420. CC BY-NC 4.0. find
  8. Su Lin Blodgett, Solon Barocas, Hal Daumé Iii, and Hanna Wallach. 2020. Language (Technology) is Power: A Critical Survey of “Bias” in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454–5476, Online. Association for Computational Linguistics.
  9. Amber E Boydstun, Dallas Card, Justin H Gross, Philip Resnik, and Noah A Smith. 2014. Tracking the Development of Media Frames within and across Policy Issues. Manuscript. find
  10. Tanise Ceron, Neele Falk, Ana Barić, Dmitry Nikolaev, and Sebastian Padó. 2024. Beyond prompt brittleness: Evaluating the reliability and consistency of political worldviews in llms. Transactions of the Association for Computational Linguistics, 12:1378–1400.
  11. Wei-Fan Chen, Khalid Al Khatib, Henning Wachsmuth, and Benno Stein. 2020. Analyzing Political Bias and Unfairness in News Articles at Different Levels of Granularity. In Proceedings of the Fourth Workshop on Natural Language Processing and Computational Social Science, pages 149–154, Online. Associa… find
  12. Dorottya Demszky, Nikhil Garg, Rob Voigt, James Zou, Jesse Shapiro, Matthew Gentzkow, and Dan Jurafsky. 2019. Analyzing Polarization in Social Media: Method and Application to Tweets on 21 Mass Shootings. In Proceedings of the 2019 Conference of the North, pages 2970–3005, Minneapolis, Minnesota. As… find
  13. Asma Fazal. 2022. Ethical Issues in Conducting Cross-Cultural Research in Low-Income Countries: A Pakistani Perspective. ABR, 14(2):151–168. find
  14. Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. 2023. From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models. Association for Computational Linguistics.
  15. Genevieve Gorrell, Elena Kochkina, Arkaitz Zubiaga, Maria Liakata, and Kalina Bontcheva. 2019. SemEval-2019 task 7: RumourEval: Determining rumour veracity and support for rumours. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 845–854.
  16. Felix Hamborg. 2020. Media Bias, the Social Sciences, and NLP: Automating Frame Analyses to Identify Bias by Word Choice and Labeling. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 79–87, Online. Association for Computati… find
  17. James Christopher Harman. 2018. The Psychological Spectrum: Political Orientation and its Origins in Perception and Culture. Undergraduate Journal of Politics and International Relations, 1(1):2. find
  18. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint. ArXiv:1810.04805 [cs].
  19. Jochen Hartmann, Jasper Schwenzow, and Maximilian Witte. 2023. The political ideology of conversational AI: Converging evidence on ChatGPT’s pro-environmental, left-libertarian orientation. SSRN Journal. find
  20. Toqeer Ehsan and Thamar Solorio. 2025. Enhancing NER Performance in Low-Resource Pakistani Languages using Cross-Lingual Data Augmentation. In Proceedings of the Tenth Workshop on Noisy and User-generated Text, pages 117–132, Albuquerque, New Mexico, USA. Association for Computational Linguistics. find
  21. Chadi Helwe, Oana Balalau, and Davide Ceolin. 2025. Navigating the political compass: Evaluating multilingual llms across languages and nationalities. In Findings of the Association for Computational Linguistics: ACL 2025, pages 17179–17204, Bangkok, Thailand. Association for Computational Linguisti… find
  22. Waqas Ejaz, Muhammad Ittefaq, and Sadia Jamil. 2023. Politics triumphs: a topic modeling approach of analyzing news media coverage of climate change in Pakistan. JCOM, 22(01):A02. find
  23. Lisa Fan, Marshall White, Eva Sharma, Ruisi Su, Prafulla Kumar Choubey, Ruihong Huang, and Lu Wang. 2019. In Plain Sight: Media Bias Through the Lens of Factual Reporting. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer… find
  24. Mats Faulborn, Indira Sen, Max Pellert, Andreas Spitz, and David García. 2025. Only a little to the left: A theory-grounded measure of political bias in large Sarmad Hussain. 2004. Urdu Localization Project. In Proceedings of the Workshop on Computational Approaches to Arabic Script-based Languages,… find
  25. Kristen Johnson and Dan Goldwasser. 2016. Identifying Stance by Analyzing Political Discourse on Twitter. In Proceedings of the First Workshop on NLP and Computational Social Science, pages 66–75, Austin, Texas. Association for Computational Linguistics. find
  26. Sachin Kumar, Vidhisha Balachandran, Lucille Njoo, Antonios Anastasopoulos, and Yulia Tsvetkov. 2023. Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Lingui…
  27. Nayeon Lee, Yejin Bang, Tiezheng Yu, Andrea Madotto, and Pascale Fung. 2022. NeuS: Neutral Multi-News Summarization for Mitigating Framing Bias. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 31… find
  28. Ruibo Liu, Chenyan Jia, Jason Wei, Guangxuan Xu, Lili Wang, and Soroush Vosoughi. 2021. Mitigating Political Bias in Language Models through Reinforced Calibration. AAAI, 35(17):14857–14866. find
  29. Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo, Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. 2024. Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models’ Alignment. arXiv preprint. ArXiv:2308.05374 [cs].
  30. Saif M. Mohammad, Svetlana Kiritchenko, and Xiaodan Zhu. 2016. SemEval-2016 task 6: Detecting stance in tweets. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 31–41.
  31. Djamel Mostefa, Khalid Choukri, Sylvie Brunessaux, and Karim Boudahmane. 2012. New language resources for the Pashto language. find
  32. Fabio Motoki, Valdemar Pinho Neto, and Victor Rodrigues. 2024. More human than human: measuring ChatGPT political bias. Public Choice, 198(1-2):3–23. find
  33. Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long…
  34. Tariq Rahman. 1996. Language and Politics in Pakistan. Oxford University Press, Karachi, Pakistan. find
  35. Tariq Rahman. 2011. From Hindi to Urdu: A Social and Political History. Oxford University Press, Karachi, Pakistan. find
  36. Paul Röttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Rose Kirk, Hinrich Schütze, and Dirk Hovy. 2024. Political compass or spinning arrow? towards more meaningful evaluations for values and opinions in large language models. In Proceedings of the 62nd Annual Meeting of the Associ…
  37. Shamik Roy and Dan Goldwasser. 2020. Weakly Supervised Learning of Nuanced Frames for Analyzing Polarization in News Media. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7698–7716, Online. Association for Computational Linguistics. find
  38. David Rozado. 2024. The political preferences of LLMs. PLoS ONE, 19(7):e0306621. find
  39. Diego Saez-Trumper, Carlos Castillo, and Mounia Lalmas. 2013. Social media news communities: gatekeeping, coverage, and statement bias. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management, pages 1679–1684, San Francisco California USA. ACM. find
  40. Patrick Schramowski, Cigdem Turan, Nico Andersen, Constantin A. Rothkopf, and Kristian Kersting. 2022. Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Do. arXiv preprint. ArXiv:2103.11790 [cs]. find
  41. Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. 2021. Societal Biases in Language Generation: Progress and Challenges. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (V…
  42. Eric Michael Smith, Melissa Hall, Melanie Kambadur, Eleonora Presani, and Adina Williams. 2022. “I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9180–92…
  43. Maximilian Spliethöver, Maximilian Keiff, and Henning Wachsmuth. 2022. No Word Embedding Model Is Perfect: Evaluating the Representation Accuracy for Social Bias in the Media. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2081–2093, Abu Dhabi, United Arab Emirates.… find
  44. Tianxiang Sun, Junliang He, Xipeng Qiu, and Xuanjing Huang. 2022. BERTScore is Unfair: On Social Bias in Language Model-Based Metrics for Text Generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3726–3739, Abu Dhabi, United Arab Emirates. Asso… find
  45. Munief Hassan Tahir, Sana Shams, Layba Fiaz, Farah Adeeba, and Sarmad Hussain. 2025. Benchmarking the Performance of Pre-trained LLMs across Urdu NLP Tasks. In Proceedings of the First Workshop on Challenges in Processing South Asian Languages (CHiPSAL 2025), pages 17–34, Abu Dhabi, UAE. Internation… find
  46. Surendrabikram Thapa, Ashwarya Maratha, Khan Md Hasib, Mehwish Nasim, and Usman Naseem. 2023. Assessing political inclination of bangla language models. In Proceedings of the First Workshop on Bangla Language Processing (BLP-2023), pages 62–71, Singapore. Association for Computational Linguistics. find
  47. Surendrabikram Thapa, Kritesh Rauniyar, Ehsan Barkhordar, Hariram Veeramani, and Usman Naseem. 2024. Which Side Are You On? Investigating Politico-Economic Bias in Nepali Language Models. Proceedings of the 22nd Annual Workshop of the Australasian Language Technology Association (ALTA 2024), pages 1… find
  48. Tariq Umrani and Faraz Ali Bughio. 2020. Language politics and role of english in pakistan. ARIEL: An International Journal of Research in English Language and Literature, pages 114–124. find
  49. Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. 2023. Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias. arXiv preprint. ArXiv:2306.15895 [cs].
  50. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv preprint. ArXiv:2306.05685 [cs].
  51. Caleb Ziems and Diyi Yang. 2021. To Protect and To Serve? Analyzing Entity-Centric Framing of Police Violence. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 957–976, Punta Cana, Dominican Republic. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nadeem-2026-framing-political-bias-multilingual-llms/