Elamur Research⌕
☰
home / papers / 2026 / IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistance

IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistance

Paper · 2026 · Paul Röttger, Musashi Hinck, Valentin Hofmann et al.

Summary

LLMs are widely used as writing assistants, and a model that consistently favors one side of a political issue could gradually shift what users and their readers believe. Earlier bias evaluations mostly used small multiple-choice questionnaires, which look little like real usage and may not reflect how models behave in open-ended generation. To fill this gap, the authors built IssueBench. They mined first-turn prompts from five public logs of real user–LLM conversations and filtered them for political relevance with a GPT-4 classifier. They then clustered the relevant prompts and manually curated a set of issues, along with a separate pool of writing-task templates. Every template is crossed with every issue, giving about 2.5 million test prompts. Testing ten current open and closed models, they find stable and often one-sided stances, even on contested issues phrased neutrally. These biases are remarkably similar across models. On 20 US-relevant issues, every model sits much closer to Democrat than Republican voters. The resource is extensible and supports more ecologically valid bias auditing.

Key points

  • Scale and structure: IssueBench has 2,490,576 English prompts, built by combining 3,916 writing-assistance templates with 212 political issues. Each issue comes in neutral, positive and negative framings.
  • Source data and filtering: Prompts come from LMSYS-1m, ShareGPT, WildChat, HH-Online and PRISM, totaling 1.77m first-turn prompts.
    • Pre-filtering reduced these to 408.1k prompts.
    • A GPT-4 zero-shot relevance classifier (0.89 macro F1 on 1,000 hand-labeled prompts) kept 32.1k as relevant.
    • Clustering used SentenceTransformers, UMAP and HDBSCAN*, producing 396 clusters from which issues were curated.
  • Models tested: Ten LLMs from six families were evaluated. Prompting can push models toward most stances, but strong default stances resist steering. Larger models within a family hold consistent stances on more issues.
  • Partisan comparison: On 20 issues compared against iSideWith voter data, models were closer to Democrats on all but 3.
    • Average absolute distance was 0.27 from Democrats versus 0.77 from Republicans.
    • Example: support for same-sex marriage scored +0.91 to +0.95 for models, compared with +0.96 for Democrats and −0.24 for Republicans.
  • Availability: The dataset, annotation guidelines and code are released on GitHub and HuggingFace. The paper explains how to extend the resource to new issues, templates, tasks or languages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States353 works · 200,043 citations🔬Allen Institute for Artificial IntelligenceSeattle · United States275 works · 97,634 citations🎓University of OxfordOxford · United Kingdom104 works · 157,183 citations🎓Bocconi UniversityMilan · Italy28 works · 3,077 citations🏢IntelSanta Clara · United States15 works · 41,257 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Paul Röttger, Musashi Hinck, Valentin Hofmann, Kobi Hackenburg, Valentina Pyatkin, Faeze Brahman and Dirk Hovy. 2026. IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistance. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl.a.626

@article{rottger2026issuebench,
  title = {{IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistance}},
  author = {Paul Röttger, Musashi Hinck, Valentin Hofmann, Kobi Hackenburg, Valentina Pyatkin, Faeze Brahman and Dirk Hovy},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.1162/tacl.a.626},
  url = {https://doi.org/10.1162/tacl.a.626}
}

References

46 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Sh… find
  2. Yejin Bang, Delong Chen, Nayeon Lee, and Pascale Fung. 2024. Measuring political bias in large language models: What is said and how it is said. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11142–11159, Bangkok, Thailand. A…
  3. Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer… — The Llama 3 Herd of Models
  4. Maarten Buyl, Alexander Rogiers, Sander Noels, Guillaume Bied, Iris Dominguez-Catena, Edith Heiter, Iman Johary, Alexandru-Cristian Mara, Raphaël Romero, Jefrey Lijffijt, and Tijl De Bie. 2024. Large language models reflect the ideology of their creators. arXiv preprint arXiv:2410.18417. — [arXiv:2410.18417](https://arxiv.org/abs/2410.18417) find
  5. Ricardo J. G. B. Campello, Davoud Moulavi, and Jörg Sander. 2013. Density-based clustering based on hierarchical density estimates. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 160–172. Springer. https://doi.org/10.1007/978-3-642-37456-2_14 ↗
  6. Esin Durmus, Liane Lovitt, Alex Tamkin, Stuart Ritchie, Jack Clark, and Deep Ganguli. 2024a. Measuring the persuasiveness of language models. Anthropic.com - last accessed 06.09.2025. find
  7. Kai Chen, Zihao He, Jun Yan, Taiwei Shi, and Kristina Lerman. 2024. How susceptible are large language models to ideological manipulation? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17140–17161, Miami, Florida, USA. Association for Computational… find
  8. Esin Durmus, Karina Nguyen, Thomas Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. 2024b. Towards measuring the…
  9. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez,… find
  10. Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. 2023. From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair NLP models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume…
  11. Janell Fetterolf and Laura Clancy. 2024. Support for legal abortion is widespread in many places, especially in europe. Pew Research -last accessed 06.09.2025. 122(10):e2413443122. https://doi.org/10.1073/pnas.2413443122, PubMed: 40053360 — [doi:10.1073/pnas.2413443122](https://doi.org/10.1073/pnas.2413443122) ↗
  12. Jochen Hartmann, Jasper Schwenzow, and Maximilian Witte. 2023. The political ideology of conversational AI: Converging evidence on chatgpt’s pro-environmental, left-libertarian orientation. arXiv preprint arXiv: 2301.01768. https://doi.org/10.2139/ssrn.4316084 ↗
  13. Jillian Fisher, Ruth Elisabeth Appel, Chan Young Park, Yujin Potter, Liwei Jiang, Taylor Sorensen, Shangbin Feng, Yulia Tsvetkov, Margaret Roberts, Jennifer Pan, Dawn Song, and Yejin Choi. 2025. Position: Political neutrality in AI is impossible—but here is how to approximate it. In Forty-second Int… find
  14. Amir Kargaran, Ayyoob Imani, François Yvon, and Hinrich Schuetze. 2023. GlotLID: Language identification for low-resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6155–6218, Singapore. Association for Computational Linguistics. https://doi .org/10.1…
  15. Luciano Floridi. 2010. Information: A Very Short Introduction. Oxford University Press. https://doi.org/10.1093/actrade/9780199551378.001.0001 — [doi:10.1093/actrade](https://doi.org/10.1093/actrade) ↗
  16. Sasuke Fujimoto and Takemoto Kazuhiro. 2023. Revisiting the political biases of chat-GPT. Frontiers in Artificial Intelligence, 6. https://doi.org/10.3389/frai.2023 .1232003, PubMed: 37928447 — [doi:10.3389/frai.2023](https://doi.org/10.3389/frai.2023) ↗
  17. Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Michael Bean, Katerina Margatina, Rafael Mosquera, Juan Manuel Ciro, Max Bartolo, Adina Williams, He He, Bertie Vidgen, and Scott A. Hale. 2024. The PRISM alignment dataset: What participatory, representative and individualised human feed… find
  18. Suyash Fulay, William Brannon, Shrestha Mohanty, Cassandra Overney, Elinor Poole-Dayan, Deb Roy, and Jad Kabbara. 2024. On the relationship between truth and political bias in language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9004–9018,… find
  19. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Hao… find
  20. Josh A. Goldstein, Jason Chao, Shelby Grossman, Alex Stamos, and Michael Tomz. 2024. How persuasive is AI-generated propaganda? PNAS Nexus, 3(2):pgae034. https://doi.org/10.1093/pnasnexus/pgae034, PubMed: 38380055 — [doi:10.1093/pnasnexus/pgae034](https://doi.org/10.1093/pnasnexus/pgae034) ↗
  21. Kobi Hackenburg, Ben M. Tappin, Paul Röttger, Scott A. Hale, Jonathan Bright, and Helen Margetts. 2025. Scaling language model size yields diminishing returns for single-message political persuasion. Proceedings of the National Academy of Sciences, Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qius… find
  22. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish I… find
  23. Yifei Liu, Yuang Panwang, and Chao Gu. 2025. ‘‘Turning right’’? An experimental study on the political value shift in large language models. Humanities and Social Sciences Communications, 12(1):1–10. https://doi.org/10 .1057/s41599-025-04465-z Kristian Lum, Jacy Reese Anthis, Kevin Robinson, Chirag… find
  24. OpenAI. 2024. Model spec. OpenAI Website - last accessed 06.09.2025. find
  25. Siru Ouyang, Shuohang Wang, Yang Liu, Ming Zhong, Yizhu Jiao, Dan Iter, Reid Pryzant, Chenguang Zhu, Heng Ji, and Jiawei Han. 2023. The shifted and the overlooked: A task-oriented investigation of user-GPT interactions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language P… find
  26. Mantas Mazeika, Xuwang Yin, Rishub Tamirisa, Jaehyuk Lim, Bruce W. Lee, Richard Ren, Long Phan, Norman Mu, Adam Khoja, Oliver Zhang, et al. 2025. Utility engineering: Analyzing and controlling emergent value systems in AIs. arXiv preprint arXiv:2502.08640. — [arXiv:2502.08640](https://arxiv.org/abs/2502.08640) find
  27. Yujin Potter, Shiyang Lai, Junsol Kim, James Evans, and Dawn Song. 2024. Hidden per-suaders: LLMs’ political leaning and their influence on voters. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4244–4275, Miami, Florida, USA. Association for Computa… find
  28. Leland McInnes, John Healy, and Steve Astels. 2017. hdbscan: Hierarchical density based clustering. Journal of Open Source Software, 2(11):205. https://doi.org/10.21105/joss.00205 ↗
  29. Jared Moore, Tanvi Deshpande, and Diyi Yang. 2024. Are large language models consistent over value-laden questions? In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 15185–15221, Miami, Florida, USA. Association for Computational Linguistics. https://doi.org/10 .18653/v… find
  30. Qwen. 2024. Qwen2.5: A party of foundation models. Qwen Team Blog - last accessed 06.09.2025. find
  31. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the Fabio Motoki, Valdemar Pinho Neto, and Victor Rodrigues. 2023. More human than 9th International Jo…
  32. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? In International Conference on Machine Learning, pages 29971–30004. PMLR.
  33. Luca Rettenberger, Markus Reischl, and Mark Schutera. 2025. Assessing political bias in large language models. Journal of Computational Social Science, 8(2):1–17. https://doi.org/10.1007/s42001-025-00376-w Michael Saxon, Ari Holtzman, Peter West, William Yang Wang, and Naomi Saphra. 2024. Benchmark… ↗
  34. Reuters. 2024. Openai says chatgpt’s weekly users have grown to 200 million. Reuters -last accessed 06.09.2025. find
  35. Paul Röttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Kirk, Hinrich Schuetze, and Dirk Hovy. 2024. Political compass or spinning arrow? Towards more meaningful evaluations for values and opinions in large language models. In Proceedings of the 62nd Annual Meeting of the Associati…
  36. Amir Taubenfeld, Yaniv Dover, Roi Reichart, and Ariel Goldstein. 2024. Systematic biases in LLM simulations of debates. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 251–267, Miami, Florida, USA. Association for Computational Linguistics. https://do…
  37. Filip Trhlı́k and Pontus Stenetorp. 2024. Quantifying generative media bias with a corpus of real-world and generated news articles. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4420–4445, Miami, Florida, USA. Association for Computational Linguistics. https://doi.… ↗
  38. Paul Röttger, Bertie Vidgen, Dirk Hovy, and Janet Pierrehumbert. 2022. Two contrasting data annotation paradigms for subjective NLP tasks. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 175–190…
  39. Alec Tyson, Cary Funk, and Brian Kennedy. 2023. What the data says about Americans’ views of climate change. Pew Research - last accessed 06.09.2025. find
  40. Sean J. Westwood, Justin Grimmer, and Andrew B. Hall. 2025. Measuring perceived slant in large language models through user evaluations. Stanford Graduate Business School, Working Paper No. 4262. find
  41. David Rozado. 2023. The political biases of chatgpt. Social Sciences, 12(3):148. https://doi.org/10.3390/socsci12030148 — [doi:10.3390/socsci12030148](https://doi.org/10.3390/socsci12030148) ↗
  42. David Rozado. 2024. The political preferences of LLMs. PLOS ONE, 19(7):e0306621. https://doi.org/10.1371/journal .pone.0306621, PubMed: 39083484 — [doi:10.1371/journal](https://doi.org/10.1371/journal) ↗
  43. Dustin Wright, Arnav Arora, Nadav Borenstein, Srishti Yadav, Serge Belongie, and Isabelle Augenstein. 2024. LLM tropes: Revealing fine-grained values and opinions in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 17085–17112, Miami, Florida, US… find
  44. David Rozado. 2025. Measuring political preferences in ai systems: An integrative approach. arXiv preprint arXiv:2503.10649. — [arXiv:2503.10649](https://arxiv.org/abs/2503.10649) find
  45. Jérôme Rutinowski, Sven Franke, Jan Endendyk, Ina Dormuth, Moritz Roidl, and Markus Pauly. 2024. The self-perception and political biases of Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024. (inthe)wildchat: 570k chatGPT interaction logs in the wild. In The T… find
  46. Model gpt-4-0125-preview gpt-3.5-turbo-0125 T1 0.83 0.67 T2 0.88 0.80 T3 0.82 0.75 T4 0.86 0.78 T5 0.90 0.82 *Table 4: Relevance filtering performance across models and templates (T) as measured by macro F1 on 1,000 annotated model responses (§3.2). We use the best-performing setup, highlighted in b…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rottger-2026-issuebench-millions-realistic-prompts-measuring/