Elamur Research⌕
☰
home / papers / 2026 / Persona-Aware Evaluation of Cognitive Bias in LLMs: From Benchmark to Applied Decision-Making

Persona-Aware Evaluation of Cognitive Bias in LLMs: From Benchmark to Applied Decision-Making

Summary

The paper asks whether large language models show systematic cognitive biases, how assigning them a demographic persona changes those biases, and whether bias scores from a controlled benchmark predict behaviour in realistic decisions. The authors built three linked resources. The first is a benchmark of 120 Yes/No vignettes covering 12 bias types, drafted with GPT-4o and then manually checked, where "Yes" is always the bias-consistent answer. The second is a factorial space of 162 personas. The third is a set of 100 financial choice tasks, each presented in a gain version and a loss version.

Ten instruction-tuned models were tested with and without personas. The authors measured Yes-rate, framing reversals and self-reported confidence, and used sign tests and Spearman correlations for analysis. On average, personas reduced reversals between frames and slightly raised confidence, though the effect differed widely across model families and sizes. Benchmark scores for availability, social proof and framing correlated with sensitivity to framing in the financial tasks. This suggests that standardized bias tests can serve as early indicators of instability in real decisions made with LLM support.

Key points

  • Benchmark design: 12 categories × 10 items = 120 tasks, including anchoring, framing, confirmation, availability, loss aversion, sunk cost, social proof and endowment. Every answer must be a strict JSON object with an answer and a reason.
  • Persona grid: gender (2) × age (3) × politics (3) × income (3) × education (3) = 162 personas.
    • Randomized country and occupation fields are added, each drawn from eight options.
    • Scores are aggregated with population-style marginal weights informed by OECD and Pew sources.
  • Applied suite: 100 financial tasks contrast a certain option with a 50/50 risky option, each in matched gain and loss frames.
  • Main findings:
    • Persona prompting lowers framing reversals on average and modestly increases confidence.
    • Results are strongly heterogeneous across the ten models.
    • Availability, social proof and framing scores are the best predictors of applied framing sensitivity.
  • Availability: the data and prompts were promised for release after acceptance. The paper appeared at LREC 2026.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Language models & pretraining · Evaluation & benchmarks

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2dvjjaywrket

@inproceedings{yoshikawa2026persona,
  title = {{Persona-Aware Evaluation of Cognitive Bias in LLMs: From Benchmark to Applied Decision-Making}},
  author = {Yoshikawa, Katsumasa and Takayama, Junya and Yamazaki, Takato},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/2dvjjaywrket},
  url = {https://doi.org/10.63317/2dvjjaywrket}
}

References

38 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ryan Best and Neil Charness. 2015. Age differences in the effect of framing on risky choice: A meta-analysis. Psychology and Aging, 30:688– 698. find
  2. BSI. 1973a. Natural Fibre Twines, 3rd edition. British Standards Institution, London. BS 2570. find
  3. BSI. 1973b. Natural fibre twines. BS 2570, British Standards Institution, London. 3rd. edn. find
  4. James Byrnes, David Miller, and William Schafer. 1999. Gender differences in risk taking: A meta-analysis. Psychological Bulletin, 125:367–383. find
  5. A. Castor and L. E. Pollux. 1992. The use of user modelling to guide inference and learning. Applied Intelligence, 2(1):37–53. find
  6. J.L. Chercheur. 1994. Case-Based Reasoning, 2nd edition. Morgan Kaufman Publishers, San Mateo, CA. find
  7. N. Chomsky. 1973. Conditions on transformations. In A festschrift for Morris Halle, New York. Holt, Rinehart & Winston. find
  8. Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2019. Plug and play language models: A simple approach to controlled text generation. CoRR, abs/1912.02164. find our card
  9. Umberto Eco. 1990. The Limits of Interpretation. Indian University Press. find
  10. Federico Germani and Giovanni Spitale. 2025. Source framing triggers systematic evaluation bias in large language models. find
  11. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. CoRR, abs/2009.03300. find our card
  12. Paul Gerhard Hoel. 1971a. Elementary Statistics, 3rd edition, Wiley series in probability and mathematical statistics, pages 19–33. Wiley, New York, Chichester. ISBN 0 471 40300. find
  13. Paul Gerhard Hoel. 1971b. Elementary Statistics, 3rd edition. Wiley series in probability and mathematical statistics. Wiley, New York, Chichester. ISBN 0 471 40300. find
  14. Tiancheng Hu and Nigel Collier. 2024. Quantifying the persona effect in llm simulations. find
  15. Otto Jespersen. 1922. Language: Its Nature, Development, and Origin. Allen and Unwin. find
  16. Jingru Jia, Zehua Yuan, Junhao Pan, Paul E. McNamara, and Deming Chen. 2024. Decision-making behavior evaluation framework for llms under uncertain context. find
  17. Daniel Kahneman and Amos Tversky. 1979. Prospect theory: An analysis of decision under risk. Econometrica, 47(2):263–291. find
  18. Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019. CTRL: A conditional transformer language model for controllable generation. CoRR, abs/1909.05858. find our card
  19. Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021. Dyna… find
  20. Jiwei Li, Michel Galley, Chris Brockett, Georgios Spithourakis, Jianfeng Gao, and Bill Dolan. 2016. A persona-based neural conversation model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 994–1003, Berlin, Germany. Associat… find our card
  21. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric… find
  22. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics… find our card
  23. Gili Lior, Liron Nacchace, and Gabriel Stanovsky. 2025. Wildframe: Comparing framing in humans and llms on naturally occurring texts. find
  24. Marlene Lutz, Indira Sen, Georg Ahnert, Elisa Rogers, and Markus Strohmaier. 2025. The prompt makes the person(a): A systematic evaluation of sociodemographic persona prompting for large language models. find
  25. Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereotypical bias in pre-trained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Lon… find
  26. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953–1967, Online. Association for… find our card
  27. OECD Family Database. 2025. Sf1.4: Population by age of children and young adults, and youth dependency ratio. Indicator SF1.4. find
  28. Shumiao Ouyang, Hayong Yun, and Xingjian Zheng. 2025. Ai as decision-maker: Ethics and risk preferences of llms. find
  29. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022. BBQ: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2086–2105, Dublin, Ireland. Asso… find our card
  30. Pew Research Center. 2025. Methodology: Ideology classification (left, center, right). find
  31. Jillian Ross, Yoon Kim, and Andrew W. Lo. 2024. Llm economicus? mapping the behavioral biases of llms via utility theory. find
  32. Charles Joseph Singer, E. J. Holmyard, and A. R. Hall, editors. 1954–58. A history of technology. Oxford University Press, London. 5 vol. find
  33. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Al… find our card
  34. Jannik Strötgen and Michael Gertz. 2012. Temporal tagging on different domains: Challenges, strategies, and gold standards. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), pages 3746– 3753, Istanbul, Turkey. European Language Resource Association… find our card
  35. S. Superman, B. Batman, C. Catwoman, and S. Spiderman. 2000. Superheroes experiences with books, 20th edition. The Phantom Editors Associates, Gotham City. find
  36. Amos Tversky and Daniel Kahneman. 1981. The framing of decisions and the psychology of choice. Science, 211(4481):453–458. find
  37. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204–2213, Melbou… find our card
  38. Mingqian Zheng, Jiaxin Pei, Lajanugen Logeswaran, Moontae Lee, and David Jurgens. 2024. When "a helpful assistant" is not really helpful: Personas in system prompts do not improve performances of large language models. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yoshikawa-2026-persona-aware-evaluation-cognitive-bias/