Elamur Research⌕
☰
home / papers / 2026 / Quantifying and Mitigating Socially Desirable Responding in LLMs: A Desirability-Matched Graded Forced-Choice Psychometric Study

Quantifying and Mitigating Socially Desirable Responding in LLMs: A Desirability-Matched Graded Forced-Choice Psychometric Study

Paper · 2026 · Kensuke Okada, Yui Furukawa and Kyosuke Bunji

Summary

LLMs are increasingly audited with human personality questionnaires, but these tools assume candid answers. Models tuned to be helpful and inoffensive tend to choose answers that look good, a bias called socially desirable responding (SDR), which distorts the resulting profiles. The authors propose a psychometric framework that both measures and reduces this bias. To measure it, each model completes the same inventory under HONEST and FAKE-GOOD instructions. SDR is then the direction-corrected standardized gap between the two conditions, computed on latent scores estimated with item response theory (IRT). This makes SDR comparable across traits, response formats and human faking studies. To reduce it, they build a graded forced-choice (GFC) Big Five inventory: statements from different domains are paired so that their desirability is closely matched, and responses are scored with ordinal Thurstonian IRT. They tested nine instruction-following LLMs prompted with synthetic personas whose target profiles were known. Likert formats produced large SDR. The matched GFC version sharply reduced SDR while mostly preserving recovery of the intended personas. The authors therefore recommend reporting SDR susceptibility alongside questionnaire-based LLM profiles.

Key points

  • Item pool: Goldberg's public-domain IPIP Big-Five factor-marker inventory, with 100 items cut to 98 after removing two voting-related statements. Every item is tagged with its keying direction and assigned to one of the five domains, namely A, C, E, N or O.
  • Desirability ratings: GPT-5 and Gemini 2.5 Pro rated item desirability on a 9-point scale, following protocols from human rating studies. The ratings were checked for stability and against human norms.
  • Pair selection: Constrained optimization chose 30 cross-domain, desirability-matched pairs for the GFC inventory. The Likert version used 60 ratings, against 30 pair judgments for GFC.
  • Trade-off: The balance between SDR and persona recovery differed by model, and model size did not reliably predict SDR. Human comparison values come from a heterogeneous meta-analysis (Speer et al., 2023), so human–LLM contrasts are tentative.
  • Availability: Data and code are released on OSF (osf.io/2e6ny).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of TokyoTokyo · Japan90 works · 9,156 citations

Also on the byline: Kobe University.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Social Sciences · 1,772 worksSafety Research · 543 worksSociology and Political Science · 577 worksDecision Sciences · 209 worksManagement Science and Operations Research · 108 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Kensuke Okada, Yui Furukawa and Kyosuke Bunji. 2026. Quantifying and Mitigating Socially Desirable Responding in LLMs: A Desirability-Matched Graded Forced-Choice Psychometric Study. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1865

@inproceedings{okada2026quantifying,
  title = {{Quantifying and Mitigating Socially Desirable Responding in LLMs: A Desirability-Matched Graded Forced-Choice Psychometric Study}},
  author = {Kensuke Okada, Yui Furukawa and Kyosuke Bunji},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1865},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1865}
}

References

52 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Inger L. Abma, Maroeska Rovers, and Philip J. van der Wees. 2016. Appraising convergent validity of patient-reported outcome measures in systematic reviews: constructing hypotheses and interpreting outcomes. BMC Research Notes, 9:226. find
  2. Pranav Bhandari, Usman Naseem, Amitava Datta, Nicolas Fay, and Mehwish Nasim. 2025. Evaluating personality traits in large language models: Insights from psychological questionnaires. In Companion Proceedings of the ACM Web Conference 2025, WWW Companion ’25, pages 868–872, New York, NY, USA. Associ… find
  3. Bojana Bodroža, Bojana M. Dinić, and Ljubiša Bojić. 2024. Personality testing of large language models: limited temporal stability, but highlighted prosociality. Royal Society Open Science, 11(10):240180. find
  4. Sara Britz, Jessica Heintz, Lena Rader, Siegfried Gauggel, and Verena Mainz. 2023. An English list of trait words including valence, social desirability, and observability ratings. Behavior Research Methods, 55:2669–2686. Published online 12 Aug 2022. find
  5. Anna Brown. 2016. Item response models for forced-choice questionnaires: A common framework. Psychometrika, 81(1):135–160. find
  6. Anna Brown and Alberto Maydeu-Olivares. 2011. Item response modeling of forced-choice questionnaires. Educational and Psychological Measurement, 71(3):460–502. find
  7. Anna Brown and Alberto Maydeu-Olivares. 2013. How IRT can solve problems of ipsative data in forced-choice questionnaires. Psychological Methods, 18(1):36–52. find
  8. Anna Brown and Alberto Maydeu-Olivares. 2018. Ordinal factor analysis of graded-preference questionnaire data. Structural Equation Modeling: A Multidisciplinary Journal, 25(4):516–529. find
  9. Paul-Christian Bürkner. 2022. On the information obtainable from comparative judgments. Psychometrika, 87(4):1439–1472. find
  10. Mengyang Cao and Fritz Drasgow. 2019. Does forcing reduce faking? a meta-analytic review of forced-choice personality measures in high-stakes situations. Journal of Applied Psychology, 104(11):1347–1368. find
  11. Graham Caron and Shashank Srivastava. 2023. Manipulating the perceived personality traits of language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2370–2386, Singapore. Association for Computational Linguistics. find
  12. Bob Carpenter, Andrew Gelman, Matthew D. Hoffman, Daniel Lee, Ben Goodrich, Michael Betancourt, Marcus Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell. 2017. Stan: A probabilistic programming language. Journal of Statistical Software, 76(1):1–32. find
  13. Jacob Cohen. 1988. Statistical Power Analysis for the Behavioral Sciences, 2 edition. Lawrence Erlbaum Associates. find
  14. Linda Anne Coker, Douglas B. Samuel, and Thomas A. Widiger. 2002. Maladaptive personality functioning within the big five and the five-factor model. Journal of Personality Disorders, 16(5):385–401. find
  15. College Board. 2024. Assessment framework for the digital SAT suite. Version 3.01 (August 2024). find
  16. Adrian F. Furnham. 1997. Knowing and faking one’s five-factor personality score. Journal of Personality Assessment, 69(1):229–243. find
  17. Lewis R. Goldberg. 1999. A broad-bandwidth, public-domain, personality inventory measuring the lower-level facets of several five-factor models. In I. Mervielde, I. J. Deary, F. De Fruyt, and F. Ostendorf, editors, Personality Psychology in Europe, volume 7, pages 7–28. Tilburg University Press, Til… find
  18. Akshat Gupta, Xiaoyang Song, and Gopala Anumanchipalli. 2024. Self-assessment tests are unreliable measures of LLM personality. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 301–314, Miami, Florida, US. Association for Computational Linguis… find
  19. James J. Heckman and Tim Kautz. 2012. Hard evidence on soft skills. Labour Economics, 19(4):451–464. find
  20. Jen-tse Huang, Wenxiang Jiao, Man Ho Lam, Eric John Li, Wenxuan Wang, and Michael Lyu. 2024. On the reliability of psychological scales on large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6152–6173, Miami, Florida, USA. Associati… find
  21. Hang Jiang, Xiajie Zhang, Xubo Cao, Cynthia Breazeal, Deb Roy, and Jad Kabbara. 2024. PersonaLLM: Investigating the ability of large language models to express personality traits. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3605–3627, Mexico City, Mexico. Associat… find
  22. Daniël Lakens. 2013. Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and anovas. Frontiers in Psychology, 4:863. find
  23. Daniël Lakens, Anne M. Scheel, and Peder M. Isager. 2018. Equivalence testing for psychological research: A tutorial. Advances in Methods and Practices in Psychological Science, 1(2):259–269. find
  24. Seungbeen Lee, Seungwon Lim, Seungju Han, Giyeong Oh, Hyungjoo Chae, Jiwan Chung, Minju Kim, Beong-woo Kwak, Yeonsoo Lee, Dongha Lee, Jinyoung Yeo, and Youngjae Yu. 2025. Do LLMs have distinct and consistent personality? TRAIT: Personality testset designed for LLMs with psychometrics. In Findings of… find
  25. Mengtong Li, Bo Zhang, Lingyue Li, Tianjun Sun, and Anna Brown. 2025a. Mix-keying or desirability-matching in the construction of forced-choice measures? an empirical investigation and practical recommendations. Organizational Research Methods, 28(2):296–329. find
  26. Xiaoyu Li, Haoran Shi, Zengyi Yu, Yukun Tu, and Chan-jin Zheng. 2025b. Decoding LLM personality measurement: Forced-choice vs. Likert. In Findings of the Association for Computational Linguistics: ACL 2025, pages 9234–9247, Vienna, Austria. Association for Computational Linguistics. find
  27. Alexandra Martínez and Jesús F. Salgado. 2021. A meta-analysis of the faking resistance of forced-choice personality inventories. Frontiers in Psychology, 12:732241. find
  28. Marilù Miotto, Nicola Rossberg, and Bennett Kleinberg. 2022. Who is GPT-3? an exploration of personality, values and demographics. In Proceedings of the Fifth Workshop on Natural Language Processing and Computational Social Science (NLP+CSS), pages 218–227, Abu Dhabi, UAE. Association for Computatio… find
  29. OECD. 2021. Beyond Academic Learning: First Results from the Survey of Social and Emotional Skills. OECD Publishing, Paris. find
  30. OECD. 2025. Pisa 2022 technical report (revised edition). Technical report, OECD Publishing, Paris. find
  31. Kensuke Okada and Kyosuke Bunji. 2021. Increase of reliability by incorporating response time into the paired-comparison psychological measurement. Behaviormetrika, 48:169–177. find
  32. Delroy L. Paulhus. 2002. Socially desirable responding: The evolution of a construct. In H. I. Braun, D. N. Jackson, and D. E. Wiley, editors, The role of constructs in psychological and educational measurement, pages 49–69. Erlbaum, Mahwah, NJ. find
  33. Max Pellert, Clemens M. Lechner, Indira Sen, and Markus Strohmaier. 2025. Neural network embeddings recover value dimensions from psychometric survey items on par with human data. arXiv preprint.
  34. Max Pellert, Clemens M. Lechner, Claudia Wagner, Beatrice Rammstedt, and Markus Strohmaier. 2024. AI psychometrics: Assessing the psychological profiles of large language models through psychometric inventories. Perspectives on Psychological Science, 19(5):808–826. find
  35. Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, and 44 others. 2023. Dis… — Discovering Language Model Behaviors with Model-Written Evaluations
  36. PROMIS. 2013. Promis® instrument development and validation scientific standards. Version 2.0 (revised May 2013). Patient-Reported Outcomes Measurement Information System (PROMIS), funded by NIH. find
  37. Matthias Rose, Jakob B. Bjorner, Barbara Gandek, Bonnie Bruce, James F. Fries, and John E. Ware. 2014. The PROMIS physical function item bank was calibrated to a standardized metric and shown to improve measurement efficiency. Journal of Clinical Epidemiology, 67(5):516–526. find
  38. Aadesh Salecha, Molly E. Ireland, Shashanka Subrah-manya, João Sedoc, Lyle H. Ungar, and Johannes C. Eichstaedt. 2024. Large language models display human-like social desirability biases in big five personality surveys. PNAS Nexus, 3(12):pgae533. find
  39. Fumiko Samejima. 1969. Estimation of latent ability using a response pattern of graded scores. Psychometrika, 34(Suppl 1):1–97. Psychometrika Monograph Supplement, No. 17. find
  40. Lizabeth M. Sankis, Elizabeth M. Corbitt, and Thomas A. Widiger. 1999. Gender bias in the English language? Journal of Personality and Social Psychology, 77(6):1289–1295. find
  41. Gregory Serapio-García, Mustafa Safdari, Clément Crepy, Luning Sun, Stephen Fitz, Peter Romero, Marwa Abdulhai, Aleksandra Faust, and Maja J. Matarić. 2023. Personality traits in large language models. arXiv preprint, arXiv:2307.00184.
  42. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. 2024. T… — Towards Understanding Sycophancy in Language Models
  43. Bangzhao Shu, Lechen Zhang, Minje Choi, Lavinia Dunagan, Lajanugen Logeswaran, Moontae Lee, Dallas Card, and David Jurgens. 2024. You don’t need a personality test to know these models are unreliable: Assessing the reliability of large language models on psychometric instruments. In Proceedings of t… find
  44. Charles Spearman. 1904. "General Intelligence," Objectively Determined and Measured. The American Journal of Psychology, 15(2):201–292. find
  45. Andrew B. Speer, Lauren J. Wegmeyer, Andrew P. Tenbrink, Angie Y. Delacruz, Neil D. Christiansen, and Rouan M. Salim. 2023. Comparing forced-choice and single-stimulus personality scores on a level playing field: A meta-analysis of psychometric properties and susceptibility to faking. Journal of App… find
  46. Luning Sun, Zijie Qin, Shan Wang, Xuetao Tian, and Fang Luo. 2024. Contributions to constructing forced-choice questionnaires using the thurstonian IRT model. Multivariate Behavioral Research, 59(2):229–250. find
  47. Dimitri van der Linden, Jan te Nijenhuis, and Arnold B. Bakker. 2010. The general factor of personality: A meta-analysis of Big Five intercorrelations and a criterion-related validity study. Journal of Research in Personality, 44(3):315–327. find
  48. Wim J. van der Linden, editor. 2018a. Handbook of Item Response Theory: Three-Volume Set. Chapman & Hall/CRC Statistics in the Social and Behavioral Sciences. Chapman and Hall/CRC. find
  49. Wim J. van der Linden, editor. 2018b. Handbook of Item Response Theory: Volume 3: Applications. Chapman & Hall/CRC Statistics in the Social and Behavioral Sciences. Chapman and Hall/CRC, Boca Raton, FL. find
  50. Haoran Ye, Jing Jin, Yuhang Xie, Xin Zhang, and Guojie Song. 2025. Large language model psychometrics: A systematic review of evaluation, validation, and enhancement. arXiv preprint. find
  51. Bo Zhang, Jing Luo, and Jian Li. 2024. Moving beyond Likert and traditional forced-choice scales: A comprehensive investigation of the graded forced-choice format. Multivariate Behavioral Research, 59(3):434–460. find
  52. Jingyao Zheng, Xian Wang, Simo Hosio, Xiaoxian Xu, and Lik-Hang Lee. 2025. LMLPA: Language model linguistic personality assessment. Computational Linguistics, 51:599–640. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/okada-2026-quantifying-mitigating-socially-desirable-responding/