Elamur Research⌕
☰
home / papers / 2024 / Role-Play Paradox in Large Language Models: Reasoning Performance Gains and Ethical Dilemmas

Role-Play Paradox in Large Language Models: Reasoning Performance Gains and Ethical Dilemmas

Paper · 2024 · Jinman Zhao, Zifan Qian, Linbo Cao et al.

Summary

Role-play prompting, where an LLM is told to act as a teacher, scriptwriter, or even an object like a coin, is known to improve reasoning. This paper asks whether the same technique weakens safety. The authors test ChatGPT (gpt-3.5-turbo-1106), GPT-4o and Mixtral 8x7B on three safety benchmarks: CrowS-Pairs, StereoSet and HarmfulQ. They use roles taken from earlier reasoning-oriented work and combine them with zero-shot, zero-shot CoT and RE2 prompting.

They also build a training-free "role autotune" setup. In it, GPT-4 picks a role suited to each question, and the tested model then answers in that role. A further study keeps the occupation fixed but varies the assigned race, gender or religion.

In nearly every configuration, adopting a role made models more likely to give stereotyped or harmful answers than using no role. Automatic role selection was especially damaging. Identity-specific roles produced over-calibrated behavior but generally stayed below the safety of the no-role baseline. The paper argues that a technique promoted for reasoning gains carries real risks in sensitive deployments.

Key points

  • Benchmarks: CrowS-Pairs (1508 sentence pairs, nine stereotype dimensions), StereoSet (17,000 instances) and HarmfulQ (200 explicitly toxic questions).
  • Scoring: for the stereotype sets, a response counts as correct only when the model picks the "unknown" option rather than the biased or anti-biased one.
  • Setup: temperature 0.7, with each question asked 3 times.
  • Role autotune on GPT-3.5: accuracy fell from 57% to 28% on StereoSet and from 75% to 58% on CrowS-Pairs.
  • Role autotune on HarmfulQ: the share of refused harmful questions dropped for every model:
    • GPT-3.5: 92.5 → 74.5
    • GPT-4: 97.5 → 92
    • GPT-4o: 97 → 95
  • Gender-specific occupational roles (CrowS-Pairs, GPT-3.5): average accuracy was 53 for female and 54 for male roles, versus a base of 58. Non-binary roles scored 64, which the authors read as over-calibration.
  • Race and religion roles: results also varied across categories, with "Buddhism" and "From Europe" scoring notably higher.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TorontoToronto · Canada96 works · 142,007 citations🎓Renmin University of ChinaBeijing · China114 works · 16,118 citations🎓University of WaterlooWaterloo · Canada67 works · 13,592 citations🎓McGill UniversityMontreal · Canada87 works · 11,027 citations🎓University of AlbertaEdmonton · Canada44 works · 3,236 citations🎓Australian National UniversityCanberra · Australia23 works · 6,522 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 worksBias, fairness & safety · 517 works

Cite

Cite as Jinman Zhao, Zifan Qian, Linbo Cao, Yining Wang, Yitian Ding, Yulan Hu, Zeyu Zhang, Zeyong Jin University of Toronto, University of Alberta, University of Waterloo, McGill University, Renmin University of China and The Australian National University). 2024. Role-Play Paradox in Large Language Models: Reasoning Performance Gains and Ethical Dilemmas. arXiv:2409.13979.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2409.13979 · arXiv 2409.13979

@misc{zhao2024role,
  title = {{Role-Play Paradox in Large Language Models: Reasoning Performance Gains and Ethical Dilemmas}},
  author = {Jinman Zhao, Zifan Qian, Linbo Cao, Yining Wang, Yitian Ding, Yulan Hu, Zeyu Zhang, Zeyong Jin University of Toronto, University of Alberta, University of Waterloo, McGill University, Renmin University of China and The Australian National University)},
  year = {2024},
  doi = {10.48550/arxiv.2409.13979},
  eprint = {2409.13979},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2409.13979}
}

References

24 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. ference on learning representations. Retrieved from https://openreview.net/forum?id=kGteeZ18Ir Ahn, J., Kim, J., & Sung, Y. (2022). The effect of gen-Huang, X., Liu, W., Chen, X., Wang, X., Wang, H., der stereotypes on artificial intelligence recommendations. Lian, D., . . . Chen, E. (2024). Underst… ↗
  2. Lan-Hutchinson, B. (2024). Modeling the sacred: Considerations guage models are few-shot learners. Retrieved from when using religious texts in natural language processing. https://arxiv.org/abs/2005.14165 Retrieved from https://arxiv.org/abs/2404.14740 Chu, C. H., Donato-Woodger, S., Khan, S. S., N… ↗
  3. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., (2023). Large language models are zero-shot reasoners. Kaiser, L., . . . Schulman, J. (2021). Training verifiers to Retrieved from https://arxiv.org/abs/2205.11916 solve math word pro… ↗
  4. Dev, S., Monajatipoor, M., Ovalle, A., Subramonian, Lin, Z., Wang, Z., Tong, Y., Wang, Y., Guo, Y., Wang, Y., & A., Phillips, J., & Chang, K.-W. (2021, November). find
  5. Shang, J. (2023). Toxicchat: Unveiling hidden challenges of Harms of gender exclusivity and challenges in non-toxicity detection in real-world user-ai conversation. arXiv binary representation in language technologies. In preprint arXiv:2310.17389. — [arXiv:2310.17389](https://arxiv.org/abs/2310.17389) find
  6. M.-F. Moens, X. Huang, L. Specia, & S. W.-t. Yih Lu, K., Yu, B., Zhou, C., & Zhou, J. (2024, August). Large lan-(Eds.), Proceedings of the 2021 conference on empirical guage models are superpositions of all characters: Attaining methods in natural language processing (pp. 1968–1994). arbitrary role-… — Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies
  7. Meade, N., Poole-Dayan, E., & Reddy, S. (2022, May). find
  8. An Garg, T., Masud, S., Suresh, T., & Chakraborty, T. (2023). empirical survey of the effectiveness of debiasing techniques Handling bias in toxic speech detection: A survey. ACM for pre-trained language models. In S. Muresan, P. Nakov, Computing Surveys, 55(13s), 1–32. & A. Villavicencio (Eds.), Pr…
  9. A., Clark, P., Sabharwal, A., & Khot, T. (2024). 10.18653/v1/2022.acl-long.132 Bias runs deep: Implicit reasoning biases in persona-assigned LLMs. In The twelfth international con-Morzhov, S. (2020). Avoiding unintended bias in toxicity classification with neural networks. In 2020 26th conference of…
  10. Nadeem, A., Marjanovic, O., Abedin, B., et al. (2022). Gender bias in ai-based decision-making systems: a systematic literature review. Australasian Journal of Information Systems, 26. find
  11. Nadeem, M., Bethke, A., & Reddy, S. (2021, August). StereoSet: Measuring stereotypical bias in pretrained language models. In C. Zong, F. Xia, W. Li, & R. Navigli (Eds.), Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conferen…
  12. Nangia, N., Vania, C., Bhalerao, R., & Bowman, S. R. (2020, November). CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In B. Webber, T. Cohn, Y. He, & Y. Liu (Eds.), Proceedings of the 2020 conference on empirical methods in natural language processing (emnlp)…
  13. Omowole, A. (2021). Research shows ai is often biased. here’s how to make algorithms work for all of us. In World economic forum. how ai and machine learning can help reduce bias and discrimination. world economic forum. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., . . .…
  14. Park, M., Kim, S., Lee, S., Kwon, S., & Kim, K. (2024, May). Empowering personalized learning through a conversation-based tutoring system with student modeling. In Extended abstracts of the chi conference on human factors in computing systems. ACM. Retrieved from http://dx.doi.org/10.1145/3613905.3… — [doi:10.1145/3613905.3651122](https://doi.org/10.1145/3613905.3651122) ↗
  15. Shaikh, O., Zhang, H., Held, W., Bernstein, M., & Yang, D. (2022). On second thought, let’s not think step by step! bias and toxicity in zero-shot reasoning. arXiv preprint arXiv:2212.08061.
  16. Shanahan, M., McDonell, K., & Reynolds, L. (2023). Role play with large language models. Nature, 1–6.
  17. Sobhani, N., Sengupta, K., & Delany, S. J. (2023, September). Measuring gender bias in natural language processing: Incorporating gender-neutral linguistic forms for non-binary gender identities in abusive speech detection. In R. Mitkov & G. Angelova (Eds.), Proceedings of the 14th international con…
  18. Talmor, A., Herzig, J., Lourie, N., & Berant, J. (2019, June). CommonsenseQA: A question answering challenge targeting commonsense knowledge. In J. Burstein, C. Doran, & T. Solorio (Eds.), Proceedings of the 2019 conference of the north American chapter of the association for computational linguisti…
  19. Venkit, P. N., & Wilson, S. (2021). Identification of bias against people with disabilities in sentiment analysis and toxicity detection models. arXiv preprint arXiv:2111.13259. — [arXiv:2111.13259](https://arxiv.org/abs/2111.13259) find
  20. Wan, Y., & Chang, K.-W. (2024). White men lead, black women help: Uncovering gender, racial, and intersectional bias in language agency. arXiv preprint arXiv:2404.10508. Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., . . . Anandkumar, A. (2023). Voyager: An open-ended embodied agent…
  21. Wu, C.-K., Chen, W.-L., & Chen, H.-H. (2023). Large language models perform diagnostic reasoning. Retrieved from https://openreview.net/forum?id=N0lQfjeNWOE ↗
  22. Wu, W., Wu, H., Jiang, L., Liu, X., Zhao, H., & Zhang, M. (2024, August). From role-play to drama-interaction: An LLM solution. In L.-W. Ku, A. Martins, & V. Srikumar (Eds.), Findings of the association for computational linguistics acl 2024 (pp. 3271–3290). Bangkok, Thailand and virtual meeting: As… find
  23. Xu, X., Tao, C., Shen, T., Xu, C., Xu, H., Long, G., . . . Ma, S. (2024). Re-reading improves reasoning in large language models. In Proceedings of the 2024 conference on empirical methods in natural language processing (pp. 15549–15575). Miami, Florida, USA: Association for Computational Linguistic… find
  24. Zhao, J., Ding, Y., Jia, C., Wang, Y., & Qian, Z. (2024). Gender bias in large language models across multiple languages. arXiv preprint arXiv:2403.00277. — [arXiv:2403.00277](https://arxiv.org/abs/2403.00277) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhao-2024-role-play-paradox-large-language/