Elamur Research⌕
☰
home / papers / 2023 / Do LLMs Possess a Personality? Making the MBTI Test an Amazing Evaluation for Large Language Models

Do LLMs Possess a Personality? Making the MBTI Test an Amazing Evaluation for Large Language Models

Paper · 2023 · Keyu Pan and Yawen Zeng

Summary

The paper asks whether large language models show something like a human personality, and whether the Myers-Briggs Type Indicator (MBTI) can work as a rough evaluation tool for them. The authors give a 93-item, two-option MBTI questionnaire to six LLMs. Rather than parsing generated text, they compare the final-position logits for the tokens "A" and "B" and take the more likely option. The answers are then aggregated into the four MBTI dichotomies (E/I, S/N, T/F, J/P). The authors also test whether two kinds of prompts can shift a model's type: explicit role-play descriptions and implicit few-shot examples. Finally, they continue training Bloom and Llama models on three Chinese corpora to see how data type moves the scores. Each model receives a distinct type, and models in the same family share several letters. Prompts change the type only in sufficiently instruction-tuned models. The training corpus mostly affects T/F and J/P. The authors admit MBTI is scientifically weak, but they argue T/F and J/P may roughly reflect reasoning and planning ability.

Key points

  • Assigned types: ChatGPT = ENTJ, GPT-4* = INTJ, Bloom7b = ISTJ, BaiChuan7b = ENFP, BaiChuan13b = INFP, OpenLlama7b (v2) = INFJ. Both OpenAI models share "NTJ" and both BaiChuan models share "NFP". GPT-4 refusals to answer were excluded.
  • Smaller models in a family leaned toward E and larger ones toward I. ChatGPT and GPT-4* showed the most extreme dichotomy scores, while Bloom7b and BaiChuan13b were closer to balanced.
  • The continued-training corpora were Chinese Wikipedia, a Q&A corpus (~400M tokens) that raised the P (flexibility) score, and an examination corpus (~50M tokens, mainly APE210k). The exam data raised T scores: Bloom went from 13 to 15 and Llama from 9 to 10.
  • No consistent patterns appeared for E/I or S/N. The authors propose higher T and J values as indicators of stronger reasoning and planning, and suggest INTJ (GPT-4) as a desirable profile for assistant models.
  • Limitations: training experiments used models of about 10B parameters and corpora of at most ~400M tokens. Code is available in the HarderThenHarder/transformers_tasks GitHub repository (LLM/llms_mbti).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢ByteDanceBeijing · China47 works · 4,339 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksPsychology · 1,649 worksClinical Psychology · 21 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksEvaluation & benchmarks · 1,341 works

Cite

Cite as Keyu Pan and Yawen Zeng. 2023. Do LLMs Possess a Personality? Making the MBTI Test an Amazing Evaluation for Large Language Models. arXiv:2307.16180.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2307.16180 · arXiv 2307.16180

@misc{pan2023do,
  title = {{Do LLMs Possess a Personality? Making the MBTI Test an Amazing Evaluation for Large Language Models}},
  author = {Keyu Pan and Yawen Zeng},
  year = {2023},
  doi = {10.48550/arxiv.2307.16180},
  eprint = {2307.16180},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.16180}
}

References

21 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gregory J Boyle. 1995. Myers-briggs type indicator (mbti): some psychometric limitations. Australian Psychologist, 30(1):71–74. find
  2. Johana Cabrera, María Soledad Loyola, Irene Magaña, and Rodrigo Rojas. 2023. Ethical dilemmas, mental health, artificial intelligence, and llm-based chatbots. In Bioinformatics and Biomedical Engineering -10th International Work-Conference, IWBBIO 2023, Meloneras, Gran Canaria, Spain, July 12-14, 20… find
  3. Ning Han, Jingjing Chen, Guangyi Xiao, Hao Zhang, Yawen Zeng, and Hao Chen. 2021. Fine-grained cross-modal alignment network for text-video retrieval. In Proceedings of the 29th ACM International Conference on Multimedia, pages 3826– 3834. ACM. find
  4. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding.
  5. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.08322.
  6. Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023a. Halueval: A large-scale hallucination evaluation benchmark for large language models.
  7. Xingxuan Li, Yutong Li, Shafiq Joty, Linlin Liu, Fei Huang, Lin Qiu, and Lidong Bing. 2023b. Does gpt-3 demonstrate psychopathy? evaluating large language models from a psychological perspective. find
  8. Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
  9. Haocong Rao, Cyril Leung, and Chunyan Miao. 2023. Can ChatGPT assess human personalities? a general evaluation framework. arXiv preprint arXiv:2303.01248. find
  10. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv:1707.06347. find
  11. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2023. Language models are multilingual chain-of-thought reasoners. In The Eleventh International Conference on Learning Representa…
  12. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed Chi, Denny Zhou, and Jason Wei. 2023. Challenging big-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics…
  13. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowledge.
  14. Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jianshu Chen, and Dong Yu. 2023. A stitch in time saves nine: Detecting and mitigating hallucinations of llms by validating low-confidence generation. CoRR, abs/2307.03987.
  15. Huanwen Wang, Yawen Zeng, Jianguo Chen, Zhouting Zhao, and Hao Chen. 2022. A spatiotemporal graph neural network for session-based recommendation. Expert Systems with Applications, 202:117114. find
  16. Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with chatgpt.
  17. Jakob Mökander, Jonas Schuett, Hannah Rose Kirk, and Luciano Floridi. Auditing large language models: a three-layered approach. AI and Ethics. find
  18. Benfeng Xu, An Yang, Junyang Lin, Quan Wang, Chang Zhou, Yongdong Zhang, and Zhendong Mao. 2023. Expertprompting: Instructing large language models to be distinguished experts.
  19. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? find
  20. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023. A survey of l…
  21. Wei Zhao, Mingyue Shang, Yang Liu, Liang Wang, and Jingming Liu. 2020. Ape210k: A large-scale and template-rich dataset of math word problems. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pan-2023-llms-possess-personality-making-mbti/