Elamur Research⌕
☰
home / papers / 2023 / A Categorical Archive of ChatGPT Failures

A Categorical Archive of ChatGPT Failures

Paper · 2023 · Ali Borji

Summary

Most early evaluations of ChatGPT either celebrated its fluency or examined only its mathematical weaknesses. Borji instead assembles a broad, qualitative archive of cases where the model breaks down. Most examples were gathered from Twitter and community collections, including the error-tracking effort by Gary Marcus and Ernest Davis and Giuseppe Venuto's GitHub repository. The author also wrote some prompts himself. He sorts the failures into eleven loosely defined, overlapping categories, among them reasoning, logic, math, factual errors, coding and bias. He illustrates each category with concrete prompts and the model's wrong answers. The study runs no quantitative benchmark. Its main finding is that ChatGPT lacks a coherent world model: it produces confident text that falls apart on simple spatial, temporal, physical, psychological and commonsense puzzles. The paper also covers ethical and societal risks, including bias, misinformation, energy cost and opacity. It argues that this collection could become a standardized question set for tracking future chatbots and for generating synthetic training and test data.

Key points

  • Eleven failure categories. Reasoning is split into spatial, temporal, physical, psychological (Theory of Mind) and commonsense subtypes. The author admits the taxonomy is arbitrary and that one failure can fit several categories.
  • Typical errors. ChatGPT got these simple puzzles wrong:
    • a six-person seating puzzle whose answer is "X";
    • a party arrival-order question;
    • the "five birds, shoot one" riddle, where it answered four instead of none;
    • a version of the Sally–Anne false-belief test.
  • Improvement between versions. The Winograd Schema trophy/suitcase item, which an older version failed, was answered correctly by the Jan 30, 2023 release.
  • Inconsistent safety behavior. Whether ChatGPT agreed to tell jokes about Persians and Arabs depended on the order of the requests.
  • Evaluation proposal. Because LLMs keep crawling the web, test items leak into training data. The author therefore suggests private held-out test sets, similar to those used in image recognition competitions. He also cites Meta's LLaMA as a step toward open models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksSocial Sciences · 744 worksSafety Research · 221 works

Cite

Cite as Ali Borji. 2023. A Categorical Archive of ChatGPT Failures. arXiv:2302.03494.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2302.03494 · arXiv 2302.03494

@misc{borji2023categorical,
  title = {{A Categorical Archive of ChatGPT Failures}},
  author = {Ali Borji},
  year = {2023},
  doi = {10.48550/arxiv.2302.03494},
  eprint = {2302.03494},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2302.03494}
}

References

68 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abubakar Abid, Maheen Farooqi, and James Zou. Persistent anti-muslim bias in large language models. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 298–306, 2021.
  2. Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. arXiv preprint arXiv:1905.13319, 2019.
  3. Amos Azaria. Chatgpt usage and limitations. arXiv, 2022. find
  4. Lea Bishop. A computer wrote this paper: What chatgpt means for education, research, and writing. Research, and Writing (January 26, 2023), 2023. find
  5. Som Biswas. Chatgpt and the future of medical writing, 2023. find
  6. Ali Borji. Pros and cons of gan evaluation measures. Computer Vision and Image Understanding, 179:41–65, 2019. find
  7. Ali Borji. Pros and cons of gan evaluation measures: New developments. Computer Vision and Image Understanding, 215:103329, 2022. find
  8. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326, 2015. find
  9. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  10. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023.
  11. Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency, pages 77–91. PMLR, 2018. find
  12. Lindsey J Byom and Bilge Mutlu. Theory of mind: Mechanisms, methods, and new directions. Frontiers in human neuroscience, 7:413, 2013. find
  13. Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. arXiv preprint arXiv:2301.13188, 2023. — [arXiv:2301.13188](https://arxiv.org/abs/2301.13188) find
  14. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In USENIX Security Symposium, volume 6, 2021.
  15. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
  16. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
  17. Ching-Yao Chuang, Varun Jampani, Yuanzhen Li, Antonio Torralba, and Stefanie Jegelka. Debiasing vision-language models via biased prompts. arXiv preprint arXiv:2302.00070, 2023. — [arXiv:2302.00070](https://arxiv.org/abs/2302.00070) find
  18. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
  19. Ernest Davis. Benchmarks for automated commonsense reasoning: A survey, 2023. find
  20. Payal Dhar. The carbon impact of artificial intelligence. Nat. Mach. Intell., 2(8):423–425, 2020. find
  21. Zenan Dou. Exploring gpt-3 model’s capability in passing the sally-anne test a preliminary study in two languages. OSF Preprints, 2023. find
  22. Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. Gpts are gpts: An early look at the labor market impact potential of large language models. arXiv preprint arXiv:2303.10130, 2023. — [arXiv:2303.10130](https://arxiv.org/abs/2303.10130) find
  23. Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier, and Julius Berner. Mathematical capabilities of chatgpt, 2023.
  24. Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.07597, 2023.
  25. Jochen Hartmann, Jasper Schwenzow, and Maximilian Witte. The political ideology of conversational ai: Converging evidence on chatgpt’s pro-environmental, left-libertarian orientation. arXiv preprint arXiv:2301.01768, 2023. find
  26. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021.
  27. Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. How good are gpt models at machine translation? a comprehensive evaluation. arXiv preprint arXiv:2302.09210, 2023. find
  28. Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. Is chatgpt a good translator? a preliminary study. arXiv preprint arXiv:2301.08745, 2023.
  29. Subbarao Kambhampati. Changing the nature of ai research. Communications of the ACM, 65(9):8–9, 2022. find
  30. John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. arXiv preprint arXiv:2301.10226, 2023.
  31. Christof Koch and Giulio Tononi. A test for consciousness. Scientific American, 304(6):44–47, 2011. find
  32. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916, 2022.
  33. Gerd Kortemeyer. Could an artificial-intelligence agent pass an introductory physics course? arXiv preprint arXiv:2301.12127, 2023. find
  34. Michal Kosinski. Theory of mind may have spontaneously emerged in large language models. arXiv preprint arXiv:2302.02083, 2023.
  35. Sebastian Krügel, Andreas Ostermaier, and Matthias Uhl. The moral authority of chatgpt. arXiv preprint arXiv:2301.07098, 2023. — [arXiv:2301.07098](https://arxiv.org/abs/2301.07098) find
  36. Anne Lauscher, Tobias Lueken, and Goran Glavaš. Sustainable modular debiasing of language models. arXiv preprint arXiv:2109.03646, 2021.
  37. Joel Lehman. Machine love. arXiv preprint arXiv:2302.09248, 2023. — [arXiv:2302.09248](https://arxiv.org/abs/2302.09248) find
  38. Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning, 2012. find
  39. Xingxuan Li, Yutong Li, Linlin Liu, Lidong Bing, and Shafiq Joty. Is gpt-3 a psychopath? evaluating large language models from a psychological perspective. arXiv preprint arXiv:2212.10529, 2022. — [arXiv:2212.10529](https://arxiv.org/abs/2212.10529) find
  40. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 378(6624):1092–1097, 2022.
  41. Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. Towards understanding and mitigating social biases in language models. In International Conference on Machine Learning, pages 6565–6576. PMLR, 2021.
  42. Valentin Liévin, Christoffer Egeberg Hother, and Ole Winther. Can large language models reason about medical questions? arXiv preprint arXiv:2207.08143, 2022. find
  43. Kyle Mahowald, Anna A Ivanova, Idan A Blank, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. Dissociating language and thought in large language models: a cognitive perspective. arXiv preprint arXiv:2301.06627, 2023. find
  44. Fadel M. Megahed, Ying-Ju Chen, Joshua A. Ferris, Sven Knoth, and L. Allison Jones-Farmer. How generative ai models such as chatgpt can be (mis)used in spc practice, education, and research? an exploratory study, 2023. find
  45. Moin Nadeem, Anna Bethke, and Siva Reddy. Stereoset: Measuring stereotypical bias in pretrained language models. arXiv preprint arXiv:2004.09456, 2020.
  46. Oded Nov, Nina Singh, and Devin M Mann. Putting chatgpt’s medical advice to the (turing) test. medRxiv, pages 2023–01, 2023. find
  47. Miguel Ortega-Martı́n, Óscar Garcı́a-Sierra, Alfonso Ardoiz, Jorge Álvarez, Juan Carlos Armenteros, and Adrián Alonso. Linguistic ambiguity analysis in chatgpt. arXiv preprint arXiv:2302.06426, 2023. find
  48. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191, 2021.
  49. Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. Is chatgpt a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476, 2023.
  50. Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. Can ai-generated text be reliably detected? arXiv preprint arXiv:2303.11156, 2023.
  51. Karthik Abinav Sankararaman, Sinong Wang, and Han Fang. Bayesformer: Transformer with uncertainty estimation. arXiv preprint arXiv:2206.00826, 2022. — [arXiv:2206.00826](https://arxiv.org/abs/2206.00826) find
  52. Abulhair Saparov and He He. Language models are greedy reasoners: A systematic formal analysis of chain-of-thought. arXiv preprint arXiv:2210.01240, 2022.
  53. John R Searle, S Willis, et al. Consciousness and language. Cambridge University Press, 2002. find
  54. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. find
  55. Dominik Sobania, Martin Briesch, Carol Hanna, and Justyna Petke. An analysis of the automatic bug fixing performance of chatgpt. arXiv preprint arXiv:2301.08653, 2023. — [arXiv:2301.08653](https://arxiv.org/abs/2301.08653) find
  56. Chris Stokel-Walker and Richard Van Noorden. What chatgpt and generative ai mean for science. Nature, 614(7947):214–216, 2023. find
  57. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243, 2019.
  58. Teo Susnjak. Chatgpt: The end of online exam integrity? arXiv preprint arXiv:2212.09292, 2022. find
  59. Eva AM van Dis, Johan Bollen, Willem Zuidema, Robert van Rooij, and Claudi L Bockting. Chatgpt: five priorities for research. Nature, 614(7947):224–226, 2023. find
  60. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 33:12388–12401, 2020. find
  61. Eric Wallace, Tony Z Zhao, Shi Feng, and Sameer Singh. Concealed data poisoning attacks on nlp models. arXiv preprint arXiv:2010.12563, 2020. find
  62. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019.
  63. Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, and Po-Sen Huang. Challenges in detoxifying language models. arXiv preprint arXiv:2109.07445, 2021.
  64. Frank F Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, pages 1–10, 2022. find
  65. Blaise Agüera y Arcas. Do large language models understand us? Daedalus, 151(2):183–197, 2022. find
  66. Xianjun Yang, Yan Li, Xinlu Zhang, Haifeng Chen, and Wei Cheng. Exploring the limits of chatgpt for query or aspect-based text summarization. arXiv preprint arXiv:2302.08081, 2023.
  67. Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, et al. A comprehensive survey on pretrained foundation models: A history from bert to chatgpt. arXiv preprint arXiv:2302.09419, 2023. find
  68. Terry Yue Zhuo, Yujin Huang, Chunyang Chen, and Zhenchang Xing. Exploring ai ethics of chatgpt: A diagnostic analysis. arXiv preprint arXiv:2301.12867, 2023.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/borji-2023-categorical-archive-chatgpt-failures/