Elamur Research⌕
☰
home / papers / 2025 / On Benchmarking Human-Like Intelligence in Machines

On Benchmarking Human-Like Intelligence in Machines

Paper · 2025 · Lance Ying, Katherine M. Collins, Lionel Wong et al.

Summary

This perspective paper argues that widely used AI benchmarks cannot reliably show whether a model reasons in a human-like way. The authors distinguish human-like from human-level ability. Matching human accuracy is not enough. A human-like system should also reproduce people's typical mistakes, their graded confidence and the spread of answers across a population. They work at the behavioral level, which roughly corresponds to Marr's computational level. From there they identify three weaknesses in current practice:

  • labels that were never validated against real human judgments;
  • single gold answers that hide disagreement and uncertainty;
  • oversimplified tasks with weak ecological validity.

To support these claims, they collected human responses on nine existing benchmarks, each of which assigns one ground-truth label per item. The human data revealed substantial problems in how the tasks and labels were designed. The authors then propose five recommendations drawn from cognitive modeling. The work matters most for evaluating subjective tasks such as morality, humor or social judgment. In those tasks, a fixed answer key misrepresents genuine human responses.

Key points

  • Benchmarks studied: seven BIG-Bench common-sense tasks (Fantasy reasoning, Social IQA, Moral permissibility, Simple ethical questions, Social support, Irony identification, Dark humor detection), plus the ambiguous story task from ToMBench and the Theory of Mind task from BigToM.
  • Conceptual framing: "human-level" means quantitative parity with human performance, while "human-like" means qualitative similarity in error profiles and response distributions. Representational alignment, which concerns whether internal representations match human ones, is acknowledged but not the focus.
  • Recommended practices: gather human labels whenever claiming human-likeness, and evaluate against population-level answer distributions and each annotator's graded beliefs. Tasks should be naturalistic and theory-driven. Ambiguity in the stimulus itself should be kept separate from confusion about what the task is asking.
  • Explicit scope limits: the advice does not target domains where the goal is superhuman performance, such as protein structure prediction, weather forecasting or theorem proving. It also does not target expert-level abstract reasoning tests like ARC-AGI.
  • Open challenges:
    • Human data collection is costly and hard to scale; crowdsourcing on Amazon Mechanical Turk or Prolific helps but can introduce sampling bias.
    • It remains unresolved which human cognitive biases AI systems should reproduce and which they should avoid.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York UniversityNew York · United States233 works · 104,003 citations🎓Johns Hopkins UniversityBaltimore · United States254 works · 106,461 citations🎓University of CambridgeCambridge · United Kingdom287 works · 125,174 citations🎓Massachusetts Institute of TechnologyCambridge · United States274 works · 82,013 citations🎓Princeton UniversityPrinceton · United States137 works · 53,849 citations🎓Harvard UniversityCambridge · United States137 works · 45,850 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

Cite

Cite as Lance Ying, Katherine M. Collins, Lionel Wong, Ilia Sucholutsky, Ryan Liu, Adrian Weller, Tianmin Shu, Thomas L. Griffiths and Joshua B. Tenenbaum. 2025. On Benchmarking Human-Like Intelligence in Machines. arXiv:2502.20502.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2502.20502 · arXiv 2502.20502

@misc{ying2025benchmarking,
  title = {{On Benchmarking Human-Like Intelligence in Machines}},
  author = {Lance Ying, Katherine M. Collins, Lionel Wong, Ilia Sucholutsky, Ryan Liu, Adrian Weller, Tianmin Shu, Thomas L. Griffiths and Joshua B. Tenenbaum},
  year = {2025},
  doi = {10.48550/arxiv.2502.20502},
  eprint = {2502.20502},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2502.20502}
}

References

104 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alan Turing. Computing machinery and intelligence. Mind, 59(236):433, 1950. find
  2. Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017. find
  3. Stephen Cave and Kanta Dihal. Imagining AI: how the world sees intelligent machines. Oxford University Press, 2023. find
  4. Joseph Weizenbaum. Eliza—a computer program for the study of natural language communication between man and machine. Communications of the ACM, 9(1):36–45, 1966. find
  5. John R Anderson, C Franklin Boyle, Albert T Corbett, and Matthew W Lewis. Cognitive modeling and intelligent tutoring. 1990. find
  6. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022.
  7. Zhuang Chen, Jincenzi Wu, Jinfeng Zhou, Bosi Wen, Guanqun Bi, Gongyao Jiang, Yaru Cao, Mengting Hu, Yunghwei Lai, Zexuan Xiong, and Minlie Huang. Tombench: Benchmarking theory of mind in large language models, 2024.
  8. Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah Goodman. Understanding social reasoning in language models with language models. Advances in Neural Information Processing Systems, 36, 2024. find
  9. Ned Block. Psychologism and behaviorism. The Philosophical Review, 90(1):5–43, 1981. find
  10. David Marr. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. W.H. Freeman, San Francisco, 1982. find
  11. Ilia Sucholutsky, Lukas Muttenthaler, Adrian Weller, Andi Peng, and Andreea Bobu et al. Getting aligned on representational alignment, 2023a. find
  12. Lukas Muttenthaler, Klaus Greff, Frieda Born, Bernhard Spitzer, Simon Kornblith, Michael C Mozer, Klaus-Robert MÞller, Thomas Unterthiner, and Andrew K Lampinen. Aligning machine and human visual representations across abstraction levels. arXiv preprint arXiv:2409.06509, 2024. — [arXiv:2409.06509](https://arxiv.org/abs/2409.06509) find
  13. Frank Rosenblatt. The perceptron: a probabilistic model for information storage and organization in the brain. Psychological review, 65(6):386, 1958. find
  14. David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. (1986) d. e. rumelhart, g. e. hinton, and r. j. williams, ”learning internal representations by error propagation,” parallel distributed processing: Explorations in the microstructures of cognition, vol. i, d. e. rumelhart and j. l. mcc… — [doi:10.7551/mitpress/](https://doi.org/10.7551/mitpress/) find
  15. Marvin Minsky. Society of mind. Simon and Schuster, 1988. find
  16. Melanie Mitchell. The turing test and our shifting conceptions of intelligence, 2024. find
  17. Albert Bandura. Social cognitive theory: An agentic perspective. Annual review of psychology, 52 (1):1–26, 2001. find
  18. Vittorio Gallese. Before and below ‘theory of mind’: embodied simulation and the neural correlates of social cognition. Philosophical Transactions of the Royal Society B: Biological Sciences, 362 (1480):659–669, 2007. find
  19. Katherine M Collins, Ilia Sucholutsky, Umang Bhatt, Kartik Chandra, Lionel Wong, Mina Lee, Cedegao E Zhang, Tan Zhi-Xuan, Mark Ho, Vikash Mansinghka, et al. Building machines that learn and think with people. Nature Human Behaviour, 8(10):1851–1863, 2024a. find
  20. Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, and Sanjit et al Seshia. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32, 2019. find
  21. Mark K Ho and Thomas L Griffiths. Cognitive science as a source of forward and inverse models of human decisions for robotics and control. Annual Review of Control, Robotics, and Autonomous Systems, 5:33–53, 2022. find
  22. Tan Zhi-Xuan, Lance Ying, Vikash Mansinghka, and Joshua B Tenenbaum. Pragmatic instruction following and goal assistance via cooperative language-guided inverse planning. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pages 2094– 2103, 2024. find
  23. Anca D Dragan, Kenton CT Lee, and Siddhartha S Srinivasa. Legibility and predictability of robot motion. In 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pages 301–308. find
  24. Jaime F Fisac, Chang Liu, Jessica B Hamrick, Shankar Sastry, and J Karl et al Hedrick. Generating plans that predict themselves. In Algorithmic Foundations of Robotics XII: Proceedings of the Twelfth Workshop on the Algorithmic Foundations of Robotics, pages 144–159. Springer, 2020. find
  25. Ryan Liu, Howard Yen, Raja Marjieh, Thomas L. Griffiths, and Ranjay Krishna. Improving interpersonal communication by simulating audiences with language models, 2023. find
  26. Omar Shaikh, Valentino Emil Chai, Michele Gelfand, Diyi Yang, and Michael S Bernstein. Rehearsal: Simulating conflict to teach conflict resolution. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–20, 2024. find
  27. Ashwini Ashokkumar, Luke Hewitt, Isaias Ghezae, and Robb Willer. Predicting results of social science experiments using large language models. Work. Pap., New York Univ., New York, NY, 2024. find
  28. Joon Sung Park, Carolyn Q Zou, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Robb Willer, Percy Liang, and Michael S Bernstein. Generative agent simulations of 1,000 people. arXiv preprint arXiv:2411.10109, 2024.
  29. Dorottya Demszky, Diyi Yang, David S Yeager, Christopher J Bryan, Margarett Clapper, Susannah Chandhok, Johannes C Eichstaedt, Cameron Hecht, Jeremy Jamieson, Meghann Johnson, et al. Using large language models in psychology. Nature Reviews Psychology, 2(11):688–701, 2023. find
  30. Perttu Hämäläinen, Mikke Tavast, and Anton Kunnari. Evaluating large language models in generating synthetic hci research data: a case study. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–19, 2023. find
  31. James Brand, Ayelet Israeli, and Donald Ngwe. Using llms for market research. Harvard Business School Marketing Unit Working Paper, (23-062), 2023. find
  32. Marco Gerosa, Bianca Trinkenreich, Igor Steinmacher, and Anita Sarma. Can ai serve as a substitute for human subjects in software engineering research? Automated Software Engineering, 31 (1):13, 2024. find
  33. Michael C Frank. Baby steps in evaluating the capacities of large language models. Nature Reviews Psychology, 2(8):451–452, 2023. find
  34. Anna A Ivanova. How to evaluate the cognitive abilities of LLMs. Nature Human Behaviour, 9(2): 230–233, 2025. find
  35. Richard M Shiffrin and Melanie Mitchell. Probing the psychology of AI models. Proceedings of the National Academy of Sciences, 120(10):e2300963120, 2023. find
  36. Konstantinos Voudouris, Matthew Crosby, Marta Halina, and José Hernández-Orallo. Bringing comparative cognition to computers. Trends in Cognitive Sciences, 2024. find
  37. Shrestha Rane, Nick Chater, and Mike Oaksford. Principles of animal cognition for LLM evaluations: A case study on transitive inference. In NeurIPS 2024 Workshop on Behavioral Machine Learning, 2024. find
  38. Chaz Firestone. Performance vs. competence in human–machine comparisons. Proceedings of the National Academy of Sciences, 117(43):26562–26571, 2020. find
  39. Jennifer Hu and Michael C Frank. Auxiliary task demands mask the capabilities of smaller language models. In First Conference on Language Modeling, 2024.
  40. Andrew Lampinen. Can language models handle recursively nested grammatical structures? A case study on comparing models and humans. Computational Linguistics, 50(4):1441–1476, 2024.
  41. Jennifer Hu and Roger Levy. Prompting is not a substitute for probability measurements in large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5040–5060, 2023.
  42. Heinz Wimmer and Josef Perner. Beliefs about beliefs: Representation and constraining function of wrong beliefs in young children’s understanding of deception. Cognition, 13(1):103–128, 1983. find
  43. Jesse Graham, Jonathan Haidt, and Brian A Nosek. Liberals and conservatives rely on different sets of moral foundations. Journal of personality and social psychology, 96(5):1029, 2009. find
  44. Jesse Graham, Peter Meindl, Erica Beall, Kate M Johnson, and Li Zhang. Cultural differences in moral judgment and behavior, across and within societies. Current Opinion in Psychology, 8: 125–130, 2016. find
  45. Sydney Levine, Max Kleiman-Weiner, Laura Schulz, Joshua Tenenbaum, and Fiery Cushman. The logic of universalization guides moral judgment. Proceedings of the National Academy of Sciences, 117(42):26158–26169, 2020. find
  46. Rosa Lafer-Sousa, Katherine L Hermann, and Bevil R Conway. Striking individual differences in color perception uncovered by ‘the dress’ photograph. Current Biology, 25(13):R545–R546, 2015. find
  47. Curtis G Northcutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks. arXiv preprint arXiv:2103.14749, 2021. find
  48. Neeltje J Boogert, Joah R Madden, Julie Morand-Ferron, and Alex Thornton. Measuring and understanding individual differences in cognition, 2018. find
  49. Joan Danielle K Ongchoco, Isaac M Davis, Julian Jara-Ettinger, and LA Paul. When new experience leads to new knowledge: A computational framework for formalizing epistemically transformative experiences. Open Mind, 8:1291–1311, 2024. find
  50. Joseph Henrich, Steven J Heine, and Ara Norenzayan. The weirdest people in the world? Behavioral and brain sciences, 33(2-3):61–83, 2010. find
  51. Mitchell L Gordon, Michelle S Lam, Joon Sung Park, Kayur Patel, and Jeff et al Hancock. Jury learning: Integrating dissenting voices into machine learning models. In CHI Conference on Human Factors in Computing Systems, pages 1–19, 2022. find
  52. Joshua C Peterson, Ruairidh M Battleday, Thomas L Griffiths, and Olga Russakovsky. Human uncertainty makes classification more robust. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9617–9626, 2019. find
  53. Ilia Sucholutsky, Ruairidh M Battleday, Katherine M Collins, Raja Marjieh, and Joshua et al Peterson. On the informativeness of supervision signals. In Uncertainty in Artificial Intelligence, pages 2036–2046. find
  54. Katherine M Collins, Umang Bhatt, Weiyang Liu, Vihari Piratla, Ilia Sucholutsky, Bradley Love, and Adrian Weller. Human-in-the-loop mixup. In Uncertainty in Artificial Intelligence, pages 454–464. PMLR, 2023a. find
  55. Alexandra Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, and Barbara et al Plank. A case for soft loss functions. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 8(1):173–177, Oct. 2020. find
  56. Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Michael Bean, Katerina Margatina, Rafael Mosquera, Juan Manuel Ciro, Max Bartolo, Adina Williams, He He, Bertie Vidgen, and Scott A. Hale. The PRISM alignment dataset: What participatory, representative and individualised human feedback r… — The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models
  57. Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, et al. A roadmap to pluralistic alignment. arXiv preprint arXiv:2402.05070, 2024.
  58. Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. A survey of confidence estimation and calibration in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn…
  59. Amos Tversky and Daniel Kahneman. Judgment under uncertainty: Heuristics and biases: Biases in judgments reveal some heuristics of thinking under uncertainty. science, 185(4157):1124–1131, 1974. find
  60. Nick Chater and Christopher D Manning. Probabilistic models of language processing and acquisition. Trends in cognitive sciences, 10(7):335–344, 2006. find
  61. Thomas L Griffiths, Nick Chater, and Joshua B Tenenbaum. Bayesian models of cognition: reverse engineering the mind. MIT Press, 2024. find
  62. Alexandra Uma, Dina Almanea, and Massimo Poesio. Scaling and disagreements: Bias, noise, and ambiguity. Frontiers in Artificial Intelligence, 5, 2022. find
  63. Lars Schmarje, Vasco Grossmann, Claudius Zelenka, Sabine Dippel, and Rainer et al Kiko. Is one annotation enough? a data-centric image classification benchmark for noisy and ambiguous label estimation. arXiv preprint arXiv:2207.06214, 2022. — [arXiv:2207.06214](https://arxiv.org/abs/2207.06214) find
  64. Mark Steyvers, Heliodoro Tejeda, Gavin Kerrigan, and Padhraic Smyth. Bayesian modeling of human–ai complementarity. Proceedings of the National Academy of Sciences, 119(11): e2111547119, 2022. find
  65. Katherine Maeve Collins, Matthew Barker, Mateo Espinosa Zarlenga, Naveen Raman, and Umang et al Bhatt. Human uncertainty in concept-based ai systems. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, pages 869–889, 2023b. find
  66. A. O’Hagan, C. E. Buck, A. Daneshkhah, J. R. Eiser, and P. H. Garthwaite et al. Uncertain Judgements: Eliciting Expert Probabilities. John Wiley, Chichester, 2006. find
  67. Anthony O’Hagan. Expert knowledge elicitation: Subjective but scientific. The American Statistician, 73(sup1):69–81, 2019. doi: 10.1080/00031305.2018.1518265. — [doi:10.1080/00031305.2018.1518265](https://doi.org/10.1080/00031305.2018.1518265) find
  68. Lukas Muttenthaler, Lorenz Linhardt, Jonas Dippel, Robert A Vandermeulen, Katherine Hermann, Andrew Lampinen, and Simon Kornblith. Improving neural network representations using human similarity judgments. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in N… find
  69. Lukas Muttenthaler, Jonas Dippel, Lorenz Linhardt, Robert A Vandermeulen, and Simon Kornblith. Human alignment of neural network representations. In The Eleventh International Conference on Learning Representations, 2022. find
  70. Gideon Keren. Calibration and probability judgements: Conceptual and methodological issues. Acta psychologica, 77(3):217–273, 1991. find
  71. Joshua Tenenbaum. Bayesian modeling of human concept learning. Advances in neural information processing systems, 11, 1998. find
  72. Paul D Windschitl and Gary L Wells. Measuring psychological uncertainty: Verbal versus numeric methods. Journal of Experimental Psychology: Applied, 2(4):343, 1996. find
  73. José Hernández-Orallo. Evaluation in artificial intelligence: from task-oriented to ability-oriented measurement. Artificial Intelligence Review, 48:397–447, 2017. find
  74. Lee J Cronbach and Paul E Meehl. Construct validity in psychological tests. Psychological Bulletin, 52(4):281–302, 1955. find
  75. Samuel Messick. Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9): 741–749, 1995. find
  76. Andrew Bean et al. Measuring what matters: Construct validity in large language model benchmarks. arXiv preprint, 2025. find
  77. Matthew Le, Y-Lan Boureau, and Maximilian Nickel. Revisiting the evaluation of theory of mind through question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)… find
  78. Michal Kosinski. Evaluating large language models in theory of mind tasks. Proceedings of the National Academy of Sciences, 121(45):e2405460121, 2024.
  79. Cindy Beaudoin, Élizabel Leblanc, Charlotte Gagner, and Miriam H Beauchamp. Systematic review and inventory of theory of mind measures for young children. Frontiers in psychology, 10:2905, 2020. find
  80. Paul Bloom and Tim P German. Two reasons to abandon the false belief task as a test of theory of mind. Cognition, 77(1):B25–B31, 2000. find
  81. Jennifer Hu, Felipe Sosa, and Tomer Ullman. Re-evaluating theory of mind evaluation in large language models. Philosophical Transactions of the Royal Society B: Biological Sciences, 380 (1932), 2025. find
  82. Ian W Eisenberg, Patrick G Bissett, A Zeynep Enkavi, James Li, David P MacKinnon, Lisa A Marsch, and Russell A Poldrack. Uncovering the structure of self-regulation through data-driven ontology discovery. Nature Communications, 10(1):2319, 2019. find
  83. Amir Jacovi, Avi Caciularu, Yuval Pinter, and Yoav Goldberg. Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5075–5084, 2023. find
  84. Chris L Baker, Julian Jara-Ettinger, Rebecca Saxe, and Joshua B Tenenbaum. Rational quantitative attribution of beliefs, desires and percepts in human mentalizing. Nature Human Behaviour, 1(4):0064, 2017. find
  85. Julian Jara-Ettinger, Laura E Schulz, and Joshua B Tenenbaum. The naive utility calculus as a unified, quantitative framework for action understanding. Cognitive Psychology, 123:101334, 2020. find
  86. Lance Ying, Tan Zhi-Xuan, Vikash Mansinghka, and Joshua B Tenenbaum. Inferring the goals of communicating agents from actions and instructions. In Proceedings of the AAAI Symposium Series, volume 2, pages 26–33, 2023. find
  87. Wilka Carvalho and Andrew Lampinen. Naturalistic computational cognitive science: Towards generalizable models and theories that capture the full range of natural behavior. arXiv preprint arXiv:2502.20349, 2025. — [arXiv:2502.20349](https://arxiv.org/abs/2502.20349) find
  88. Alex H Taylor, Amalia P M Bastos, Rachael L Brown, and Colin Allen. The signature-testing approach to mapping biological and artificial intelligences. Trends in Cognitive Sciences, 26(9): 738–750, 2022. find
  89. Chuanyang Jin, Yutong Wu, Jing Cao, Jiannan Xiang, Yen-Ling Kuo, Zhiting Hu, Tomer Ullman, Antonio Torralba, Joshua B Tenenbaum, and Tianmin Shu. MMToM-QA: Multimodal theory of mind question answering. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024.
  90. Lance Ying, Xinyi Li, Shivam Aarya, Yizirui Fang, Yifan Yin, Jason Xinyu Liu, Stefanie Tellex, Joshua B Tenenbaum, and Tianmin Shu. Pragmatic embodied spoken instruction following in human-robot collaboration with theory of mind. arXiv preprint arXiv:2409.10849, 2024. — [arXiv:2409.10849](https://arxiv.org/abs/2409.10849) find
  91. Lexin Zhou, Wout Schellaert, Fernando Martı́nez-Plumed, Yael Moros-Daval, Cèsar Ferri, and José Hernández-Orallo. Larger and more instructable language models become less reliable. Nature, pages 1–8, 2024. find
  92. John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žı́dek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. nature, 596(7873):583–589, 2021.
  93. Remi Lam, Alvaro Sanchez-Gonzalez, Matthew Willson, Peter Wirnsberger, Meire Fortunato, Ferran Alet, Suman Ravuri, Timo Ewalds, Zach Eaton-Rosen, Weihua Hu, et al. Learning skillful medium-range global weather forecasting. Science, 382(6677):1416–1421, 2023. find
  94. Cristian Bodnar, Wessel P Bruinsma, Ana Lucic, Megan Stanley, Johannes Brandstetter, Patrick Garvan, Maik Riechert, Jonathan Weyn, Haiyu Dong, Anna Vaughan, et al. Aurora: A foundation model of the atmosphere. arXiv preprint arXiv:2405.13063, 2024. find
  95. Thomas L Griffiths. Understanding human intelligence through human limitations. Trends in Cognitive Sciences, 24(11):873–883, 2020. find
  96. Falk Lieder and Thomas L Griffiths. Resource-rational analysis: Understanding human cognition as the optimal use of limited computational resources. Behavioral and brain sciences, 43:e1, 2020. find
  97. Martie G Haselton, Gregory A Bryant, Andreas Wilke, David A Frederick, Andrew Galperin, Willem E Frankenhuis, and Tyler Moore. Adaptive rationality: An evolutionary perspective on cognitive bias. Social Cognition, 27(5):733–763, 2009. find
  98. Ryan Liu, Jiayi Geng, Joshua C Peterson, Ilia Sucholutsky, and Thomas L Griffiths. Large language models assume people are more rational than we really are. arXiv preprint arXiv:2406.17055, 2024. — [arXiv:2406.17055](https://arxiv.org/abs/2406.17055) find
  99. Andrew K Lampinen, Ishita Dasgupta, Stephanie C Y Chan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. Language models, like humans, show content effects on reasoning tasks. Proceedings of the National Academy of Sciences, 121(41):e2322911121, 2024. find
  100. Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. Advances in Neural Information Processing Systems, 36: 59008–59033, 2023.
  101. Katherine M Collins, Najoung Kim, Yonatan Bitton, Verena Rieser, Shayegan Omidshafiei, Yushi Hu, Sherol Chen, Senjuti Dutta, Minsuk Chang, Kimin Lee, et al. Beyond thumbs up/down: Untangling challenges of fine-grained feedback for text-to-image generation. In Proceedings of the AAAI/ACM Conference o… find
  102. John Joon Young Chung, Jean Y Song, Sindhu Kutty, Sungsoo Hong, Juho Kim, and Walter S Lasecki. Efficient elicitation approaches to estimate collective crowd answers. Proceedings of the ACM on Human-Computer Interaction, 3(CSCW):1–25, 2019. find
  103. Zijian Wang and David Jurgens. It’s going to be okay: Measuring access to support in online communities. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 33–45, 2018. find
  104. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728, 2019.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ying-2025-benchmarking-human-like-intelligence-machines/