📄 CogToM: A Comprehensive Theory of Mind Benchmark inspired by Human Cognition for Large Language Models

Summary

Benchmarks that test Theory of Mind (ToM) in LLMs mostly rely on false-belief stories. This narrow focus covers only a small part of what developmental psychology studies. CogToM addresses the gap by converting 46 established and newly designed psychological paradigms into scene-based multiple-choice items in Chinese and English. Each item is linked to the ATOMS-based taxonomy of ToM abilities. The data were seeded by experts, expanded with GPT-5.1 and then checked through several rounds of human review and double annotation, with 49 people involved in total.

The authors evaluated 22 LLMs of different sizes, families and release dates. Scores varied widely, which shows the benchmark can separate models well. Perception-related tasks were the hardest for every model, even though people find them trivial. Models also often did better on skills that children acquire late than on those acquired early. The authors interpret this as a form of Moravec's Paradox, suggesting that LLM "ToM" comes from linguistic pattern matching rather than grounded, human-like development.

Key points

• Scale and structure: CogToM contains 8,513 bilingual (Chinese-English) entries across 46 tasks, with 54–130 data groups per task. The tasks map onto 7 capability categories and 36 sub-capabilities. Four paradigms are entirely new, including Expanding Tasks [Synesthetic Fallacy] and a Spatial Construction Task.
• Construction: A six-stage pipeline produced the data, and each entry passed at least five rounds of human supervision. GPT-5.1 generated the expansions, and 42 graduate students in Philosophy and AI performed the annotation.
• Models tested: 22 LLMs were evaluated, including GPT-5.1 and Qwen3-Max. Chain-of-thought prompting improved Belief tasks and, for frontier models, Percept tasks. Even so, Percept remained the lowest-scoring dimension for all models.
• Human-model mismatch: On tasks where human inter-annotator agreement exceeded 90%, average model accuracy dropped to 30–80%. On Percept tasks, humans agreed 100% while models scored below 30%. Most of these hard cases come from the newly introduced paradigms.
• Developmental inversion: GPT-5.1 reached 62% on the early-childhood See-Know Task but 96% on the later-acquired TEC: Hidden Emotions task. Qwen-7B-Chat climbed from 5% to 71% along a sequence of tasks ordered by the age at which children master them, the reverse of the human pattern.
• Availability: Code and data are released at github.com/Beijing-AISI/CogToM.