Elamur Research⌕
☰
home / papers / 2024 / H-ARC: A Robust Estimate of Human Performance on the Abstraction and Reasoning Corpus Benchmark

H-ARC: A Robust Estimate of Human Performance on the Abstraction and Reasoning Corpus Benchmark

Paper · 2024 · Solim LeGris, Wai Keen Vong, Brenden M. Lake et al.

Summary

The ARC benchmark tests whether a solver can infer a hidden transformation rule from a few example grid pairs and apply it to a new input. Earlier human baselines covered only 40 training tasks or used modified ARC variants, so it was unclear how well people do on the full public set, especially the harder evaluation split. The authors recruited online workers through Amazon Mechanical Turk and CloudResearch. Each participant tackled five randomly drawn tasks in an interface that mimics the original ARC editor. People had three tries per task, were told only whether each try was right or wrong, and wrote down their solutions in words. All 400 training and 400 evaluation tasks were covered. Average accuracy was roughly three quarters on training tasks and under two thirds on evaluation tasks. Even so, nearly every task was solved by someone, and humans clearly beat the strongest LLM-based systems, particularly in fixing mistakes after feedback. The released H-ARC dataset gives both AI researchers and cognitive scientists a solid human reference point and detailed behavioral traces.

Key points

  • Scale: 1729 participants took part, about 10–12 per task. The resulting H-ARC dataset holds 15,744 attempts, with step-by-step action traces and natural-language rule descriptions.
  • Training set (3 attempts): humans averaged 76.2% (bounds 73.3–77.2% after accounting for dropouts).
  • Evaluation set (3 attempts): humans averaged 64.2% (bounds 55.9–68.9%). These figures are below the earlier 83.8% estimate, which was based on 40 training tasks.
  • Solvability: 790 of 800 tasks (98.8%) were solved by at least one person within three tries.
  • Comparison with AI: the best AI baselines scored far lower on the evaluation set. GPT-4o-NS reached 38.5/42.0% and Claude-3.5-N reached 19.3/20.7% (1/2 attempts). Humans also gained much more from a second attempt, about +21–26% proportional versus +7–9% for the models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York UniversityNew York · United States233 works · 104,003 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksComputer Science Applications · 184 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,948 worksEvaluation & benchmarks · 1,394 works

Cite

Cite as Solim LeGris, Wai Keen Vong, Brenden M. Lake and Todd M. Gureckis. 2024. H-ARC: A Robust Estimate of Human Performance on the Abstraction and Reasoning Corpus Benchmark. arXiv:2409.01374.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2409.01374 · arXiv 2409.01374

@misc{legris2024h,
  title = {{H-ARC: A Robust Estimate of Human Performance on the Abstraction and Reasoning Corpus Benchmark}},
  author = {Solim LeGris, Wai Keen Vong, Brenden M. Lake and Todd M. Gureckis},
  year = {2024},
  doi = {10.48550/arxiv.2409.01374},
  eprint = {2409.01374},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2409.01374}
}

References

16 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. (2023). GPT-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Acquaviva, S., Pu, Y., Kryven, M., Sechopoulos, T., Wong, C., Ecanow, G., Nye, M., Tessler, M., and Tenenbaum, J. (2022). Communicating natural programs to humans and machines. Advances in Neural Information Processing Systems, 35:3731–3743.
  3. Anthropic (2024). Claude 3.5 sonnet model card addendum. preprint. Accessed on 8/15/24. find
  4. Cheyette, S. J. and Piantadosi, S. T. (2024). Response to Difficulty Drives Variation in IQ Test Performance. Open Mind, 8:265–277. find
  5. Chollet, F. (2019). On the measure of intelligence. arXiv preprint arXiv:1911.01547. find
  6. Greenblatt, R. (2024). Getting 50% (SoTA) on ARC-AGI with GPT-4o. https://redwoodresearch.substack.com/p/getting-50-sota-on-arc-agi-with-gpt. Accessed on 8/15/24. ↗
  7. Hauser, D. J., Moss, A. J., Rosenzweig, C., Jaffe, S. N., Robinson, J., and Litman, L. (2023). Evaluating cloudresearch’s approved group as a solution for problematic data quality on mturk. Behavior Research Methods, 55(8):3953–3964. find
  8. Johnson, A., Vong, W. K., Lake, B., and Gureckis, T. M. (2021). Fast and flexible: Human program induction in abstract reasoning tasks. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 43. find
  9. Kamradt, G. (2024). Using frontier models on arc agi via langchain. https://www.kaggle.com/code/gregkamradt/using-frontier-models-on-arc-agi-via-langchain? scriptVersionId=185410590. Kaggle Notebook, Version 185410590, Accessed on 8/15/24. ↗
  10. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). Scaling laws for neural language models.
  11. Mitchell, M., Palmarini, A. B., and Moskvichev, A. (2023). Comparing humans, gpt-4, and gpt-4v on abstraction and reasoning tasks. arXiv preprint arXiv:2311.09247. — [arXiv:2311.09247](https://arxiv.org/abs/2311.09247) find
  12. Moskvichev, A., Odouard, V. V., and Mitchell, M. (2023). The conceptarc benchmark: Evaluating understanding and generalization in the arc domain. arXiv preprint arXiv:2305.07141. find
  13. Russek, E., Acosta-Kane, D., van Opheusden, B., Mattar, M. G., and Griffiths, T. (2022). Time spent thinking in online chess reflects the value of computation. find
  14. Wang, R., Zelikman, E., Poesia, G., Pu, Y., Haber, N., and Goodman, N. D. (2023). Hypothesis search: Inductive reasoning with language models. arXiv preprint arXiv:2309.05660. find
  15. Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022). Emergent abilities of large language models. arXiv preprint arXiv:2206.07682.
  16. Zettersten, M. and Lupyan, G. (2020). Finding categories through words: More nameable features improve category learning. Cognition, 196(104135):104135. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/legris-2024-h-arc-robust-estimate-human/